Evaluation

You are currently browsing the archive for the Evaluation category.

He doesn't know either: Gilbert demonstrating the magnet before Queen Elizabeth /Wellcome Images

Last year I saw A C Grayling talk on happiness and it’s importance. He said that he bans his students from using the word ‘happy’, that it’s a lazy portmanteau word. He thinks that if you are forced to choose a different word – hopeful, exhilarated, content – you’ll think about what you really mean far more clearly.

I think the same can sometimes be true of jargon. Scientists taking part in I’m a Scientist have told us before that explaining yourself without using jargon is hard work, but unexpectedly rewarding. It means you have to think through what you mean and it exposes your mental shortcuts.

We saw a great example of this in Imaging Zone. A fairly innocuous-seeming question (Why do magnets attract and repel?) pretty much lead to the scientists realising that they don’t actually know how magnets worked. Not really. Not when they tried to actually explain it to other people, without using jargon.

Now these scientists include a man who spends many of his days working with an fMRI scanner, containing a magnet so powerful that you have to remove any ferrous object from your person before entering the room. But still, magnetism turned out to be one of those things that he learned about years ago, and sort of assumed he understood.

I’m expecting that some of you have the same feeling I did when reading that Q+A – ‘OMG, I don’t REALLY understand how magnets work either. How did I not notice that before?’ I think the thing is, most of us rarely discuss how magnets work. And when we do, we use technical words (‘dipole’, ‘electromagnetic’, ‘electron shell’) which we and our listeners all know, which can obscure the fact that you don’t truly understand the underlying mechanism.

By all accounts this question, and the attempts to answer it, lead to an awful lot of magnetism-related discussions at scientific breakfast tables and coffee machines around the country. So, one outcome, of just this question, has been much thinking about and discussing the mysteries of physics, by scientists, with colleagues, and others, about a subject they all thought they understood, but actually it turns out they’ve got lots of questions about it.

If the essence of science is asking questions and taking nothing for granted, then I’d call this a result.

This post started life as part of a mammoth post I’ve been writing about how June’s I’m a Scientist event went. The post has taken about three days so far and we’re up to 3,000 words. So I thought I should really break it into bits and start bunging some up now.

Well, of course, I’m going to say YES to that question.

What’s This?

Here is the evaluation report on the first half of the debate kits project, comprising the first two kits (from Summer and Autumn 2009). Obviously I was supposed to publish this MONTHS ago, but it’s fallen by the wayside, until now.

Why now?

We have just brought out the third IAS debate kit, on Stem Cells. We posted out 1,500 stem cells debate kits earlier this week.  I got an email from a teacher this morning saying ‘Just got the Stem cell pack…fantastic!’ so I’m hopeful they’ll be as popular as previous kits:-)

What are the debate kits?

The debate kits are designed to get students in their science lessons debating and discussing science issues, seeing things from different points of view and learning to back up their opinions with evidence. Each kit is developed in consultation with teachers, and fact checked with scientists.

Evaluation summary

“This is great I couldn’t have done a debate without it”

I’m a Scientist debate kits is a project to help get more debate and discussion going in the UK’s science classrooms. We are developing and distributing debate kits to UK science teachers. The kits are designed to give a teacher everything they need to run a structured debate on a set topic and help their students develop their discussion skills. The activity gives them a structured way to start discussing and gets them practicing the behaviour. The Wellcome Trust have funded Gallomanor to produce and distribute three sets of kits in 2009/10.

The first kit was developed as part of I’m a Scientist, Get me out of Here!, in 2008. This first kit was re-printed and distributed to 1,500 teachers and science communicators in Summer term 2009 as the first nationally distributed IAS debate kit. We also provide an electronic version of each kit, which more people can download from the website. The second kit was produced in Autumn term 2009. The topic (suggested and picked by teachers) was ‘Are we too clean?’ The third kit will be produced and distributed in Spring term 2010. A fourth kit will be developed in Summer term and distributed electronically.

Outcomes

Key aims met:-

  • Kits well used
  • The resources worked ‘straight out of the box
  • They really engaged the students
  • They developed students’ discussion and critical thinking skills

Kits also:-

  • Were good way for students to learn new content
  • Were effective with a range of students
  • Got quieter students talking
  • Inspired teachers to start making their own!

Recommendations

  • Next kit in Summer term 2010, on stem cells (done!)
  • Produce an electronic large print version (done!)
  • Produce DIY debate kit templates for teachers to use (done!)
  • Increase print run to 2,000 (done!)

More details in the full Debate Kits Interim Report (pdf).

If you would like to order a copy of the latest debate kit (they are free!) go here

If you would like to download a copy of one of the kits go here

If you’d like to know more about the kits or the project there’s more info here, or feel free to get in touch and ask us!

We launched our I’m a Scientist debate kits on 18th June and have now distributed all 1,500 we printed. We’ve been really pleased by the response from teachers. The next kit will be coming out in the Autumn. We’ve written a report on this first phase of the project and we thought a summary of that may of interest to others, so here you go:-

Project: I’m a Scientist Debate Kits

A Wellcome Trust funded project to distribute structured debate kits to science teachers

Key milestones

18/06/09    Promoted kit by email to our contacts and on twitter
22/06/09    Mailers arrived in 5,000 UK secondary schools
23/06/09    Posted on psci-comm
26/06/09    Featured in Planet Science
01/07/09    1st kits arrived in schools
06/07/09    Posted on TES messageboards
27/08/09    Final kits posted out

Other marketing

  • Complimentary kits sent to the 323 specialist science colleges and 177 largest secondary schools plus various IAS contacts (e.g. at BSA, SLCs, ASE) with request to pass info on to teachers who may be interested.
  • Ongoing twitter and blog coverage.
  • Contact with Aimhigher who agreed to help publicise us to their target schools.
  • Attended Wiltshire STEM ambassador event.
  • Presentation to student teachers and to their mentors at UWE.
Orders over time

Orders over time

I think it’s clear that the events I’ve highlighted have had the biggest effect on teachers ordering kits. The dates of other interventions did not coincide with a spike. However, other activities, especially word of mouth between teachers, are invaluable and contribute to the long term take up of the kits, but will not give the same sort of spike. We are still seeing new visitors to the site every day with teachers signing up to hear about the next kit.

Distribution

Comps: 523
Online orders: 644
Offline orders: 283    Total: 1450 sent out
50 distributed in person

Additionally 167 people have downloaded the online version.

Schedule for kit 2

31/08/09    Ask teachers for topic suggestions
11/09/09    Email feedback reminder to teachers
14/09/09    Put up topic shortlist for voting
21/09/09    Decide topic for kit 2
………………….Put up info about it and start taking orders
………………….Make any topic-related connections necessary
28/09/09    Start research/writing kit 2
………………….Make decision about whether we produce a mailer for this kit
01/10/09    Meet w mailing house about arrangements for batch 2
05/10/09    Kit 2 to design
07/10/09    Kit 2 to printers
19/10/09    Post out 1st batch kits

Hi everyone, here’s a summary of how the March 2009 event went (pdf download). I was supposed to write a couple of pages so of course it’s 12 pages long (I’m the same with cooking – if I have a couple of friends round I make enough food for ten and we’re eating leftover curry ’til thursday…). Fortunately Shane (my boss) is busy being terribly important in London running his G20 bloggers tent, and hanging out with Bob Geldof (pictured), so he can’t tell me off about it.

shane-karina-and-st-bob

Shane (just behind Sir Bob) at the G20Voice

What we found was pretty similar to the pilot (but it’s important to check everything is still working:-)). One of my favourite quotes was the one I used as the title. Students were over and over amazed that ’scientists are just like real people’, and that they were taking time to talk to young people. They also couldn’t quite get over that we were giving them a say about something, “[I liked] being able to vote as a child and make a difference.” I find that quite moving – we all want to make a difference, don’t we? Do we give young people so few opportunities for doing that?

As for the grown-ups, all the scientists and all the teachers (who filled in the feedback survey) would take part again, and recommend the event to a colleague. Everyone enjoyed taking part, the scientists developed their communication skills and got inspired about public engagement, “I got a tremendous amount out of it, and I think I probably learnt a lot more from the students than they learnt from me!”, and the teachers felt their students benefitted enormously.

The key elements mentioned, once again, were:-

•    Exciting and intimate medium for interaction
•    Real scientists and real science
•    Taking young people seriously and giving them actual decision-making power
•    Supported by thought-provoking classroom discussion activities

But looking through all this data has confirmed for me again how important each of the elements is and how they work together.

For example, one thing that works is that live chats are an intimate and familiar medium for young people – so they can ask questions when they might normally be shy in class. But it’s the fact that they have a real decision to make about real science, primed by appropriate classroom discussion exercises, that gives some purpose to that conversation, and means it’s not just students asking what nipples are for.

We can’t claim this is entirely due to our genius as event producers (although I’m sure it’s a factor;-)), it’s also to do with the organic way I’m a Councillor, and then I’m a Scientist have developed*. We’ve seen what works and what doesn’t as we’ve gone along and built the event up. We also owe the biggest debt of gratitude to the teachers, scientists and young people who’ve helped us. Thank you everyone! I was going to talk about that lots more, but I have to go and catch a train to see my baby sister’s band play so it will have to wait for another time. Have a great weekend everyone!

*I particularly like how this theory confirms all the prejudices of my biologist worldview: Evolution, much better at engineering than engineers are.

Tags: , ,

Here is is, all 126 glorious pages of it. The I’m a Scientist Evaluation Report [pdf 1mb].

If you’re interested in how the event went and what people got out of it then look at Section 1 (findings) and Section 3 (case studies). If you’re a fellow science communication professional, and want to know more about how we did things and learn from our experience, then look at Section 2 (development) and Appendix 3 (formative evaluation findings).

The key messages were:-

Students said: “i learnt loads and feel much more confident to put my hand up and ask questions and know that ok sometimes i will get it wrong”

Teachers said: “The pupils were looking forward to their science lessons, asking about them in the corridor etc.”

Scientists said: “It engaged the kids in a way I’ve never seen before.”

Every scientist and teacher surveyed said they would recommend it to a colleague.

Key Outcomes

  • Students realised scientists are real, interesting, fun people
  • Students were inspired and enthused
  • Developed debate and discussion and How Science Works (HSW) skills

Key reasons it worked

  • Giving some power to young people gives them a reason to engage and shows that they are trusted
  • The fact that it’s real – real scientists, real science, real prize money – makes it far more vivid
  • The intimacy of the medium makes it easier to break down barriers and make connections

The key lessons for others were

  • Teachers want to use more debate and discussion for the new GCSE but feel they and/or the students don’t know how to do it. So resources which help are welcome.
  • Many existing resources to support the new GCSE feel ‘rushed out’ and don’t always ‘get’ what How Science Works is.
  • A teacher panel is a really useful way to include teachers in development of your project.
  • A project blog is a great way to be transparent and keep everyone informed.

I hope this is useful to people.

Lots of people have written back to me with comments on the draft plans for the teacher packs. Thanks everyone!

Read the rest of this entry »

Tags: , , , , , , , ,

Well we’ve been beavering away here in I’m a Scientist Towers. We’ve taken on a freelancer to develop the teacher packs. Becky Davies usually works at Techniquest Read the rest of this entry »

Tags: , , , , , , ,