It is hard to think of any other situation in which improvement in a key metric (pass rates) is seen as a problem. Generally we think that if something is important, and measured, then over time the measured value will increase because there will be a focus on improvement. This philosophy is rather at the heart of KPIs.
Of course there are complicating factors. Newspapers tend to sell more copies when telling us that all is doomed and rubbish, rather than saying how bright and sunny things are. This does lead them to finding some quite novel ways of highlighting the black lining in a silver cloud.
Politicians of course also have their own agendas - usually how rubbish their predecessors were and how brilliant they have been. The spin again often has little basis in objective reality (I note that this year the government were keen to push the idea grade inflation has stopped, when in reality more passed than last year - which in previous times was seen as a symptom of grade inflation).
One of the essential problems here is that we don't really know what we are measuring or why, and what we actually want to happen. Are exams a way of determining the brightest and best? Or are they about people meeting a certain standard?
If it is about meeting a standard then it is not surprising that pass rates increase over time - even if the standard itself is raised. This is what we would normally expect in a business context - conformance increases, and output increases over time as people learn how to do their job more effectively and efficiently. You would expect teachers to be the same.
If it is about identifying the brightest and the best, then there is no reason for pass rates to increase, and we could just be clear that it is a competition and top marks will go to the top say 5%, and so on down the bell curve. Of course this does rather mean than only have of the students can be better than average (rather than most being so, as Michael Gove once rather optimistically targeted). And it does mean that in a good year good but not great students will be penalised, and in a bad year poorer students will be rated higher.
Is it possible to achieve both meeting a standard and identifying the brightest and best? Probably. But that requires rather more sophisticated and complex measures than can be printed on the front page of a newspaper.
Which is where Sport comes in. There is a suggestion that all athletes should publish their blood data, and Paula Radcliffe has had a few fingers pointed at her for suggesting that this is not a good idea. No one has yet accused her of being a cheat, but there is a feeling some would like to. I think she has a point. The data is complex and needs considerable expertise to interpret. My statistics are not great, but I do understand the main points that any repeated measurement has a variation in it, and that rogue measurements do occur. Not only are 50% of measurements above the average (mean, assuming a Gaussian distribution) but 2% will be significantly above (and below) the mean - more than 2 standard deviations. So we WILL get outliers. Odd results. Random high scores.
Do you think that newspapers seeing one very odd high result will say "oh that is with the margin of error, and is probably random noise on the measurement"? Or will they say "cheat"?
And of course if you are a female athlete do you want your pregnancy to be announced through your drug testing results? Particularly in the very early days when the body adapts but the pregnancy is not yet secure.
The point of both of these examples is that measurement is vital to improving performance - but we do have to be very clear about what we are trying to achieve, how the measurements will support that, and how we are going to interpret those KPIs. It is not as simple as asking "what can we measure".
So, good luck to all those students who got their results this summer. Your lives are ahead of you. Go and do things better than us.
Posts
