KPIs in Education and Sport

It is exam result time, and as every year we have a strange situation.  Every year pass rates go up and (almost) every year this is seen as a sign of failure.  If the pass rate has gone up, then the standards must have gone done. 

It is hard to think of any other situation in which improvement in a key metric (pass rates) is seen as a problem.  Generally we think that if something is important, and measured, then over time the measured value will increase because there will be a focus on improvement.  This philosophy is rather at the heart of KPIs.

Of course there are complicating factors.  Newspapers tend to sell more copies when telling us that all is doomed and rubbish, rather than saying how bright and sunny things are.  This does lead them to finding some quite novel ways of highlighting the black lining in a silver cloud.

Politicians of course also have their own agendas - usually how rubbish their predecessors were and how brilliant they have been.  The spin again often has little basis in objective reality  (I note that this year the government were keen to push the idea grade inflation has stopped, when in reality more passed than last year - which in previous times was seen as a symptom of grade inflation).

One of the essential problems here is that we don't really know what we are measuring or why, and what we actually want to happen.  Are exams a way of determining the brightest and best?  Or are they about people meeting a certain standard?

If it is about meeting a standard then it is not surprising that pass rates increase over time - even if the standard itself is raised.  This is what we would normally expect in a business context - conformance increases, and output increases over time as people learn how to do their job more effectively and efficiently.  You would expect teachers to be the same.

If it is about identifying the brightest and the best, then there is no reason for pass rates to increase, and we could just be clear that it is a competition and top marks will go to the top say 5%, and so on down the bell curve.  Of course this does rather mean than only have of the students can be better than average (rather than most being so, as Michael Gove once rather optimistically targeted).  And it does mean that in a good year good but not great students will be penalised, and in a bad year poorer students will be rated higher.

Is it possible to achieve both meeting a standard and identifying the brightest and best?  Probably.  But that requires rather more sophisticated and complex measures than can be printed on the front page of a newspaper.

Which is where Sport comes in.  There is a suggestion that all athletes should publish their blood data, and Paula Radcliffe has had a few fingers pointed at her for suggesting that this is not a good idea.  No one has yet accused her of being a cheat, but there is a feeling some would like to.  I think she has a point.  The data is complex and needs considerable expertise to interpret.  My statistics are not great, but I do understand the main points that any repeated measurement has a variation in it, and that rogue measurements do occur.  Not only are 50% of measurements above the average (mean, assuming a Gaussian distribution) but 2% will be significantly above (and below) the mean - more than 2 standard deviations.  So we WILL get outliers.  Odd results.  Random high scores. 

Do you think that newspapers seeing one very odd high result will say "oh that is with the margin of error, and is probably random noise on the measurement"?  Or will they say "cheat"? 

And of course if you are a female athlete do you want your pregnancy to be announced through your drug testing results?  Particularly in the very early days when the body adapts but the pregnancy is not yet secure.

The point of both of these examples is that measurement is vital to improving performance - but we do have to be very clear about what we are trying to achieve, how the measurements will support that, and how we are going to interpret those KPIs.  It is not as simple as asking "what can we measure".

So, good luck to all those students who got their results this summer.  Your lives are ahead of you.  Go and do things better than us.

Business Ethics revisited

The Chief Executive of the Glasgow 2014 Commonwealth Games, John Scott has resigned after accepting gifts and hospitality from potential suppliers - see here.

I have no idea of the extent of these gifts, or hospitality - only one instance is referred to by the BBC article. However after the problems in Dehli in 2010 it is essential that 2014 is squeeky clean. Glasgow has had a bad image in the past for perceived public sector corruption, and 2014 will need to ensure that the highest possible standards are met in order to overcome that image. Whether it is deserved or not, is not the point. The point is that the Games are to showcase Glasgow, and to a lesser extent Scotland and the rest of the UK, and therefore need to able to come through repeated and intense scrutiny with no blemishes.

This is difficult because the process of identifying sponsors requires a great deal of "shmoozing" and relationship building to encourage the sponsors to contribute hard cash. However the line between sponsors, supporters and suppliers can be quite blurred in major sporting events - and companies move from being one to being another quite often.

Whatever the rights and wrongs of this particular case it was right for John Scott to go - regardless of his past contribution to the Games, in the future he media would not let go of this incident, and would concentrate on identifying other potential wrong doing. The contagion has to stop at the first point.

The fact that with a year to go before the London 2012 Olympics we are not seeing this sort of story about them is testiment to their hard work in establishing and maintaining standards of business ethics. But many a slip between cup and lip.

Basics of Business - Ethics Part 2

I had the interesting experience of running a course in Qatar duing the FIFA election debacle, in which one candidate Sheikh Bin Hamman of Qatar was suspended ensuring another unopposed election for Sepp Blatter.

The whole election, and previously the award of the World Cup to Russia in 2018 and Qatar in 2022, was mired in allegations of corruption, brown envelopes and people putting their own interests ahead of the organisation. Qatar was accused of "buying" the World Cup, though the precise meaning of that phrase was disputed.

In the UK we watched with rightous indignation, and a certain fury, as we saw evidence that our bid for the World Cup in 2018 had been ignored by FIFA because the FA had refused to offer "sweetners". This may have been the case - certainly there had been accusations of corruption at FIFA for many years, and surely the FA were aware of that before the fact.

It may be salutary though, as we pride ourselves on our incorruptable British nature, that Qatar actually ranks above the UK (i.e. better) in the Transparency International Perceptions of Corruption index - see here. Only one rank, but still - Qatar has been improving its score, whilst the UK has been showing considerable decline with the score (out of 10) declining from 8.3 in 2003 to 7.6 in 2010.

This index looks at perceptions of the public sector, and as always you can argue with the methodology and scoring - and the UK is still 20 out of 186 nations, and above the USA (22nd). However it is a worrying trend, and if continued could lead to an erosion of trust in the public sector and public sector procurement in particular. Which is not in our national interest.

Unusual Key Performance Metrics

The writer Terry Pratchett has a word – Substition. It is the opposite of superstition. Superstition is things that everyone thinks work, but in reality don’t – like crossing your fingers for luck. Substition is things that no one thinks work, but in reality do.

I’ve just been reading Moneyball by Michael Lewis, which to me is about substation. He writes about a baseball team, the Oakland A’s, who in the early part of this century were successful on a very modest budget because they had developed metrics for measuring the worth of a player that were different to everyone else’s. Rather than focussing on the outputs, they were focussing on the process. Players who did the right things at affordable prices, rather than the stars. Rather than RBIs (runs batted in) they evaluated players on their ability to get to first base (i.e. not get out). I suppose in football terms it is like picking players who have a good record of winning corners or drawing fouls rather than scoring goals – the goals will come from the corners if you get enough of them. In cricket maybe it would be bowlers who bowl very few no balls in 20-20 rather than take a lot of wickets.

Terry Pratchett also coined the word “narrativium” for the fact that sometimes the world seems to be driven by a good story rather than anything else. People love a good story, and in business and in sport, we are sometimes guilty of wanting to believe the story rather than the facts. Indeed in Moneyball there are times when the “good story” element dominates over the business lessons.

I’ve read a couple of Michael Lewis’s books – Liars Poker, the New New Thing – and he is always extremely readable although you should always remember that he is writing as a journalist rather than an academic. His books make you see the world in a slightly different way, and make you think about what else you might be missing.

In this case it has made me think a couple of things about business in general. Firstly, I wonder whether there might be novel metrics in business that we could use that would better track business success rather than the traditional ones of turnover, return on assets. Rather than customer satisfaction, perhaps ease of doing business? I don’t know. If anyone has any ideas, I would be most interested. The key point from Moneyball is that these should be measurable so that they can be proven, rather than just a “gut feel”. My first thought is that there is a huge (and sometimes contradictory) set of knowledge held by Business Schools and Academia, that most small businesses just don’t know about. How do we get it out there to where it can be used?

The other thought was to wonder about the superstitions and substitions of business. What are they? What are the things that work, though no one thinks they are important, and what are the things that everyone does but don’t actually work? Again, any thoughts and ideas gratefully received.

Rio 2016

Hard to believe that it is more than 4 years since London won the right to stage the 2012 Olympics. What a thrill it was, coupled with a certain Schadenfreude at the French disappointment. London 2012 made a great pitch - pointing out the UK is such as young country, and that London is one of the great international cities. Maybe only New York has that same buzz. And now less than 3 years until the Games themselves.

Although it may have been easier for UK companies to get business in Madrid or Chicago because of proximity or a common language, there should be lots of opportunities in Rio. The fact that this is the first Olympics in South America, means that there will not be the level of established infrastructure and preferred suppliers that there is in Europe and North America.
Obviously some of that will develop during the World cup of 2014 - but the page will be blanker than it is for most Games, and UK companies should have the momentum coming out of 2012 to do well.

Delivering the London 2012 Olympic Games

We are luckily enough to be involved in helping companies look at supplying the London 2012 and other major events - details when possible.
In the meantime, here is a link to YouTube from the CIPS website, showing one of the showcase speeches at the 2008 conference by John Armitt, Chairman of the Olympic Delivery Authority. In case you were in doubt about the role of the ODA, it is often said that they build theatre and LOCOG put on the show - which indicates the difference in the size of the budgets.

Beijing 2008

A couple of weeks before Beijing 2008 I had the pleasure of attending a seminar given by Prof. Steve Haake of Sheffield Hallam University about Newton, Physics and Sports Science. It was one of the few events I have been to where the whole audience would have happily stayed twice as long.

During the presentation he talked about work his Sports Engineering Group had carried out for Team GB cycling ahead of Beijing 2008. Without giving too much away, he told us that every single aspect of the bikes had been studied - and to demonstrate it showed us graphics of the calculated airflow over the nuts holding the peddles onto the bikes.

The value of this phenomenal level of detail in preparation was demonstrated as the cycling team romped to 6 Gold medals out of 10 during the Olympics.

It made me think about what we in business could learn from this attention to creating the environment in which our team members can perform - and know that there is no excuse for not performing to the highest level. I suspect that very few companies could claim to do much more than getting of their employees way, rather than actually giving them all the tools to do the job and resolving any problems getting in their way.

Food for thought...

Survey of Yorkshire Sports Businesses

We are carrying out a survey of R&D in Sports Businesses in Yorkshire. If you work for one, and would like to take part please click on the link below.

Many thanks

Click Here to take survey

London 2012

There is a new website listing all the contracts available relating to the London 2012 Olympics bid www.competefor.com

In theory not only the lead contracts will be publicised here, but sub-contracting arrangements from all suppliers should also be listed. How well this will work in practice, we will have to wait and see - but the people at the ODA I have spoken to are clearly committed to using the opportunity of London 2012 to improve some of the standard procurement processes in the UK public sector (particularly in construction).