What gets measured gets gamed

measure-upOKR (Objectives & Key Results) are an implementation of Peter Druker’s “Management by Objectives.” as described in his 1954 book, The Practice of Management.  Roughly, every department/team/person is set a handful of measurable targets quarterly or so. There is transparency across the whole company – both about what the targets are and to what extent the previous targets were met.

OKR have a stamp of cool because Google use them. Google is wildly successful. Google use OKRs. So, if I use OKRs, I will be successful too. Well…

Something else is going on. Recent technological change has cause an explosion in the amount of data collected about all sorts of things. Also the timeliness/accuracy/etc… in which it is collected has improved dramatically. I can change a button on my website for 1% of my site visitors and see straight away whether they click on it. I can work out precisely how many extra sales I will get (or lose) from making this change permanent for all visitors. I can quantify my sleep patterns to the extent that this determines when my alarm goes off. I know in advance if the traffic will be busy up ahead.

As a result there has been a huge lurch towards data-driven everything. Data can solve everything, didn’t you know? Measure it! Measure it! Measure it!  Give teams and individuals measurable goals.

Hold your horses! Edward Deming’s cautionary tones resonate with me. Deming sees this kind of thinking as “an attempt to manage without knowledge of what to do.” The focus of the organisation becomes on outcomes without looking into the processes that produce them. Deming estimates that the system that people work in and the interaction with other people accounts for 90 or 95 percent of performance. Individuals really have very little control over what really determines their performance. As a result, most of the energy of the organisation goes into gaming the metrics rather than the real hard work of improving processes. Improving processes usually takes time and requires collaboration with a large number of people across the organisation. It’s really tough. If any actual change is achieved, this will be to make the visible metrics look good at the expense of the invisible and difficult to measure – it is usually some kind of quality which suffers which will hurt the organisation in the long-term.

Deming  alsomakes an interesting point about targets. A process is either predictable (“in control” in Deming’s terms), in which case there is no point setting goals – you will get what you will get. Or a process is unpredictable – in which case there is no knowing what you might get – so again, there is no point setting goals. In a corporate context, most meaningful activities will be complex enough to be unpredictable. What is an appropriate goal then? Who knows? If true performance is really unpredictable, how is it that some people can fairly consistently hit their targets and other not? We are creating a system to select for particular behaviours – an ability to find areas where it is easy to take credit for quick measurable progress, an ability to negotiate targets down, an ability to rationalise and explain why failure is somebody else’s fault. People who master these behaviours will thrive. Does this really make for a high performing organisation?

Is there anything good about the OKR process? Yes! The alignment that results from the initial conversations about what we would ideally like to measure and improve if we could. What is important? What would we like to improve next if we could? This is great! The wheels come off in the second stage of the OKR process, where we reject all the things we can’t measure and focus down on a handful of metrics and set an individual or a team a target for the quarter/year.

To be clear, Deming himself wasn’t anti-data. In fact, he coined the immortal phrase; “In God we Trust, all others bring data.” He was just alive to its limitations. He believed data should act as the foundation for decision-making, not as a substitute for judgement. He believed that the most important figures for management of any organisation are unknown and unknowable. An unprovable assertion. Does it square with your experience? It does with mine.

There will be a backlash sooner or later against this swing towards the idea that data can solve everything. Mark my words, the Oracle has spoken!

More about Deming vs. Drucker at:

Deming’s Point 11.b of 14 – Deming versus Drucker

Leave a Reply