Measurement of human performance at work, has since times immemorial, been a crucial process to ensure that organisational goals and objectives are attained at optimal costs. Metrics have always played an important role towards this aim. With availability of high-power compute at the office desk, managers, executives, bureaucrats, and politicians have gone overboard with measuring leading to ‘metric fixation’. This book by Jerry Z Muller focusses on the issue of ‘metric fixation’. It is a well-researched, evidence-based book on the overuse / incorrect use of metrics in all spheres of life clearly demonstrating the limitations of quantification.
The book is logically organised in multiple parts where in the first few chapters he lays out his ‘argument in a nutshell’ followed by the origins of metrics and its fixation. A substantial part of the book comprises case studies that brings out various metrics, their successes, and shortcomings in the fields of K-12 education, higher education, medicine, policing, military, business philanthropy and foreign aid. In the last section, the author lists out the various unintended negative consequences of metric fixation and suggests some guidelines to use metrics sensibly. The book uses most examples from the US ecosystem and may not relate directly in all aspects and measures to our Indian context, however at the conceptual level it is relevant to us too.
Most organisations today believe that the road to success is governed by quantification of various attributes. The idea that there exists a relationship between “measurement” and “improvement”. “If you cannot measure it, you cannot improve it.” or Tom Peters, motto, “What gets measured gets done, which became the cornerstone belief of metrics. Believers in metrics advocate “accountability” through measurement and assume that only by measuring can institutions be truly responsible. Performance is therefore equated with what can be reduced to standardized measurements. The result as per the author is the demand for ever more documentation, ever more mission statements, ever more “goal-setting. Muller lists out three key components of metric fixation – (a) the belief that it is possible and desirable to replace judgment, acquired by personal experience and talent, with numerical indicators of comparative performance based upon standardized data (metrics); (b) the belief that making such metrics public (transparent) assures that institutions are actually carrying out their purposes (accountability); (c) the belief that the best way to motivate people within these organizations is by attaching rewards and penalties to their measured performance, rewards that are either monetary (pay-for-performance) or reputational (rankings). This according to him has resulted in the “tyranny of metrics” that threatens most institutions. The authors main argument is “Not everything that can be counted counts, and not everything that counts can be counted.”
He goes on to support his thesis using various examples and cites works by American social psychologist Donald T. Campbell who held that “the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor”. He also cites the Goodhart’s Law, which states, “Any measure used for control is unreliable.” To put it another way, anything that can be measured and rewarded will be gamed.
Recurring Flaws is a chapter in the book devoted to the most frequent types of flaws in the use of performance metrics. Metric dysfunction leads to distortion of information and some problems that arise from it are: (a) Measuring the most easily measurable – results in rarely measuring what is the most important and sometimes not important at all. (b) Measuring inputs rather than outcomes; (c) Gaming through creaming – simpler targets or prefer clients with less challenging circumstances, making it easier to reach the metric goal, but excluding cases where success is more difficult to achieve. Later in the book he cites various examples to support this issue where doctors who tweak / maintain high ratings by not taking on complex cases or hospitals who tweak survival rate of patients. (d) Improving numbers by lowering standards – improving metric scores by lowering the criteria for scoring. Thus, for example, graduation rates of high schools and colleges can be increased by lowering the standards for passing. (e) Improving numbers through omission or distortion of data – this strategy involves leaving out inconvenient instances or classifying cases in a way that makes them disappear from the metrics. Police forces can “reduce” crime rates by booking felonies as misdemeanours, or by deciding not to book reported crimes at all. (f) Cheating – Muller cites how the No Child Left Behind Act raised the stakes for schools of the test scores of their pupils where teachers and principals in many cities responded by altering students’ answers on the test.
Measurement demands judgment: judgment about whether to measure, what to measure, how to evaluate the significance of what is been measured, whether rewards and penalties will be attached to the results, and to whom to make the measurements available. One can agree that many metrics are misconceived, obsolete, counter-productive, and to some extent stupid also. Issues mentioned in “Recurring Flaws’ are valid, and people do find their way around any method to control their performance. The examples cited in the book are real and do merit careful consideration of the metric as well as the workaround people have invented to bypass them. The author has a chapter devoted to “When and How to Use Metrics” and gives out a Checklist – a list of questions people involved in policy making should ask and factors one should keep in mind in considering whether to use measured performance, and if so, how to use it.
The author believes that there is no silver bullet, no substitute for actually knowing one’s subject and one’s organization, which is a matter of both experience and of unquantifiable skill. Many matters of importance are subject to judgment and interpretation and cannot be solved by standardized metrics alone. Muller feels that the issue is not of metrics versus judgment, but knowing how much weight to give to metrics, recognizing their characteristic distortions, and appreciating what cannot be measured. This as per him has been lost sight of by politicians, business leaders, policymakers, and academic officials.
The book is an insightful read for corporate and social policy makers who deal with metrics / the obsession with metrics within their organisations. This book is not about the “evils of measuring,” he writes. “It is about the unintended consequences of trying to substitute standardized measures of performance for personal judgment based on experience. The problem is not measurement, but excessive measurement and inappropriate measurement—not metrics but metric fixation.”. The research he cites, and the examples he chooses, are compelling.
Muller sums up the problem nicely. “There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge—knowledge that seems solid but is actually deceptive.“
