If you’re pursuing any goal, quantifying it and measuring regularly is a good way to get there in incremental steps, whether it’s paying off debt, getting in shape or mastering a subject.
Statistics have also helped public policy in innumerable ways. One of the earliest was tracking outbreaks of disease in Victorian London to identify the cause. Others include determining the efficacy of medical treatments, supplying armies and setting interest rates.
However, this focus on measurement can also lead us down the garden path.
What gets measured, gets overmanaged
If we have figures on any issue that people care about, that will naturally make us more inclined to care about that thing, especially if those figures are prominently published and widely discussed.
This can be a good thing. The nightly news death tallies of modern wars put a cap on developed countries’ participation in distant conflicts. The road toll reminds us not to drive drunk. High unemployment figures put pressure on governments to Do Something.
However, any such statistics need context. The daily Covid infection and death numbers caused a lot of panic that would have eased had people known that most of the infections don’t matter, and that most of the deaths were among those already unwell.
A number only means something once you compare it to other numbers. To hear that a man is six foot tall means nothing if you don’t know how tall any other man is.
Further, improved data collection has reduced many policy debates to ‘which approach would cause more deaths,’ as though the only purpose of public policy is to maximize years of life.
Another example of statistics leading us astray is standardized tests in schools. These are potentially useful numbers, but in many countries they are obsessed over at the expense of things like extracurricular activities, sport and even recess. ‘One number to rule them all.’
On the other hand, I reject the common assertion that GDP is overused as a measure of a country’s progress. It’s not perfect but it is pretty good, and I think overall GDP is used about the right amount, and is considered in light of other important measures.
What gets managed, gets mismeasured
Once any statistic becomes politically significant, there is an incentive for people to massage those numbers.
This happened in both directions during Covid, with some pumping the numbers to get more funding while others minimized them to make their policies look more efficacious than they were.
Most governments have learned how to hide the true unemployment rate. In Australia, you’re not counted as unemployed if you have completed one hour of paid work over the previous week, or any unpaid work for a family business, or if you haven’t actively looked for work, or if you’re not available for immediate start.
If the unemployment rate simply meant ‘anyone who wants 30+ more hours of work per week,’ the stats would look much worse.
Standardized testing in schools, especially where it is tied to funding, creates obvious incentives for chicanery. Some common types of misbehaviour were parodied in the Simpsons episode, How the Test Was Won.
No number is neutral. To understand it, you must know who gathered the data and what their incentives were.
What can’t be measured, doesn’t get managed
Kids learn a lot of different things at school, but only some of those get graded. Parents will have a clear indication if their child is struggling with maths, for example, but there’s no simple number showing that they’re socially awkward. Not many schools give students a percentage on how well they make others feel reassured or back up their mates in a stoush.
You have to be very, very smart to succeed despite having poor social skills. In contrast, if you’re Mr Popular but as thick as two bricks, most likely you’ll do fine.
The fact that it’s easy to measure academic performance makes us focus too much on that outcome, to the expense of other important outcomes.
Returning to Covid, it was hard to fight back against extreme policies because you can’t quantify goods like freedom and joy. There’s no nightly ‘kids having fun playing in the park’ tally on the news. That doesn’t make it less important than how many people got Covid.
There are some measures of freedom and happiness. These tend to be clunky.
The Freedom House report appears to be a US State Department project and anyway, it doesn’t measure such pleasant liberties as being able to drink in the park or ride a bicycle without a helmet.
The World Happiness Report stumbles on cultural differences. Japanese will generally rank themselves about 5 out of 10 on any survey because they feel the need to conform and be average, where as Australians will rank their happiness as a 7 out of 10 even if they’re unhappy because they feel like they have no good excuse for being a misery guts.
No matter how hard you try, some things evade measurement.
What gets measured, gets boring
Whether you agree with my opinions on how important various stats are or not, you’ll concede that once these stats have been displayed for a long time, we become accustomed to them and they seem normal and therefore unimportant.
This eventually happened with Covid. In some countries, virus deaths reached their peak in 2023 but were ignored because everyone was over it.
This also happened with deaths from terrorist attacks in post-war Iraq. With bomb after bomb after bomb, we soon put the whole country down as ‘catastrophe’ and stopped paying attention to the specifics.
One death is a tragedy; a million deaths are a statistic.
Old Joe
We know there are huge numbers of deaths from heart disease, but few of us make lifestyle changes because we’re used to those statistics and we take it as natural. The same goes for the obesity rate. Everybody’s fat now. Oh well. We accept these as a set reality, like the background hum of traffic or mild air pollution, and we focus on other things.
Conclusion
A scientific approach to policy has led to improvements, but can also cause problems.
When confronted with statistics, we need to remember that any given number does not necessarily entail the implementation of any particular policy. We get to discuss that.
Any stats that become prominent in those discussions will thereby be compromised and require close scrutiny.
We must not be afraid to champion unquantifiable goods against quantifiable ones.
Finally, we must not fall into the opposite trap of ignoring stats just because we’ve gotten used to them.
Numbers need interpretation, and no one has a monopoly on that.
SovietMen 
Goodhart’s Law says that when a statistical measurement is used to set policy or measure success, it ceases to be a good measurement.
An example of using test scores to measure teacher performance. As soon as this is set in place, teachers will be incentivized to focus teaching on test scores and neglect overall learning.
Another was body count in Vietnam. As soon as body count was used as a measure of success, commanders were incentivized to maximize dead bodies, whether or not they were sure the bodies were of enemies.
LikeLiked by 2 people
I forgot to say – Excellent post!
LikeLike
I’ve been musing over an observation (I can’t remember who made it) that moderns attempt to quantify everything, while Medievals tended to quality everything. We have far too much math and, after the model of hard sciences and their successes, conclude everything must be a number. The Medievals, on the other hand, did not think number had any necessary connection with reality. Ptolemy’s and Copernicus’s maths both worked, but that didn’t mean either of them were necessarily true. But qualities adhere to the substance of a person.
Education being my thing, I’m down with the Medievals: being educated is a quality of the soul, easily recognized by other educated people. The idea that education is a number – grades, SAT score, IQ or any numeric measure – would, I think, have baffled them.
Severian once proposed that, in lieu of grading, he could simply take a walk across the college quad and have a chat with his students, and determine if they understood what he had been teaching, and that that method would yield results every bit as objective and more accurate than all the grading rigmarole.
LikeLiked by 3 people
That is how they used to assess students, back when there were few enough of them to do it that way. They’d sit down and chat about the topic. Post-war mass education and computers led to fill-the-bubble, multiple choice exams, which limit answers and encourage overtesting.
LikeLiked by 1 person
“Returning to Covid, it was hard to fight back against extreme policies because you can’t quantify goods like freedom and joy. There’s no nightly ‘kids having fun playing in the park’ tally on the news.”
The only news that’s really important – all of what we see and hear is calculated to generate desired reactions/results, when all most people want to know is “are the kids having fun playing in the park?”
LikeLiked by 2 people
There’s this repeatedly told story about the planned economy of the Soviet Union, where plant managers had to fulfill their plan or get fired. If the measurement was to get X million screws produced per year, but the raw materials didn’t come in (as was often the case), smaller screws were produced, despite nobody needing them, just to get the numbers up to X million. But this is a very common problem with target agreements in business. Typical management practice now is to have a multi-factor analysis and agreements with conditions to be met. Sadly most politicians never even worked for a living, never were managers, never produced anything, but know how to win public opinion with single talking points.
Math, statistics, measurements are tools we can use. As with any tool, it can be used for bad ends.
LikeLiked by 1 person
In that case, it wasn’t the “make X screws” part that was bad, but the “or get fired” part. We have a culture, especially here in America, of expecting to make the impossible happen, or else.
I had a buddy that worked at an electronics store near a major Army base, circa 2003. Before deployment started, his department was #1 in the company because all those future deployees were stocking up on laptops, camcorders, etc. to take with them to Iraq. As soon as they deployed, sales dropped to the basement. My friend was told to get his department back up to #1 within two months or be fired. No understanding at all of why he was previously #1, just make it happen. He chose to go elsewhere.
LikeLike
The reasoning behind not including people who are not looking for or actively working in the unemployment rate is that the statistic is based upon the concept of then civilian labor force or even more specifically the civilian, non farm labor force. So adults who were in military service or worked on farms were excluded. But so were adults who were incarcerated, who were engaged in education or church service, the idle rich and most importantly stay at home mothers.
Basically, it was supposed to measure only the industrial labor force, which was a small fraction of the total adult population up to WWII.
Countries have just stuck with it out of habit, even though it is essentially meaningless in a post industrial modern economy. Especially as it now includes government wmployees at all levels – who are clearly not in a private sector industrial labor force. the US there are alternate measurements that include people who say they want to work but are not actively looking for employment.
LikeLike
Excellent post (because it agrees with my worldview). But seriously I agree about GDP and the person capita version. Not perfect but a decent steer.
LikeLike