What happens when you pay a surgeon’s salary based on the number of successful operations they perform? (successful = patients who survive)
Hint: they don’t become better surgeons.
They simply choose to operate on patients who they know have a higher chance of surviving, and then selectively discriminate against patients that they think won’t make it through the surgery. The result is that an unfortunate number of patients don’t get access to the surgeries they need, just because surgeons don’t think it’s worth the effort. Not all surgeons, of course, and not all the time. But this only needed to happen once for it to be an issue.
For those who are surprised, outraged even — I ask, would you rather have a surgeon who is primarily motivated by money, or someone who wants to perform the best surgery possible? When lives are at stake, we should expect professionals to be 100% competent regardless of their salaries.
Similarly, when you put pressure on kids in school to pass standardized exams, we know some of them will cheat. They cheat because of social pressure and because without good grades they won’t be able to move up. Cheating is unfair to those who study hard, but nobody is dying. When you grade doctors, the consequences can be lethal.
But the concept comes from exactly the same set of beliefs— judging a complex set of systems on a small set of standardized metrics.
A little bit of history
The history of using metrics to measure performance stems back to the mid-1800’s in Britain, where Robert Lowe, who oversaw the education committee, decided to tie the funding of schools to the performance of the students in three areas: reading, writing and math.
Critics argued that school should be used to cultivate a general level of knowledge and skills, but when the focus turns to standardized tests, it narrows the focus and ignores other subjects.
Despite the naysayers, the match had been lighted. The idea that teachers should be judged based on student grades spread to the United States, and was adapted throughout government institutions and business sectors.
In Jerry Muller’s book, The Tyranny of Metrics, he outlines five core reasons why metrics gained so much popularity:
- Numbers give the appearance of transparency.
- The West values scientific judgement over subjectivity. Numbers are objective.
- A growing distrust of government in the 60’s led to a demand for greater accountability. Numbers provided an avenue to do that.
- You need metrics to see where you can cut costs in a big company. Organizational complexity thus created demand for more measurement.
- As we developed more tools to measure (like spreadsheets), the lure to use them increased.
Draw downs of the pay for performance model
When you’re incentivizing people in jobs that have primarily repetitive tasks, like working in a factory or a transactional sales job, the carrot-stick approach of paying for performance works well as an extrinsic motivator.
But more often than not, many of us work in jobs that require creativity, and in complex organizations. When people’s promotions, salaries and reputations are on the line, these metrics can be gamed, and have unintended consequences.
Here are some disturbing ways metrics can go wrong.
Law enforcement: Police officers who are pressured by their police chiefs (who are pressured by the government) to reduce the crime rate have intentionally misclassified cases to record lesser charges, making it appear as if a crime rate isn’t as bad as it seems. A theft gets bumped down to ‘lost property.’ Major offenses become minor crimes. Even rapes get relabeled as lower offenses, or worse, go unreported. The crime rate looks like it’s decreasing, but it’s really not.
University: Rankings are given to universities based on the number of academic journals they publish and the number of times journals are cited. While there can be some merit in these, often times the result is corruption and the rise of “scientific cartels” of researchers who team up and cite each others papers, thereby increasing their rankings. Also, while we see an increase in publications, there comes a decrease in the quality of actual research.
Business: In an effort to boost profits, Wells Fargo bank encouraged its employees to cross-sell various financial products, like getting people to sign up for overdraft protection and credit cards — failure to do so meant clear penalties for employees like not receiving a bonus. This target was difficult to achieve, so employees reverted to fraud and made fake accounts for customers. Once management found out, 5,000 employees were terminated. Management’s poor choice of pay-for-performance incentives severely backfired. (*since then, the fraud has continued and the government is fining them $1 billion USD for fraudulent actions)
As the famous quip goes, “What gets measured gets managed.”
But it’s equally true to say, “That which gets measured gets gamed.”
In all of the above cases, the longer term goals of reducing crime, improving education, and growing a business were sacrificed for short term gains of gaining money or recognition.
While the intentions by higher-ups were good — transparency, objectivity, better results — the systems in which we operate are complex, made up of humans, and can rarely be reduced to a number.
There’s a Zen story recounted by Alfie Kohl, a psychologist and critic of standardized exams.
“A student asked his Zen master how long it would take to reach enlightenment. “Ten years,” the master said. But, the student persisted, what if he studied very hard? “Then 20 years,” the master responded. Surprised, the student asked how long it would take if he worked very, very hard and became the most dedicated student in the Ashram. “In that case, 30 years,” the master replied. His explanation: “If you have one eye on how close you are to achieving your goal, that leaves only one eye for your task.”
Here are a few more examples of how blinding metrics can really be.
Mindless measurement
The online media industry — while there are some promising alternatives like the Brave browser — is a classic example of metric obsession. It’s easy to take advantage of human nature; we crave the sensational, the shocking, the shareable.
Unfortunately, when you optimize for what’s shareable, you get lots of garbage. There are many media outlets, like the Huffington Post, where writers have a target to pump out four or five 500-word articles per day. It’s hard to produce something of quality when you’re emphasizing quantity (page views, likes on Facebook etc).
The impact on the reader — you and me — is wasted time. It’s what Ryan Holiday describes in his book Trust me I’m Lying, as a tendency towards “narcotizing dysfunction, when people come to mistake the busyness of the media with real knowledge, and confuse spending time consuming that with doing something.”
In another example, in his fascinating TED Talk Tristan Harris warns about mindless measurement, referring to Snapchat’s new feature Snapstreaks, which measures the consecutive number of days two people communicate with each other. The scary part is that kids actually pay a lot of attention to this number; after all, if you have a 100-day streak of Snaps with your friend John you don’t want to lose that.
When kids go on vacation, they give their phones and passwords to their friends so they can continue taking Snaps for them while they’re gone. Their friends take pictures of walls and doors, just to keep up this streak. And what for? People are valuing a number over actual conversations.
How to confuse cause and effect
One study found that people who get on average 8 hugs a day were healthier and happier. Does that mean we should start measuring the number of hugs we get, and set a “hug quota?” Sure, increasing the number of hugs in your day might give you a boost of oxytocin, momentarily.
But it’s more likely that the hugs are a “symptom” (or an effect) of a positive and rewarding social life, rather than the cause of it. Eating well, exercise, close relationships/social integration, gratitude, religion, environment, location, financial stability, meaningful work where you can find flow, etc. all influence happiness. Probably in some sort of balanced combination.
It might thus make sense to focus on the bigger picture goals in life and perhaps the frequency of hug-giving and hug-getting will increase from there. But you don’t need to measure the amount of hugs you get, nor should anyone be keeping track.
Measuring the wrong thing
Counting calories is a cult. Despite its ubiquity, it’s a poor indicator for those who want to maintain fat loss. We know that hormones play a greater role in determining this. For example, if you eat a meal that is full of protein, your body will burn fat much faster than ingesting a meal full of carbs, of equal caloric value. The interplay between how much fat, carbs and protein you consume also determines your ghrelin/leptin levels, two hormones that influence how hungry you get and thus likelihood of overeating.
People with insulin resistance — half of all Americans — will actually do worse following the traditional advice of eating a standard diet with less calories. The point is, what you eat can be more important than how much you eat. And just because you can count calories, doesn’t mean that it’s relevant. You’re better off adopting a diet that will make it easier for you to naturally lose weight and suppress your hunger (like a ketogenic diet). But if you did want to measure something, getting a blood test and checking your insulin sensitivity would be a good first step.
Is success opaque?
I was in sales for several years and pretty consistently outperformed my revenue-targets. There were a dozen or so key performance indicators (KPI’s) that I had to hit, which I followed for the first few months… and then subsequently ignored.
Historically, these numbers provided a sense of how employees were doing on the job. This made it easier for us to forecast performance for periods of weeks/months. While this is certainly true, as I got better at the job I realized this logic fell apart. How was I able to ignore these indicators myself, but still exceed in my results?
Simply put, there were factors you couldn’t measure like…
- The quality of relationships I had built which made me more effective, despite my “low” numbers.
- My intuition about certain deals (how likely I was to close the sale) that allowed me to take an 80/20 approach —being confident about where to focus my time.
- My brand in the industry that had developed from customer service + tenure. Meaning clients came to me, and I had to put in less leg-work.
Ignoring my KPI’s was seen as OK, since I was hitting my revenue target. Ironically, focusing on the main target (revenue) instead of daily targets reduced my stress and decreased my admin time, which freed up more time for actual sales activity. While I had to start with KPI’s, I eventually developed the intuition to graduate from them.
I recognize certain numbers are worth following, some indicators are better than others, and it can take time to build the “intangible” sales skills I described. But this led to me to consider that perhaps there’s less power to the numbers than was being touted — or perhaps the qualities of success were more opaque, non-measurable.
“Measure what you can, evaluate what you measure, and appreciate that you cannot measure the vast majority of what you do.”
-Ed Catmull, President of Walt Disney and Pixar
All the reasons metrics can be bad…
To summarize:
- They can cause suffering and literally kill if used improperly. (like surgeons with bad incentives)
- They are easily gamed. (like police officers who tweak their reports)
- Metrics can be poor corollaries for reality. (people are nuanced, but numbers hardly leave room for nuance)
- It’s easy to measure the wrong thing. (like calories)
- The most important things to measure are often the hardest to measure. (like sales ability)
- You can’t measure everything, nor should you. (like the value of a relationship)
What’s the alternative?
It may not seem like we have much option; that we’ve come too far on the train of technological progress, from measuring our performance at work, in our schools, all the way to our fitness trackers. Metrics have trickled down from the institutional level to the personal level with little hope for recourse.
Yuval Harari, the author of Sapiens, described our current path of technological progress as the data religion.
“If you experience something — record it. If you record something — upload it. If you upload something — share it.”
The dystopian version of this world would be like the movie The Circle, where all of our actions are recorded. But as we’ve already seen with misaligned incentives of pay-per-performance doctors, the problem has already arrived in a very toxic way.
I’m not saying we should rewind the clock of progress and throw away metrics. That isn’t feasible, nor is it desirable. Used properly, metrics can be useful tools.
Rather, we should make sure we are measuring the right things, things that matter, and challenge those who blindly follow the cult of metrics.
A few heuristics to keep in mind:
- Question what you are measuring
When everyone is counting calories, that doesn’t mean it’s the right metric. And just because you aren’t obese, doesn’t mean you don’t have diabetes (the rise of the skinny diabetic)— the scale, like calories, doesn’t tell you much. Blood sugar levels and insulin sensitivity, in this case, are more telling.
Another example: one study tracked 1900 young adults and found that those who weighed themselves every day were significantly more likely to show symptoms of depression and poor self-esteem. Basically, regular weighing may put one at risk for developing an eating disorder.
- Recognize the complexity of human nature: Intrinsic and extrinsic motivators.
Bringing money (extrinsic reward) into the equation can muddle outcomes in unexpected, and often negative ways. On the other hand, those inside of the organization, like competent managers, can improve performance if the goals are aligned with their intrinsic motivators — and if given the freedom to do so.
For example, in his book Checklist Manifesto, surgeon Atul Gawande showed how developing a simple checklist significantly decreased patient deaths in hospitals across America — and had nothing to do with pay per performance. Nurses, doctors and staff were all aligned with the common goal of saving lives, a great enough cause in itself! The key here is that some big-brother isn’t dictating what the metrics should be, rather, it came from bottom-up.
- Understand the desired outcomes and goals.
The best measure of our sales results are…our actual sales results. It can be useful to ask whether or not we need all of these metrics, and if we can eliminate some of them. Is the added overhead cost of tracking, measuring, reporting and the time it takes do so worth it? What’s the ROI and costs of going through the trouble?
If you have a general goal to be more healthy, measuring your daily steps isn’t going to actually help; you’ll likely see more success developing a good system to get you off the couch and to the gym. The point is, once we deeply assess our goals, we might find that there’s less need to measure than we thought.
- What is often most important can be the hardest to measure.
Team spirit, cross-functional communication, emotional intelligence, and leadership ability are all touted as very desirable qualities by managers all over the world. And as many managers will tell you, it’s hard to attach number on these cherished skills.
On the opposite end of the spectrum, let’s not forget about the superstar engineer who can build a great product but is an absolute jerk to everyone — what is their negative impact on morale? The “no-asshole rule” (don’t hire jerks, even if they have great skills) is much harder to quantify, but you know it when you see it.
- Accept that not everything is measurable.
Without privacy, there would be no intimacy. We can’t ignore the variance of human nature and that our lives are built upon so many nuanced moments. Perhaps the best sign of strong relationships is the quality of the time we spend; the intangible, non quantifiable feeling we get after having a great conversation, or when we’re having a hard time and are greeted open arms and ears by those closest to us. How do you measure that?
I hope that the above examples have given you some insight into the problems that come with an over-reliance, and often obsession with metrics. This affects you and me and extends to our most trusted institutions — all the way to our daily lives. While we can’t throw them all away, it’s our responsibility to respectfully challenge them when necessary.
Recommended Reading