Rule #2 of business: it's more complicated than you think (or want it to be)

Recently the culture has been elevating "evidence based" as an adjective. It's a great goal to make decisions based on evidence rather than conventional wisdom or gut feel. We would like to know the truth, and act based on that truth. The problem is that the truth is often complicated and sometimes uncertain, so we settle on a small list of measures or even a single measure to determine what a good outcome will look like. This can be a trap. Let me give you some examples.

Example 1: LOC as a measure of software developers productivity

I manage a team of software developers. I'd like to know if that team is being efficient and productive. How can I measure that? In the past some organizations have looked to Lines of Code (LOC) as a measure of productivity. The reasoning being that if one developer writes more lines of code than another then she's been more productive. The problem is that not all lines of code are created equal. A highly efficient developer can often solve a given problem with fewer lines of code than a less efficient developer. Compound this with the fact that the larger the program the greater the maintenance burden of bug fixes and feature additions to that code base, so maybe a lot of lines of code isn't even desirable. 

So why did those manager choose LOC as a measure of productivity? Because it was easy to measure and easy to understand, and that is the trap. Looking for a single measure, and managing based on that measure is to ignore the complexity of the situation. 

Example 2: Standardized testing as a measure of educational success

When the government wanted to have a way to measure the quality of public education they reached for a measuring stick. And the stick they found was--standardized testing. Standardized testing has been around for decades. I can remember taking Iowa Basics tests as an elementary school student in the 1970's. Articles comparing the success of various educational systems have often used Standardized test scores to make value judgments about the outcomes of those system. But educators will tell you that at best standardized test scores are only one measure of a student's academic success. They ignore the many real goals we have for our schools: preparation for career, preparation to participate in our democracy, exposure to art, literature, other cultures, problem solving skills, and how to get along with other people. 

So why did the government choose standardized testing as a measure of the effectiveness of our schools? Once again, because it was easy to measure and easy to understand. And once again managing the situation based on that single measure has caused some bad decisions to be made, both on a micro and macro level. 

So what?

Why should we care? If poor, simplistic metrics are used to measure things, it can drive poor decision making. Depending on the incentives of meeting or improving those measures all kinds of bad things can happen. Imagine the first scenario where a manager decides to promote or compensate developers based on the number of lines of code they produce. Will the developers produce more lines of code? You bet they will. Will those lines of code produce a better program? It is doubtful. In the second case, we've seen and are seeing what happens when schools (and teachers) are measured on the basis of the test scores of their students. The very best outcome of this is that we have hours of instructional time being spent to "prepare" students for the test. In some pathologically bad situations, we have teachers and administrators actively subverting the test by helping their students to cheat. 

The real "so what?"

At this point you might be saying, "Great, I don't use any of those obvious false metrics to measure what I, or my team does." The fact is that you probably do. Take a look at the metrics you use to measure success. Ensure that those measures really do drive the outcomes you're looking for. And don't just use one metric, use a variety of metrics in various areas. Especially choose metrics which might be affected negatively by positive movement in other metrics. For example, if you were designing an engine, you might want to measure fuel economy, horsepower, and emissions. If you are measuring a software development effort, you might measure bugs found in production, development lead time and number of features developed over time. If you're measuring education, you might measure sense of wellbeing, success at vocation and success at higher education. These are just examples; you should develop your own metrics with extreme care and attention.

TL;DR:

Drucker said "you can't manage what you can't measure." Make sure you're managing the right things, by measuring the right things.