Data vs. Experts: Nine Years On
Posted by Mark Bennett on December 1, 2012
The war between Big Data and Experts rages on, but is it really as much a fight to the death as people make it out to be? What does it mean for leaders who must choose which to consult when making important decisions?
The most recent battle in the war of course was the Great 2012 Presidential Election, where pundits, posing as “experts”, were soundly defeated and fled the field in the face of incredibly accurate predictions made by data analysts. The most often cited reason: pundits couldn’t look past their own biases to see what the data was indicating even if they had chosen to look at it, which many didn’t. And even if they did look at data, they tended to focus only on the data that fit their preconceptions.
“Actually, fifty-one. I don’t know why I lied just then.”
Nine years ago, “Moneyball” told the story of another battle that took place in the data-rich sport of baseball. It showed again how experts, this time portrayed by major league scouts, were misled by their “conventional wisdom,” while scrappy Jonah Hill quants ran circles around them and picked up real talent at rock bottom prices to win division pennants.
So is the conclusion that experts are doomed to extinction, like some kind of modern dinosaurs, while nimble data analytics outwit them like little proto-mammals? Not so much.
What smart leaders are doing is learning how to combine the strengths of each to make better decisions and to make better predictions.
In his terrific “The Signal and the Noise,” Nate Silver points out that in the aftermath of Moneyball, where one would have assumed the payrolls of scouts to be slashed in MLB teams (at least in the Oakland A’s right? I mean, Brad Pitt tore them apart!) But no, in fact, the number of scouts have generally increased.
We can rebuild them; we have the technology. We can make them better than before.
Why? Because the scouts are able to be even more effective because of the advances made in analyzing data about players, fields, etc. They are working off of a more accurate model of what matters when measuring the potential of a player long before they are brought up from the minors. The amount of data being collected in baseball is mind-boggling. They are close to being able to record all the data from every game using cameras, sensors and what-not so that you could basically re-create and analyze every aspect of the game; every bounce the ball made, every slip of a player’s foot, etc. Big Data for the Big League.
But they know that’s still not enough to know the value each player could bring to a team. Remember, it’s not enough to pay a player for what they did – that’s rearview mirror thinking. You will likely overpay with that kind of thinking over the long term. What you want to be good at is predicting performance and underpaying today for big value down the road (you hope – if it was certain, everybody would be already doing it.)
Silver describes how it’s those hard-to-quantify attributes where scouts will likely put more focus. An example is a player’s “mental toolkit.” Baseball life is rife with slumps of varying degrees and some players can do much better than others to get through them. Sure, maybe one day they’ll stick a band around a rookie’s head and a computer will spit out either, “Loser” or “Hall of Famer”, but that’s a ways off.
“When your enemy’s making mistakes, don’t interrupt him.”
So, what’s the advice on how to combine Data and Experts? It’s basically this: avoid having to trust your gut. Get some solid, objective data analysis (see my previous post) to set the foundation for your decision-making. But, since the data can’t tell you the whole story except in the most generalized, easy-to-be-replicated by your competition way, bring in your experts at that point to fill in the missing intangibles that help you make the right decision or come up with the most likely predictions for scenario planning.
In short, keep your experts from getting too misled by their biases and blind spots and then have them use their experience and imagination (yes, really) to come up with new ways to look at the problem. Data Analysis’s strength is in its rigorous and disciplined pursuit of objective truth; Expertise’s strength is in asking “what might happen if…”
Photo by GANDALF_GREY