Why the Eye Test Keeps Lying to Business Owners

Trusting what you see and what you feel can only go so far. Sometimes you need data to support that.

There’s a great scene in the movie Moneyball where all of the scouts are around the table discussing some baseball prospects with their boss, Billy Beane. It’s a bunch of old men saying things like, “You can really hear the baseball pop off his bat,” or “That guy looks like a baseball player.” It’s a cinematic dramatization, with the idea being to portray the “old guard” as out of touch with the new way of doing things.

But here’s the thing: I spent a decade deep in the hockey world, working in the media, working inside an organization, and regularly communicating with some of the heavyweights in the hockey industry. That scene in Moneyball was not too far off.

In all sports, but especially one as conservative as hockey, there has always been a battle between what’s known as the “Eye Test” and data/analytics. The eye test is more about anecdotal evidence, while analytics use hard data to try and show what’s actually going on. From the data perspective, it’s hard to argue with facts, yet many people do. “Facts are stubborn things,” John Adams once said, “And whatever may be our wishes, our inclinations, or the dictums of our passions, they cannot alter the state of facts and evidence.”

Yet, I experienced firsthand members of hockey’s old boys club doing exactly that. They often said things like “He’s a hockey guy.” That term, “hockey guy,” was constantly spoken to describe someone on the inside. If you didn’t have the connections or you didn’t look the part, you were just another schmo. But if you did have those connections, and if you did look the part, you were a “hockey guy.” Frankly, it didn’t matter how talented you were, just so long as you were a “hockey guy.” It was one of those unspoken rules in the hockey world, one that continues to have a stranglehold on the sport. It’s why it was one of the last sports to truly embrace analytics and data as a means of decision-making.

Source: Medium

When advanced analytics first crept their way into hockey, much of the older generation fought back. Meanwhile, the younger generation tried to chastise those who were unwilling to utilize hard data. As someone who is now nearing middle age,¹ I tend to favor a hybrid model, taking a little from each side of the aisle.

I’ve found this debate to be a big help in running a small business. The eye test is crucial, because there are always things that occur that may not show up in the data. But at the same time, you can just about always find a piece of data to support or reject a hypothesis.

Think about this: you can walk into a store and see hustle and bustle – a ton of customers lined up, a loud atmosphere, and a buzz of excitement. Great! But does the data show that it is translating into sales? Does it show that those sales are profitable? Does it show that the store is making the most use of its space? To me, the eye test is a means of telling you where in the data to look. And the opposite is true as well: I often will see something in the data that surprises me, which points me in a different direction for my own observation.

Anecdotal data is incredibly convincing. When three of your friends tell you they tried a new restaurant and it was terrible, you will probably never try that restaurant. But the Google rating may be 4.8, or it may have a James Beard nomination, or something of the ilk. In this case, the anecdotal data, or the eye test, doesn’t jive with the actual data. Something’s got to give.

The downside of missing out on a potentially great restaurant isn’t too bad. But the risk of mis-diagnosing a business issue can be catastrophic. You can watch a segment of your business operate for a few minutes and catch an issue that would never have shown up in the data. Consequently, you can pore through the data and find a nugget that you would never have considered while overseeing your company.

My strategy has always been as such: I love hearing people’s experience, listening to anecdotal data, and knowing what they saw. But my reply is almost always, “That’s interesting. Can you find the data to support that?” If they can’t, it doesn’t necessarily mean they were wrong. But if an idea can pass both the eye test and the data test, then it’s pretty much a slam dunk.

A word to the wise, though. Beware of confirmation bias. A lot of people, when forced to confront the other side, begin cherry-picking anything that supports their belief. If you force a data person to do the eye test, they can very easily weed through ten examples to find just one that supports their belief. Likewise, if you force an eye test person to look at data, they will find a single example to “prove” themselves correct. If you believe there is a particular item that is selling great, you have not proven yourself by picking out a single transaction where that item was bought. You need to be able to show that the item consistently sells, across days, customers, and seasons.

This happens in the public sphere constantly as well. Think about how often the CPI doesn’t match the public’s perception of inflation. Or how people say there are labor shortages, but at the same time the unemployment rate is high. Or headlines that make it seem like AI is driving our entire economy, all the while the vast majority of all income is being made in other industries. Data often contradicts intuition.

Ahead of his presidential campaign, John F. Kennedy commented sarcastically on the Eisenhower administration. He said, “Every bright spot the White House finds in the economy is like the policeman bending over the body in the alley who says cheerfully, ‘two of his wounds are fatal – but the other one’s not so bad.’”

If you find yourself picking through data to confirm your eye test, you will lead yourself astray. The eye test is important, but it cannot be the be-all end-all of leadership. You need to be able to support your ideas using every tool at your disposal. By utilizing everything at your fingertips, you are much more likely to make the kinds of decisions that keep your business growing and operating smoothly.

1. I think I just cried a little bit writing that.