Don’t Analyze The Anecdotes When You Need Data

In statistics, one of the first things you learn is the importance of n. The variable n, represents the size of the sample you’re studying. And the size of n has a huge impact on whether your results are significant or just a fluke. Unfortunately it’s easy to forget this lesson outside the classroom and spend too much time thinking about events with a sample size of 1.

This kind of broken overanalysis can happen when you screw up. You get up to deliver a speech and bore your audience to tears. Doesn’t it make sense to look into what mistakes you might have made, in order to fix the problem in the future? If you’ve only given one speech, it doesn’t. Comedians often practice a routine dozens of times to get the timing right, so it doesn’t make sense to draw definite conclusions after one failure.

Overthinking can also happen when you make a big win. When I started blogging, I was thrilled if one of my articles was picked up by a big website or received a huge traffic burst. The adrenaline rush of seeing your website stats make a right angle as they rocket upwards is an incredible feeling.

My initial mistake, however, was after I’d get a big win–I’d try to replicate that win with similar posts. These posts rarely worked because the first post had succeeded on luck. Trying to copy that success isn’t possible, and I probably would have been better off writing something fresh.

Why Analyzing Individual Cases Hurts You

There are a couple reasons looking too much into small samples hurts you. Here are just a few:

1) Hesitation Replaces Action

Don’t stop and think when you need to act quickly. When failing is fast and cheap, it doesn’t make sense to stop and ponder over a single event. With blog writing, it takes just over an hour to test a new post idea. Why spend time analyzing when I could be out getting actual experiences?

2) Superstition Replaces Trends

When you focus on individual events, you form false beliefs. I know people who swear that pigeons have targeting mechanisms when relieving themselves midair. They remember the one time bird droppings landed on their head, but can’t see the thousands of times it landed safely to the ground. If possible, it’s better to look at the overall trend than individual anecdotes.

3) Gimmicks Replace Strategy

You can’t analyze a strategy when you only look at the specifics. Some strategies will cause a lot of failures and a lot of wins at the same time. Babe Ruth was famous for having both the home run and strikeout record at the same time. But if you get caught up on a single event, you might switch to a strategy that avoids failures, but also avoids any wins.

How to Ignore the Events and Focus on Trends

Humans are built to reason by stories, not statistics. I’ll be more motivated to act after hearing an emotional plea than a set of pie charts. So it isn’t easy to focus on the trends. I often succumb to anecdote analysis, even if I try to focus on the numbers.

There are a couple ways I’ve found useful to stay focused on the data:

Measure Everything

You need to make the data louder than the numbers. One way to do that is to keep accurate measurements you review regularly. Taking measurements allows you to look back into the past, so you can see a broader trend over months and years, instead of days and weeks.

Use Testable Hypothesis

Looking into an individual scenario can be useful for coming up with a hypothesis. Based on a few speeches, you might hypothesize that speaking more slowly would improve your success. That hypothesis is useful if you can test it in an experiment. Wait until you’ve run a full experiment before you start drawing conclusions.

Don’t Give Yourself Time to Think

When the costs of failure are low, your immediate response to a mistake should be to get back up an try again. Don’t give yourself a long delay in-between your attempts. Writing blog articles, giving speeches, exercise routines, dating and changing habits can all work through a fast, trial and error approach.

What About When You Can’t Fail Fast?

If you’re in a business where working on the next product can take a year or two, failing fast isn’t always an option. In these cases, it can make sense to look into anecdotes. But even with long project cycles, you can still improve the way you review.

A good way is to take on “learning projects”. Learning projects are a typical project, except they test 2-3 hypothesis you’d like to measure. When I finished my last e-book, I wanted to test out subcontracting, working with AdWords and managing an affiliate system. I hadn’t done any of those before, so taking on a large project would teach me something valuable about them.

Another hint for analyzing long projects is to limit your review. Try to generalize any conclusions into hypothesis you can test in your next round. Creating specific conclusions that require a large sample to test aren’t useful. If your first research report gets a C-, you might want to draw general hypothesis like, “do more resesarch” or “spend more time on proofreading” rather than specific conclusions like “I should have reworded paragraph 6.”

The real question about whether you should spend more time thinking or doing has to do with the cost of failure. If you can fail quickly without going broke, action beats analysis. If you have to fail slowly, be smart with your review so you don’t start seeing trends that aren’t there.