The Two Types of Mistakes

There’s a concept from statistics about the two types of mistakes you can make. These are, unhelpfully, referred to as type I and type II errors.

To examine the difference between the two types, imagine you have a blood test for a rare and deadly disease.

There are four possible outcomes of the test.

The first is a true positive—you have the disease and the test shows you have the disease.

The second is a true negative—you’re free of the disease, and the test says you don’t have it.

A type I error is a false positive—you think you have the disease, but you actually don’t.

A type II error is a false negative—the test says you’re healthy, but you’re actually going to die soon.

The Important Thing About Type I and Type II Errors

One of the important things to note is that the error rates of these two types of tests can be different. In other words, it’s not simply the case that a blood test will be equally likely to make both types of mistakes. In fact, there’s often a trade-off where one mistakes ends up being more likely.

Consider a home pregnancy kid. These test for a hormone in the blood, human chorionic gonadotrophin (hCG), as a marker of pregnancy.

These tests usually don’t make Type I mistakes. If you have hCG in your urine, you’re almost certainly pregnant.

However, they can make Type II mistakes much more frequently. If you’ve just recently conceived, for instance, your hCG levels may not be detectable by the device. It says you’re not pregnant, but you actually are.

Choosing Your Mistake Trade-offs

Type I and Type II errors aren’t restricted to medical scenarios. They happen nearly every time you make a decision.

Suppose you think housing prices are going to keep going up in your neighborhood. You’d better buy, or you’ll soon miss out on the boom (or be unable to afford it). We can see the errors from before mapped onto this decision:

1. True Positive: You buy a house and prices do go up. (Yay!)
2. True Negative: You don’t buy a house, but prices don’t go up. (Phew!)
3. Type I Error: You bought a house, but prices didn’t go up. (Hmmm…)
4. Type II Error: You didn’t buy a house, but prices did go up. (Ugh)

Because we make decisions under uncertainty, there is never going to be a way of deciding that will avoid any chance of making a mistake. Therefore, considering the two types of mistakes, you can choose a strategy that can change the relative risks.

Do You Want More Type I or Type II?

Let’s go back to our deadly disease example. Suppose now, that we have a cure for this disease, which only requires a small cost and annoyance. If you don’t take the treatment, however, you’ll almost certainly die.

Here the costs of Type I and Type II mistakes are wildly different. Type I results in some unnecessary, but harmless treatment. Type II results in death.

If the disease were prevalent enough, it may make sense to always get the treatment, regardless of whether the test shows you have the disease or not. The costs are just too high.

What if we changed the parameters though? What if instead of dying, the disease only made you feel sick for a few days. And, the treatment, instead of a mild side-effect, was a painful series of injections?

Now, we might even consider avoiding treatment, even if the test says we have the disease, since the cure is almost worse than the illness.

How to Adjust Your Decision Making to Avoid Costly Mistakes

One strategy is to always choose one course of action or another. Always take the medicine, even if the test shows no disease. Always avoid it, because getting sick isn’t so bad.

Unfortunately, most decisions are somewhere in the middle. We need to adjust our actions based on what we learn, but we may not follow what we learn blindly. Because the costs of Type I and Type II errors are different, we may lean more optimistic or skeptical, depending on the situation.

Trade-Off Examples: Meditating and Investment Schemes

Consider my recent article on meditating. I mentioned how I spent some time meditating, found a large positive benefit from my retreat experience, but mixed results from a few months of meditation at home. What should I do?

The Type I error here is that I think meditating is useful, but it does nothing (or is harmful). The cost here is that I end up wasting some time and effort in an activity that doesn’t benefit me.

The Type II error here is that I don’t meditate, even though it is quite helpful. The cost here is that I miss out on the benefit, but I save some time.

Since my experience says the potential benefit is quite high, compared to the cost, I’m willing to risk Type I errors, to minimize Type II. In this sense, I lean optimistic, simply because excess skepticism is more dangerous.

On the other hand, I tend to be a lot more skeptical when it comes to investment schemes. Losing all my money will hurt a lot more than gaining a little extra, so I’d rather play it safe and be doubtful of real opportunities than gamble for a little bit more.

Applying this Principle

Look at the decisions you’re currently making. What are the Type I errors (falsely believing in something that isn’t real) and Type II errors (falsely denying something that exists) in your life. Which is more costly?

The course of action you decide shouldn’t just depend on what you think is more likely, but which is going to be more expensive if you’re wrong. A higher asymmetry between the two types of mistakes should make you more optimistic/cautious than what a neutral observation would imply.

Try this: Pick a decision you’re working on right now and write down what the correct positive, correct negative, Type I and Type II errors for it would be as an exercise. Share your work in the comments!