Chaos, Ignorance and Newton’s Great Puzzle

This is chaos:

Chaos theory is an investigation into mathematical and physical systems that are highly sensitive to initial conditions. The simulated three-pendulum systems above only differ in the tiniest way from each other, yet, after a few seconds, they are all wildly divergent in their movements.

I’ve grown to prefer the word “chaos” to the more commonly used term “luck”. People talk about luck a lot. We argue about whether someone’s success was an inevitable consequence of their skill and decisions, or whether they were just lucky. But we also talk about luck in superstitious terms, as if it was something someone possessed (“He’s just lucky.”) or that could ebb and flow with the tides of fortunes (“He’s just had a streak of bad luck.”).

Chaos, on the other hand, is a more mathematically precise concept. A system is more chaotic if small changes to its state can create wildly different outcomes. Compare the above system to the solar system. The fact that the sun rises each morning from the horizon may seem a banal fact, but it depends on the lack of chaos in our solar system. Adding a second sun creates a three-body system which, to this day, physicists do not have a way of precisely predicting the orbits.

Why Chaos Matters

Just as you can consider the amount of chaos in physical system, you can also think of the amount of chaos in different pursuits in life. The more chaotic the pursuit, the more likely small changes can result in completely different outcomes years later.

Although an absolute measurement of chaos is probably impossible, you can compare different pursuits by how chaotic they might be.

Consider two professions: becoming an actor and becoming a doctor. Both are prestigious professions. Both require a lot of hard work without much reward in the beginning, but have the potential for big payoffs.

However, acting is a much more chaotic profession than medicine. Landing the right audition can give you a name, which puts you up for bigger and bigger parts. But just as you rise, you can also fall. A bad movie or changing public tastes have also made many famous stars disappear in an equally short time.

Medicine is not chaotic. Although there will be marginal doctors who just get over the cutoff point for getting into medical school or landing a top residency, most are firmly within whatever category they eventually end up.

If you considered only a cruder metric, such as success rates, however, you might totally miss this picture. Most would-be actors and doctors fail to eventually reach success in their profession. But they may fail for different reasons. Would-be doctors fail because the profession is difficult and long, and so creates a high-dropout rate, not because it is inherently chaotic. Would-be actors may fail or succeed because small events create feedback loops causing wildly different fortunes.

The Problem with Mercury

For two hundred years since Newton, physicists had a problem: Mercury. The closest planet to the sun had a strange orbit that precessed. According to Newtonian physics, this wasn’t possible. The object would always have a fixed orbit around the sun, not one that wobbled around.

Physicists struggled against the problem of Mercury for years. Some suggested there must be a phantom planet, Vulcan, even closer to the sun, which was kicking Mercury’s orbit around.

Eventually, however, the true answer emerged: Newtonian physics is wrong. Albert Einstein introduced general relativity and its warped spacetime curvature. The new equations correctly predicted the precession and Mercury’s orbit was finally understood.

In Mercury’s case, physicists could safely rule out chaos because its precession was so orderly. However, in life, we often don’t get to observe the exact same conditions again and again to see the patterns. We only live once, so everything we experience is, in a certain sense, experienced for the first time.

As a result, the randomness we perceive in life always has a mixture of two possible components. The first is chaos. This is the amount that the system would be unpredictable, even if we had a near perfect understanding. The second is based on our own ignorance of the system, the amount it defies our expectations because we don’t really understand how it works.

How to Use Chaos in Your Thinking

In practice, it’s not possible to cleanly separate which systems are chaotic and which are merely poorly understood. However, it’s still useful to understand the existence of these two different sources of randomness.

When approaching any problem in life where outcomes are highly divergent, there’s always two possible mistakes that can be made.

The first is to overestimate the possibility of knowledge. By assuming that the system can be understood, but it is actually dominated by chaos, you may be gambling more than you realize. People who make this mistake may be convinced they’ve found the secret of success, but they’re really fooling themselves.

(Comic by XKCD)

The second error is to overestimate the role of chaos. By assuming that the system is inherently unpredictable, you may forego the possibility of mastery.

In my own life, I’ve seen this error with people new to blogging. Many people believe the field is inherently chaotic, depending on “going viral” and getting a lucky bit of publicity. The truth, however, is far more mundane. Most new bloggers I’ve met have obviously different levels of quality, consistency and work-ethic, which leads predictably to different outcomes, given enough time. Blogging success may be both rare and difficult, but I believe it’s less chaotic than many pundits would suggest.

Calibrating Between Assumptions of Chaos and Ignorance

One of my goals is to try to calibrate my expectations of chaos versus ignorance in different domains. Is my failure to predict because of inherent unpredictability or a lack of understanding?

This is difficult to do, but I’ve found there’s a few heuristics that can make the process easier:

1. Theoretical justifications. I bias towards believing success in picking individual stocks is mostly chaos. I say this, not out of any first-hand experience, but because the efficient market hypothesis strikes me as being a reasonable theory. As such, I’m not interested in learning the skill of picking stocks, and have opted to invest in index funds instead.
2. Existence of expertise. Look at the predictions of the best people. How often are they correct? If even the best don’t predict well, that doesn’t bode well for me. However, if successful predictions are common, it might demonstrate that there’s an understanding that exists which I don’t possess.
3. Diminishing randomness. Another heuristic is to see how your experience of randomness is decreasing over time. If I’ve gone from 0/10 to 2/10 on a level of mastery from newbie to expert, and my ease at predicting success has gone up substantially, I’m more inclined to believe this trend will continue.

Randomness, whether from ignorance or chaos, is not ultimately avoidable. But by better calibrating your understanding between the two, you can see where investments in learning more are worthwhile and where it is probably a waste of time.