Theories That Stop Working

A friend asked me recently why I get suspicious whenever someone hands me a grand theory of politics or society. I had to think about it. The answer I eventually arrived at is that such theories tend to stop being right. Even the ones that work, work for a while and then quietly stop working, and the people holding them are usually the last to notice.

There's a name for part of what's going on here. Soros calls it reflexivity. Theories of society affect the society they're theories of. If enough people believe markets are efficient, they start trading as if they were, and the markets behave differently as a result. If enough people believe class is the engine of history, they organize themselves into classes and act accordingly. The theory and the thing it describes are coupled. You can't run the experiment twice with the second run untouched by the first.

Reflexivity already makes universal theories of politics suspect. But there's a deeper problem. There are a lot of these theories, and they fight. Marxism, liberalism, nationalism, various flavors of conservatism, the technocratic worldview, the populist one — these compete, eat each other, mutate, sometimes merge into something the original authors wouldn't recognize. A theory that perfectly described the situation in 1955 may describe nothing in 1985, partly because it was right, and being right changed what it was describing.

So there are two forces working against any universal theory of how society works. One is reflexivity. The theory changes the thing. The other is that the landscape moves on its own, because the theories on offer compete and reshape each other. By the time you've finished your description, the description is part of what needs describing.

I think there's probably some immutable layer underneath all this. Humans get hungry, jealous, status-conscious, want their children to do well, want to belong, want to be left alone, want to feel they matter. That layer doesn't change much.

But society isn't the average of these urges. Societies get shaped much more by their extrema than by their averages. A small number of unusually ambitious or unusually angry or unusually persuasive people drag everyone else into formations the average person would never have chosen on his own. And the supply of such people, and the conditions that activate them, keep changing.

Which means even if you had a clean theory of human nature, you still wouldn't have a theory of human society. You'd have at most a theory of the substrate. The action lives in the tails, and the tails are the part that history keeps rewriting.

There's a useful distinction here from David Deutsch, who talks about explanations having reach. A good explanation in physics reaches far beyond the data that suggested it. Newton wasn't just describing apples; the same equations work for the moon, and for satellites nobody had built when he wrote them down. That kind of reach is one of the strangest and most powerful things in science.

Social theories sometimes look like they have reach too. You learn one and suddenly it explains everything you read in the newspaper for a month. But the reach is mostly descriptive. It tells you a story you can apply to a lot of cases. It does not, in general, let you predict the next case, and it definitely does not tell you what to do about it.

Most of these theories work much better as histories than as prophecies.

You can sometimes use them to say, in retrospect, why such-and-such happened. You can less often use them to say what's going to happen, and rarely to say what should be done. The jump from "this is how the world works" to "therefore we should do X" is where reflexivity bites and the landscape moves under you. By the time the prescription has been carried out, it's a prescription for a world that no longer exists, partly because you carried it out.

A lot of the worst policy of the last century looks like this if you squint. Someone had a theory with great descriptive reach. The theory said society works such-and-such a way. So they tried to push society further in the direction the theory implied was natural, or correct it back to what the theory said was healthy. The push worked for a while because the description was partly right. Then the world adapted, the theory stopped describing anything in particular, and the policy turned into a strange ritual that nobody could quite remember the original reason for.

The same thing happens in business, on a shorter timescale. Someone notices a pattern in what makes companies succeed. They write a book. The book is genuinely useful for a few years. Then everyone reads the book and starts copying the playbook, and the playbook stops working, because what made it work was that not everyone was doing it. The reflexivity is faster in markets, so the cycle is faster, but the shape is the same.

You see this clearly in management theories. Someone watches a successful company up close, identifies the practices that seem to be making it succeed, and packages them. Within a decade those practices are gospel. Within two, they get blamed for everything wrong with corporate life. The practices didn't get worse. They got common, and what made them powerful was being rare.

I think this is why the smartest people I know in finance treat their models almost the way a sailor treats charts. You use them, you'd be crazy not to, but you never forget that the map is not the territory, and that some great storm may have already changed the channel since the chart was printed.

It also explains something I used to find puzzling about old political writers. You read someone like Burke or Tocqueville or Bagehot and they're strangely cautious for people who clearly had a lot to say. They keep refusing to hand you the clean theory you can almost feel them assembling under the surface. I used to think this was a stylistic tic, or English diffidence. Now I think they understood the reflexivity problem in their bones, from having watched enough revolutions go sideways.

None of this means social theories are useless. A description that lasts twenty years can be valuable. A description that lasts a hundred is a treasure. But that's a different thing from a theory in the physics sense, and pretending otherwise is where a lot of mischief starts. You get people writing sentences with the cadence of physics and the content of forecasts, and confusing themselves and everyone else about what kind of claim they're making.

My rule of thumb, for what it's worth: when someone presents a universal theory of politics or society, ask what would have to be true for them to abandon it. If the answer is nothing, it's much less a theory and much more a dogma. Treat it accordingly. Dogmas can be useful. They can hold a community together, give people courage, generate art. They just shouldn't be confused for something greater.

The good ones work at a lower altitude.

They look at one institution, or one period, or one mechanism, and try to say something precise about it. They're suspicious of their own generalizations. When they do generalize, they hedge a lot, and the hedges aren't throat-clearing. The hedges are part of the claim. They're saying: here is what I noticed, and here are the conditions I noticed it under, and beyond those conditions I make no promises.

If you read enough of them you start to suspect that there isn't going to be a Newton of human affairs, and there shouldn't be. Newton's universe doesn't notice him. Human society does. Any sufficiently powerful theory of it will be used by some people and resisted by others, and a few years later the theory will be describing a different society than the one it started with. The ceiling on universalism in this domain isn't a temporary one, waiting for some smarter person to break through. It's structural.