The Residual Novelty Filter

Suppose you ask a competent programmer how long some task will take, and she says "two days." You write down two days. Two days later, the task is not done. Four days later, the task is not done. Eight days later, the task is mostly done, except for a thing involving character encodings that nobody anticipated, and a thing involving the staging environment that nobody anticipated, and a thing where the library you were using turns out to behave differently in production than in development, which nobody anticipated. The task ships on day eleven.

The naive model says: the programmer was wrong by a factor of five and a half. The naive model then proposes a correction: programmers are systematically optimistic, so multiply their estimates by some fudge factor — three, say, or pi, depending on which folk wisdom you've absorbed.

This correction does not work. If you've ever tried it, you know it does not work. You apply the multiplier, you get an estimate of six days, and the task still ships on day eleven. Or — more annoyingly — it ships on day three, because it turned out to be easy, and now you've burned political capital telling stakeholders it would take six. The multiplier theory predicts that estimates are biased; what we actually observe is that estimates are unreliable, which is a different and worse property.

I want to claim that almost everything written about software estimation is wrong in the same way, and that the wrongness has a clean mechanical explanation, and that once you see the mechanism you stop being surprised by the data and start being surprised by the occasional case where estimation works.

Here is the wrong view I want to prosecute. The wrong view says: estimating software is hard because programmers are bad at it — bad at introspection, bad at remembering past projects, biased toward optimism, prone to ego, insufficiently disciplined about breaking work into small pieces. The wrong view then proposes solutions: story points, planning poker, reference-class forecasting, three-point estimates, historical velocity. Some of these are mildly useful. None of them addresses the actual problem. The actual problem is not in the estimator. The actual problem is in the thing being estimated.

Consider what a software task is. A software task is a specification of a transformation from the current state of a codebase (plus its dependencies, plus the runtime environment, plus the data it will encounter, plus the users it will serve) to a new state in which some property holds that didn't hold before. The "estimate" is a prediction about how long it takes a human to find a path through the space of possible code-edits that achieves the new state without breaking the old states people care about.

Notice what this is not. It is not the estimation of the duration of a known procedure. When a contractor estimates how long it takes to drywall a room, she is estimating the duration of a procedure she has executed many times, on rooms that vary along a small number of well-characterized dimensions (square footage, ceiling height, number of corners). The variance is real but bounded, because the procedure is fixed and the inputs live in a low-dimensional space.

A software task is almost never like this. If the task were the execution of a known procedure on well-characterized inputs, somebody would have automated it already, and you would not be estimating it; you would be running it. The tasks that remain — the tasks that get estimated — are precisely the tasks that have not been reduced to a known procedure. They are, by selection, the tasks where the path through code-space has to be discovered rather than retrieved.

This is the first key idea, and I want to give it a handle: call it the Residual Novelty Filter. The work that ends up on a programmer's estimate sheet is the work that was left over after every previously-solved version of the problem was abstracted into a library, a framework, a compiler, or a language feature. By construction, the remaining work has a novelty component. By construction, you are estimating the duration of a search, not the duration of an execution.

Now: how long does a search take? In the general case, you don't know until you've done it. That's what makes it a search. If you knew how long it would take, you'd have a heuristic that predicts the answer, and that heuristic would itself shorten the search, and the search would no longer be the same search. The cases where you can estimate a search reliably are the cases where the search space is so well-mapped that it isn't really a search anymore — it's a lookup with some friction. Those are exactly the cases the Residual Novelty Filter has stripped out of your task list.

So the first reason estimates are wrong is that they are predictions about the duration of a process whose duration is, in the relevant technical sense, not predictable from the information available at estimation time. This is not a moral failing of the estimator. It is a property of the task.

But this isn't the whole story, because if it were, estimates would be merely noisy. They would be wrong in both directions about equally, and the noise would average out across many tasks. Empirically, this is not what happens. Estimates are systematically biased toward underestimation, and the bias does not wash out across portfolios of tasks; large software projects famously overrun by factors that no central limit theorem will save you from.

The second mechanism is the one that does the heavy lifting, and it's worth being careful about. Call it Conjunctive Blindness. When a programmer imagines a task, she imagines a path: first I do A, then B, then C, then it works. Her estimate is roughly the sum of her estimates for A, B, and C, perhaps with a small fudge for "integration." But the path she imagined is the path conditional on every step working as imagined. The probability that a four-step plan executes as imagined, when each step has even a 90% chance of going as planned, is 0.9^4 ≈ 0.66. The probability that an eight-step plan executes as imagined under the same assumption is 0.43. And these are wildly optimistic per-step probabilities for tasks that, by the Residual Novelty Filter, contain irreducible search.

What happens when a step doesn't go as planned? The programmer doesn't fail the task; she discovers a sub-task. The character encoding thing. The staging environment thing. The library-behaves-differently thing. Each of these is itself a search, with its own conjunctive structure, its own probability of spawning sub-sub-tasks. The task tree branches, and the total work is the sum over the tree, not the sum over the originally-imagined linear path.

The estimator, asked for a number, reports the length of the linear path. She is not lying and she is not being optimistic in any psychologically interesting sense. She is reporting the only thing she has cognitive access to: the modal happy path, the plan she would execute if reality cooperated. The branches she hasn't imagined yet are, by definition, branches she hasn't imagined yet. You cannot include in your estimate a thing you have not thought of.

This is why the multiplier doesn't work. Multipliers assume the bias is in the size of the imagined steps. The bias is in the topology — the imagined plan is a line, the actual plan is a tree, and the ratio between them depends on the depth and branching factor of the tree, which depends on the novelty of the task, which is exactly the thing your estimate cannot see clearly because if it could, the task wouldn't be novel.

Now consider two worlds, identical except in one respect. In World A, programmers estimate tasks by introspecting on the imagined happy path. In World B, programmers estimate tasks by reviewing the historical distribution of completion times for tasks they once described in similar language. Which world has better estimates?

Naive answer: World B. Reference-class forecasting is the right move; we know this from Kahneman; programmers in World B have escaped the inside view.

This is partly right and partly wrong, and the partly-wrong part is instructive. The historical reference class is built out of past tasks, but past tasks went through the Residual Novelty Filter at their moment of estimation, not the current moment. The library that didn't exist last year exists this year; the bug class that bit you last year has been compiled out of the language; the integration that took two weeks last year is now an import statement. Conversely, the new framework you're using this year will exhibit failure modes that nobody in your reference class has ever debugged. The reference class is drawn from a non-stationary distribution. You can use it, and you should, but you should expect it to be miscalibrated in directions that your intuitions about reference classes do not warn you about. The fact that it is less wrong than introspection does not make it right.

So what should you actually do? I want to resist the urge to end with a methodology, because methodologies for estimation are mostly motte-and-bailey: the motte is "think carefully and use historical data," which is fine, and the bailey is "and now your estimates will be accurate," which is false. Instead, here is a diagnostic and a generalization.

The diagnostic: when somebody gives you a software estimate, ask them to describe the plan. If they describe a linear sequence of steps, your estimate is the sum of those steps conditional on no branching, and you should treat it as a lower bound whose looseness scales with the novelty of the task and the depth of the stack being touched. If they describe a tree with explicit branches and explicit "if this fails, then…" nodes, the estimate is more trustworthy, because the estimator has actually done some of the search in their head before quoting you a number. Most estimates are linear-plan estimates. Most estimates are, accordingly, lower bounds dressed up as point predictions.

The generalization — and this is the part I actually care about — is that whenever you find yourself asking how long it will take to do a thing whose doing involves discovering how to do it, you should expect the same failure mode. Research timelines. Debugging sessions. Diplomatic negotiations. Writing a proof. Designing an experiment. Convincing a specific person of a specific thing. In all these cases the work is search, the imagined plan is a line, the actual execution is a tree, and the estimate produced by introspecting on the imagined plan will be systematically and unboundedly low. The estimate produced by reference-class forecasting will be better but miscalibrated by the non-stationarity of the underlying distribution.