We published our odds, then found they were wrong by three times

The rarest outcome in the game was advertised at 2.2%. Re-measured over 2,000 careers it is 6.6% — and the error was in the game's own disfavour.

One of the things this game is built on is that most careers fail. Not as a difficulty setting — as the subject. So the actual failure rate is a number we publish, in full, on a page of its own, along with the method that produced it.

Publishing it turned out to be the useful part, because it forced someone to re-run it.

What the numbers said, and what they say now

The figures on the page came from a measurement taken on 23 July. Between then and now, three separate balance passes went through the simulator — none of them large, none of them aimed at the outcome distribution. The published odds were never updated, because nothing about a balance tweak announces that it has moved a statistic on a website.

Re-measured at a thousand careers per play profile:

  • Lives that never turn professional at all, playing without a plan: 22.8% → 27.1%
  • Careers that reach a five-star club, playing well: 8% → 12.7%
  • The rarest legacy tier, Immortal: 2.2% → 6.6%

The last one is the interesting failure. It is not a small drift; it is off by a factor of three. And it is off in the direction that made the game sound harsher and rarer than it is — we were advertising an achievement as three times more exclusive than it actually is.

Why an understated number is still a problem

It is tempting to file "we undersold ourselves" as a harmless error. It isn't, for a reason that has nothing to do with marketing.

A published statistic is a claim about the simulation, and players use it to decide how to play. Someone reading that 2.2% of lives reach Immortal is being told, implicitly, that reaching it requires something close to optimal play across an entire career. At 6.6% the honest advice is different: it is rare, but it is a thing a good player should expect to see eventually. Those are two different games to sit down to.

The second problem is trust in the rest of it. If one figure on a page of measured figures is wrong by 3×, every other figure on that page is now unverified in the reader's mind — and correctly so, because they came from the same stale run.

The mechanism, which is the actual point

The measurement is not an estimate. probe_engagement.dart runs the real simulator, from a fixed seed, across a set of scripted play profiles — the player who never trains, the player who trains every year and always takes the ambitious option, and several in between — and tallies what happened. Same seed, same numbers, on any machine.

That determinism is what makes the figures citable, and it is also what made the drift invisible: the run is reproducible, so nobody thought of it as something that needed re-running. A number that cannot change by accident feels like a number that cannot change.

The fix is procedural and boring. Any change to the balance means re-running the probe before the release notes are written, and the page carries the date and the sample size at the top so a stale figure is visible to a reader rather than only to us.

What we are not going to do

We are not going to round the odds into ranges to buy ourselves room — "around 1 in 20" — which would make this class of error disappear by making the claim unfalsifiable. The whole value of the page is that it commits to a number somebody could check. Being wrong in public and correcting it costs less than being vague on purpose.

Want the next one?

The early-access list gets one email when the gate opens, and nothing else. The devlog is also available as a feed if you would rather not hand over an address.

Claim my place