The demo is not the product.
Budget for the gap.

A demo that works four times out of five feels almost done. In my experience it's closer to a fifth of the actual work. How can something that looks 80% finished actually be one-fifth done? Because the missing part is nearly invisible until someone goes looking for it, which is exactly why so many AI projects stall at that mark and never quite leave it.

01  ·  Why a demo can be honest and still mislead you

Nobody's lying in a demo. It's just structurally a curated best case: a handful of inputs chosen because they work, a small enough scale that latency doesn't matter yet, and usually a person in the room quietly backstopping the weird edge case if one comes up. Production is an uncurated worst case that arrives continuously, at whatever scale users actually show up in, with nobody standing by to catch the strange input.

02  ·  What the invisible 80% actually is

It's rarely one big thing. It's a list, and every item on it is easy to skip in a demo because a demo doesn't need it to look impressive:

1. 01A real eval suite, run against a golden set. Not "I tried it a few times and it seemed good." A measured, repeatable pass rate against representative cases, including the ones you expect to be hard.
2. 02Guardrails and refusals. Explicit handling for the inputs the system shouldn't attempt to answer at all, not just the ones it answers well.
3. 03Retries and fallbacks. Every external call, model or otherwise, fails sometimes. A demo never runs long enough to see it happen.
4. 04Observability and tracing. When it's wrong in production, you need to see why, on that specific request, not just that it happened.
5. 05A defined human-escalation path. The cases the system genuinely can't handle need somewhere specific to go, not a silent wrong answer.

$ eval run --set golden_v3.jsonl --n=400
pass rate (demo's 12 hand-picked cases) ...... 12/12
pass rate (golden set, n=400) ................ 331/400
worst category: multi-step numeric reasoning . 41%
# the demo tested 3% of the space it needed to. this is why.

That gap between 12 out of 12 and 331 out of 400 is not a failure of the model. It's the honest difference between a hand-picked set and a representative one, and it's exactly the number that a demo, by design, never surfaces. Nobody runs 400 adversarial cases live in a stakeholder meeting.

03  ·  Put a real number on the gap before you sell the demo

As a rough planning rule, I tell clients to budget roughly the same effort again past the point the demo feels done, sometimes more, depending on how much genuine edge-case handling the problem space actually needs. That's not a universal law, it's a starting assumption to correct once you've measured your own eval numbers, but it's a far better starting assumption than treating the demo as 80% of the project.

The projects that stall aren't the ones with hard problems. They're the ones that sold a demo as nearly finished and then discovered the real scope one production incident at a time, instead of one line item in a plan.

04  ·  Sell the demo as the cheap part, because it is

The teams that actually ship treat the working demo as proof of concept and put the unglamorous remainder in the plan and the budget from day one, before a client has fallen in love with how close it already looks. The demo is not the product. It's the cheapest 20% of it, and the only projects that stall are the ones that never budgeted for the other four fifths.