Data pipelines don't fail at 3am.
They fail six months before that.

The pager goes off at 3am and it feels, in the moment, like the pipeline just failed. It didn't, not really. So when did it actually fail? By the time an alert fires, the real failure is usually months old, a shortcut taken under deadline pressure, a test that was never written, a dependency nobody was formally responsible for. The incident is just the day it finally became visible.

01  ·  Incidents are lagging indicators, not the story itself

This is the reframe that changes how you spend your engineering time: by the time you're paged, the decision that caused the failure is long past, and no amount of incident-response skill fixes that decision retroactively. Treating incidents as the whole story means you only ever get good at responding to symptoms. The actual leverage is upstream, in the ordinary daylight choices that quietly accrue risk long before anything breaks.

$ postmortem trace-origin --incident=inc-4471
page fired ............... 2026-06-14 03:07 UTC
root cause committed ...... 2026-01-09
gap between cause and page  156 days
root cause: retry logic added under deadline,
  never covered by a test, owner since left the team
# the 3am page didn't create this problem. it just found it.

02  ·  What data debt actually looks like, specifically

It rarely announces itself as debt. It looks like a reasonable decision made under a reasonable deadline: a retry added without a test because the fix worked and there wasn't time to also write coverage for it, a transform whose owner moved teams and was never formally replaced, an alert threshold set once at launch and never revisited as volume grew tenfold, a piece of undocumented tribal knowledge that lived entirely in one engineer's head and left the company when they did.

None of those four things will show up in a code review as a problem. Each one is completely invisible until the exact conditions arrive that expose it, which is why they can sit quietly for months.

03  ·  The audit that catches it while it's still boring

The fix isn't a heroic monitoring push. It's a quarterly review, unglamorous by design, that asks a small number of pointed questions about every pipeline that matters:

1. 01Does every critical pipeline have a named owner, right now, not six months ago? Ownership decays as people change teams. An audit that doesn't check this quarterly will trust a name that's already wrong.
2. 02Are the transforms that matter actually covered by a test? Not the pipeline as a whole. The specific transformation logic where a silent error would be expensive.
3. 03Were alert thresholds set for current volume, or launch-day volume? A threshold that made sense at one-tenth the current load is either firing constantly or never firing at all. Neither is useful.
4. 04Is anything about this pipeline only known by one person? If the answer is yes, that's a debt with a specific, urgent expiration date: whenever that person is unavailable.

04  ·  The trade is almost always worth it

Ownership, tests, sane alert thresholds, and documentation are boring precisely because they work, silently, by converting a future 3am page into an ordinary daytime ticket that gets fixed on a Tuesday afternoon by someone who was already awake. That trade, a little tedium now for no emergency six months from now, is one of the very few trades in engineering that is almost never a bad one. The daylight work doesn't feel urgent. That's exactly why it's the work most worth doing before you're forced to.