Schema drift is a people problem.
The pipeline was fine.

I get pulled into engineering postmortems more often than my job title would suggest, because a striking number of them end the same way: the code did exactly what it was told, and the real finding was that two teams had quietly stopped talking to each other. This one got logged as a schema drift. If the code did nothing wrong, what actually broke, and why didn't anyone notice for days?

01  ·  The org chart is part of the architecture

There's an old observation, usually credited to Conway, that systems end up shaped like the communication structure of the organization that built them. I'd go further: when the communication structure quietly breaks, the system breaks with it, just later and much more quietly than a code bug would. Nobody gets paged when two teams stop talking. Something downstream just starts being wrong.

02  ·  The timeline, reconstructed

Here's roughly how it went, generalized from a pattern we've now seen enough times to recognize instantly. A field that started as a simple status flag grew a third possible value, because a product team needed to represent a new state and the change felt small. It was small, on their side. Nobody on the data team, three hops downstream, knew it had happened.

$ incident timeline --id=inc-2291
mon  field 'status' gains a third value (product team, unannounced)
tue  pipeline ingests new value, maps it to 'unknown' (silent)
wed  downstream dashboard metric starts drifting (unnoticed)
thu  drift compounds; nobody has connected it to monday yet
fri  analyst flags the number as 'off' → investigation starts
# four days between cause and anyone noticing the effect.

The pipeline didn't crash. It did precisely what it was built to do: accept the field, fail to recognize the new value, and quietly bucket it as "unknown." That's correct, defensive engineering. It's also exactly why nobody noticed for four days: a silent failure mode is the kind that's easiest to build and the hardest to catch, because there's no error to see.

03  ·  Why nobody announced it

Not because anyone was careless. Because there was no ritual that made announcing it the obvious next step. There was no shared understanding of who, downstream, actually depended on that field, no habit of checking, no one whose job it was to ask "does changing this break something I can't see from here." That absence is an organizational design gap, not a personal failing, and it's the kind of gap I actually know how to close.

04  ·  Contracts are a people tool before they're a technical one

The technical fix, a data contract, an explicit, versioned agreement about a field's shape and meaning, gets talked about as a data-engineering pattern. I think that undersells it. What a contract really does is force a conversation that the org chart wasn't otherwise going to produce: it names an owner, it names the consumers, and it makes a silent change require someone to actively break an explicit promise instead of just quietly shipping.

A contract is a communication structure wearing a technical costume. That's not a criticism of it. It's why it works when "just be careful" doesn't.

1. 01Name an owner for every field that crosses a team boundary. Not a team. A person, or a rotating single point of contact, who is expected to know and can be asked.
2. 02Validate at the boundary, and make the failure loud. The four days of silence were the real cost. A schema check that fails ingestion immediately turns a slow leak into same-day news.
3. 03Route the alert to whoever can actually fix it. We were routing drift alerts to the team that owned the dashboard, who had no ability to change the upstream field. Reliability improved the day we rerouted it to the team that did.
4. 04Make "who consumes this" a five-minute lookup, not an archaeology project. A lightweight, current registry of who reads what removes the excuse of not knowing who to tell.

05  ·  The lesson that keeps generalizing

I've sat in enough of these postmortems now to trust the pattern: the incidents that go deep enough stop being technical stories and become communication stories. The code was doing its job. The org chart wasn't doing its. Fixing the second one prevents more of these than any amount of additional monitoring on the first, because monitoring catches drift after it happens. A clear owner and an obligation to announce prevents the drift from happening silently in the first place.