A walk backward through data, context, and reasoning — and why most enterprise AI systems can’t survive the trip.
I’ve spent over a decade building enterprise data and analytics platforms — the kind that sit underneath dashboards, reports, and, increasingly, AI-driven recommendations at large organizations. A few years ago, on a healthcare analytics program, I watched a readmission-risk model quietly flag a batch of patients as low-risk who had no business being scored that way.
The answer wasn’t in the model. It was buried four layers back, in a semantic layer metric that had silently drifted after an upstream schema change nobody flagged. And finding it meant walking backward through a chain most enterprise teams never bother to map: data → context → reasoning → recommendation → action.
This is the story of that walk, and why I think every team shipping AI-driven decisions — especially ones built on top of enterprise analytics platforms — needs to be able to make it.
The question nobody asks until it’s too late
When a model spits out a recommendation, the natural question is “is it right?” That’s the wrong first question. The right one is “where did this come from?“
Most teams can answer that for maybe one link in the chain. They can show you the training data. Or they can show you the final output. What almost nobody can do on demand is walk the full path — this specific data point, combined with this specific context, produced this specific line of reasoning, which produced this specific recommendation, which triggered this specific action.
That gap is invisible right up until someone asks you to explain a decision under pressure. A regulator. A customer. Your own VP after a bad outcome. Then it’s the only thing that matters.
What “tracing” actually means
I want to be precise here because “explainability” has become a word that means everything and nothing. I’m not talking about SHAP values or attention maps — those tell you what the model weighted, not what actually happened in your system.
Tracing means being able to answer, for one specific decision, all five of these:
- Data: What raw inputs fed this decision, and what version were they in at the time?
- Context: What surrounding state — user history, session data, external signals — shaped how those inputs were interpreted?
- Reasoning: What logic, rules, or model inference sat between the inputs and the output?
- Recommendation: What did the system actually suggest, and with what confidence?
- Action: What happened next, who or what acted on it, and did the action match the recommendation?
Most incident postmortems I’ve sat through get maybe two of these five. Usually reasoning and recommendation, because that’s what’s easiest to log — it’s just the model call. Data and context get inferred after the fact, which means they get misremembered after the fact. Action almost never gets tied back at all.
The readmission flag, walked backward
Here’s what that walk actually looked like once we stopped guessing and started instrumenting the platform itself, not just the model.
- Action first, because that’s all we had: A cohort of patients had been excluded from a post-discharge follow-up program because they’d been scored low-risk. That exclusion was the artifact. Everything upstream of it had to be reconstructed.
- Recommendation: The model’s output log showed a risk score below the threshold for each patient — nothing alarming in isolation. No smoking gun yet.
- Reasoning: This is where it got interesting. The model consumed a set of engineered features from our semantic layer, and one of them — a rolling comorbidity-burden metric — was flatter than it should have been across the entire affected cohort. Feature contribution analysis showed that the metric was pulling scores down almost uniformly, regardless of each patient’s actual clinical history.
- Context: Why would that metric go flat? Because the semantic layer definition for it depended on a join against a diagnosis-history table, and that join was silently returning fewer matches than it should have — not zero, which would have thrown an error, just fewer, which looked like normal data variance in every dashboard we had.
- Data: And here’s the root: an upstream source system had changed how it encoded diagnosis codes during a routine schema update. Nobody had broken anything loudly. The pipeline ran green every night. But the join logic feeding that semantic metric was quietly matching on a stale code format, so a meaningful share of real diagnosis history was falling out of the calculation before it ever reached the model.
None of these five layers was individually broken. The model was doing exactly what it was trained to do. The pipeline hadn’t crashed. The semantic layer wasn’t throwing errors. But stacked together, they produced a decision nobody would have signed off on if they’d seen the whole chain at once — and in a healthcare context, “low-risk” isn’t a label you want to get wrong quietly.
That’s the part that stuck with me: failures like this rarely live in one layer. They live in the seams between layers, where nobody owns the handoff.
Why this is harder than it sounds
If tracing sounds simple in theory, here’s why it isn’t in practice.
- Versioning is inconsistent across the stack. Your data warehouse might version tables. Your feature store might version features. Your model registry versions models. But almost nobody versions the combination — this data version, with this feature logic, feeding this model version, at this exact timestamp. Without that composite fingerprint, you can’t reconstruct the past; you can only approximate it.
- Context is ephemeral by design. Session state, real-time signals, user history at the moment of inference — a lot of this is intentionally not persisted, for cost or privacy reasons. Which means the context that shaped a decision can simply be gone by the time you go looking for it.
- Reasoning isn’t always inspectable. Rule-based systems are easy to trace. Gradient-boosted trees, less so but doable. Large neural networks, genuinely hard — you can get feature attributions, but “why did the model weight this token” is a statistical story, not a causal one. The deeper the model, the blurrier this link gets.
- Actions get decoupled from recommendations. A human overrides the model half the time, or a downstream system batches recommendations and only acts on some of them. If you’re only logging model outputs, you’re not logging what actually happened in the world — and those two things drift apart faster than teams expect.
What actually helped
I’m not going to pretend we solved this cleanly. But a few things moved us from “we can’t trace anything” to “we can trace most things in under an hour,” which is a real improvement.
We started fingerprinting decisions, not just logging them. Every recommendation now carries a composite ID referencing the exact data snapshot, semantic layer definition version, feature version, and model version involved — not just a timestamp, an actual pointer to reconstructable state.
We stopped treating the semantic layer as a black box. Metric and feature definitions in the semantic layer now carry their own version history and lineage back toraceable change, not an unexplained shift in a number
We separated “recommendation” from “action” in our logs, explicitly. If a clinician, analyst, or downstream system deviates from the model’s suggestion, that deviation is logged as its own event, not silently merged into “what happened.”
We made the chain queryable by non-engineers. This mattered more than I expected. A clinical reviewer or compliance analyst shouldn’t need a data engineer to reconstruct a decision. If tracing only works when the right person is in the room, it doesn’t really work.
The uncomfortable part
Here’s what I keep coming back to. Being able to trace a decision doesn’t make the decision better. It just makes it accountable. And accountability is uncomfortable, because it means you can no longer wave at “the model” as an explanation. You have to say: this data was stale, this context signal was overweighted, this handoff had no owner.
That’s a harder conversation than “the AI decided.” But it’s the only conversation that actually fixes anything, and it’s the one your regulators, your customers, and eventually your own team are going to demand — whether or not you’re ready to have it.
If you can’t answer, right now, how a specific decision your system made last week traces back to the data behind it, start there. Pick one recommendation your platform produced this month, and see how far back you can actually walk it. That single exercise will tell you more about your AI readiness than any model benchmark will.
