Why Degraded Tools Don’t Behave Like Bugs
When a tool call fails with an unhandled exception, most frameworks surface it clearly — LangChain’s error-handling tooling and the OpenAI Agents SDK both surface visible errors into the agent’s trace. What happens when a tool returns a 503 or times out quietly is different: the agent receives what looks like a valid result, just an unhelpful one. The framework continues. The agent continues.
The agent then interprets “no data returned” as “there is no data.” A tool that returns nothing because it’s down is indistinguishable, from the agent’s perspective, from a tool that returns nothing because there is genuinely nothing to return.
The model is reasoning correctly from the information it has. The problem is upstream — the information is degraded, and the system provides no mechanism for the agent to recognize that. So the agent proceeds, confidently, producing output that looks correct but is built on inputs that were wrong.
The Failure Patterns
In my experience, this shows up in a few recurring shapes, and in each case the first symptom gets misread as a model problem.
The most common is the silent operational check. An agent running pre-deployment validation calls a security scanner. The scanner times out. The agent gets nothing back, marks the check complete, and flags the deployment as ready — not because the scan came back clean, but because it received no signal to the contrary. The eval suite ran with a working scanner and passed. Nothing in the test coverage accounted for the scanner being down.
Inventory management breaks differently. An overloaded stock API and an empty warehouse produce the same response: nothing. The agent reports no inventory, the downstream fulfillment system cancels pending orders, and tracing the problem back to a 503 takes longer than it should because the first hypothesis is always model behavior, not tool state.
Compounding decisions are the slowest to diagnose. An agent orchestrating a multi-step pipeline calls a dependency checker early in the workflow and gets an empty response on a transient failure. It skips a validation step it would otherwise have run. Every subsequent step executes on a false premise, and by the time the output surfaces, nobody is looking at a tool call from step two.
Why Current Evals Miss This
In every agent eval suite I’ve reviewed, evals run with all tools operational and measure whether output matches expected results. That tests model quality and prompt effectiveness — but it’s not sufficient for production deployment.
A system that has never been tested with degraded dependencies is not a system that handles degraded dependencies. It handles them in some unknown way that nobody has measured, and you find out what that way is when something goes down on a weekend.
The gap matters more as agents move into operational roles. An agent that retrieves the wrong document costs you an incorrect answer. An agent that proceeds with a deployment because a security scanner was unavailable costs you an incident review and, if you’re unlucky, a breach.
What Degraded-Dependency Evals Look Like
Two metrics belong in any eval suite meant to support production deployment.
The degraded-tool detection rate measures whether the agent recognizes tool failure for what it is, rather than treating absent data as meaningful. An agent that consistently interprets timeouts and empty responses as “nothing to report” scores low on this regardless of how well it performs when everything is working.
The graceful degradation rate measures what the agent does next. Does it halt and surface the problem — “I cannot complete this task because a required tool is unavailable” — or does it proceed on incomplete information and hand back output that looks complete? This is a product decision embedded in a reliability metric, and most teams haven’t made the decision because they haven’t seen the behavior.
To run these evals: inject failures into existing scenarios. Return error responses, timeouts, and empty payloads from specific tools during otherwise normal runs, and measure what the agent does at each injection point. The question isn’t whether the final output is correct — it’s what the agent does when a dependency goes down mid-task.
Where Hard Limits Break
Agents designed to halt on every tool degradation create their own problem. A 503 from an overloaded endpoint is often transient; an agent that escalates immediately on every failed tool call is going to escalate a lot, and that gets expensive fast if it requires human review.
The calibration between “halt on any degradation” and “proceed despite incomplete data” depends on what the agent is doing and what the cost of a wrong answer is versus no answer at all. That’s a product decision, and in my experience teams rarely make it explicitly — because they haven’t run the evals that would show them what their agent actually does in those situations. Once you’ve seen the agent report an empty warehouse or skip a validation step during a test, the question of where to set the threshold stops feeling abstract.
The eval your agent passed last Tuesday assumed all its tools were working. Add the one that doesn’t before Saturday.
