The Reliability Science of Agentic Systems: Why Your Retry Loops Are Exhibiting Integral Windup

You shipped an agent that scores 78% on your evaluation harness. In production it completes 45% of tasks correctly on the first attempt. By the fifth retry it's making things worse — not better. This isn't bad luck. It has a name, a mathematical explanation, and a 50-year body of engineering solutions. Your agent has integral windup.

1. The 37% Problem: Why Benchmarks Don't Predict Production Performance

Widely-reported industry estimates put the benchmark-to-production gap at 30–40%. That figure isn't peer-reviewed, but it rhymes with what controlled experiments actually measure. The first systematic attempt to study this gap carefully is Towards a Science of AI Agent Reliability (Rabanser, Kapoor et al., ICML 2026, arXiv:2602.16666). It defines 12 metrics across four dimensions — consistency, robustness, predictability, and safety — and and tests them across a variety of agent tasks.

The paper measures degradation across its four dimensions under real-world conditions. Two findings stand out:

• Robustness testing shows measurable degradation from semantic perturbations alone — rephrase the task without changing its meaning and a representative agent measurably underperforms. The gap appears even before introducing tool failures or context pressure.
• Consistency degrades significantly under repeated execution of the same task across independent runs. The agent that passed your eval on Monday is a different agent by Friday, even with no model change — a failure mode that summary metrics won't show.

The structural reason is simple but underappreciated: benchmark environments are stateless, isolated, and single-turn. Production agents operate in long, stateful, noisy environments where the model's prior actions pollute its context for the next steps. The benchmark never tests what happens on the third retry after two partial failures.

If you're still trying to close this gap by running more evals, you're solving the wrong problem. The gap is architectural. Enterprise LLM meta-evaluation covers how to build eval infrastructure that at least detects the gap — but detection isn't the same as fixing it.

2. What “Working” Means for an Agent in Production

We don't have SLOs for agents. Not real ones. We have evaluation harnesses that measure performance on a sample at a point in time, and we have vague intuitions about “acceptable” behavior. We do not have formal commitments that answer: what is the error budget for this agent? How much task-completion-rate degradation triggers an incident? At what consistency threshold do we roll back a prompt change?

For a traditional service, “working” is unambiguous: latency at the 99th percentile, error rate per unit time, availability as a percentage (99.9%, 99.99%, and so on). These are numbers you can alert on, trend, and hold teams accountable to.

The agent equivalent needs at least these dimensions:

• Task completion rate at retry depth 0: what fraction of sessions succeed without a retry. Aggregate completion rate — which pools first-attempt successes with eventual successes at retry depth 5 — hides windup entirely. The number you care about is first-attempt success rate, tracked separately.
• Consistency across independent runs at the same retry level: given the same task at the same retry level, does the agent produce the same kind of output. Measuring consistency only at the first attempt hides the fact that your agent may be very consistent when fresh and completely unpredictable after one failed retry.
• Bounded failure modes: when the agent fails, does it fail in ways that are recoverable and don't cause downstream side effects
• Drift rate: how fast does performance degrade between model updates, prompt changes, or simply the passage of time as the task distribution shifts

Microsoft's SkillOpt work explicitly names uncontrolled skill evolution as “a major obstacle on the path from agent prototype to dependable, production-grade deployment.” That observation is correct, and the absence of formal SLO frameworks for agents is the industry's most significant operational gap right now. We are running production systems without being able to articulate what “production-grade” means for them. This is where the 30–40% gap actually lives — not in model quality, but in the absence of engineering culture around agentic systems.

3. Control Theory as a Mental Model for Agent Reliability

PID controllers are one of the most thoroughly understood feedback systems in engineering. They appear in HVAC, cruise control, industrial robots, and power grids. They're also an almost exact mental model for what an agentic reasoning loop is trying to do.

A PID controller computes a control signal from three terms: the proportional response to current error, the integral of accumulated past error, and the derivative that anticipates future error from the current rate of change.

An agent's reasoning loop maps onto this cleanly:

• P — Proportional: the model's response to the current task state and immediate error (“the tool returned a 404; try a different endpoint”)
• I — Integral: the accumulated context window — conversation history, prior tool outputs, failed attempts, everything the model has seen so far
• D — Derivative: the model's anticipation of downstream consequences (“if I delete this record now, the next step will fail because the foreign key check will fail”)

The insight isn't that agents are PID controllers. The insight is that the failure modes of PID controllers have been studied for eighty years, with mathematical rigor, and most of those failure modes show up in agents in recognizable form. We already know the solutions. We just haven't applied them.

4. Integral Windup and the Retry Loop Problem

Integral windup is what happens when the integral term of a PID controller builds up without a limit. The classic scenario: the output device (called “the plant”) is at its physical limit — it cannot respond to commands — so the error never reaches zero, and the integral keeps growing. When the limit clears, the controller overcorrects badly, because it has been told, through all that accumulated history, that it has been failing for a very long time.

In agents, the mechanism is this: each failed retry appends more error context to the model's input. The model can see that it failed. It modifies its approach — often in ways that make the task state worse, introduce new failure modes, or pollute the context with partial and inconsistent intermediate results. Subsequent attempts inherit this contaminated context. The model is no longer reasoning about the original task. It's reasoning about the original task plus the history of its own failures.

The data from arXiv:2605.08563 quantifies this directly: each failed retry raises the probability of error on the subsequent attempt by 7.1×. That is not linear degradation. That is exponential growth in errors. That is integral windup — measured, in LLM agents, with the same mathematical pattern that control engineers have been studying in physical systems for decades.

The AI field calls this phenomenon “context contamination” or “infinite agentic loops.” Those framings are descriptive. They tell you what happened. The control-theory framing is prescriptive — it tells you what to do about it, because the anti-windup literature is mature. Nogueira and Skogestad (arXiv:2606.30877) apply Advanced Regulatory Control — an engineering discipline that includes anti-windup techniques — to limit LLM agent behavior in process control environments. Control engineers are already applying this framework to agents. The bridge between control theory and agentic systems isn't an established practice yet — but the behavior it describes is measurable and the engineering fixes apply directly.

The translation of anti-windup strategies to agent design:

5. Failure Mode Taxonomy: Where Agentic Reliability Breaks Down

The Rabanser et al. framework gives us a cleaner taxonomy than “the agent didn't work.” Four dimensions, twelve metrics. Here's what each dimension looks like as a production failure:

Consistency failures

The agent produces meaningfully different outputs for inputs that mean the same thing. Caused by the model's built-in randomness, but also by context-window state that varies between sessions (cached vs. fresh), tool output variance, or prompt sensitivity that was never measured. Dangerous because it makes the system unpredictable in ways that don't show up in summary metrics. You won't see it in your average numbers; you'll see it in customer support tickets.

Robustness failures

8.8% degradation from rephrased inputs alone. In practice: a user rephrases a request slightly, the agent takes a completely different path. Harmless on its own; catastrophic in multi-agent pipelines where the next agent receives an unexpected output format, missing fields, or a slightly different meaning for a shared piece of data.

Predictability failures

The hardest one to detect. The agent performs poorly on tasks that differ from your test set. You discover this when you expand to a new customer segment, a new language, or a new use case, and reliability collapses — without any model change, without any prompt change. Your test set was not representative, and you didn't know it.

Safety failures

The agent takes actions outside its intended scope. In long-horizon tasks this usually manifests as goal drift — the agent, in pursuit of an immediate sub-goal, makes decisions that undermine the original objective. Silent goal drift in multi-agent systems covers the full taxonomy of how this propagates across agent boundaries and why it's structurally harder to detect than a straightforward failure.

The practical implication of this four-way taxonomy: a single task completion rate metric obscures three of the four dimensions entirely. You need separate instrumentation for each.

6. Observability for Agentic Systems: What to Instrument

Most agent observability setups I've seen instrument tool call latency and total LLM token spend. That tells you when things are slow and expensive. It does not tell you when things are wrong.

The minimum set of things to track for production AI agent reliability, in priority order:

1. Per-session trajectory. The full sequence of steps — what the agent saw, what it did, and what it got back — for every session. Not just the final output. You need the path: which branches were taken, which tools were called in what order, and where the agent went off track. Without this, any post-incident analysis is guesswork.
2. Retry depth histogram. Distribution of how many attempts before success or abandonment. A healthy agent shows a sharp drop-off: most tasks complete first-try, a small fraction need one retry, very few need more. An agent with windup has a heavy tail — tasks that needed five retries are almost as common as tasks that needed two. That tail is expensive and indicative of systematic failure.
3. Context size change per retry. How many tokens are added to the context on each retry attempt. If this grows with every retry, you are accumulating windup. This is the single most direct measurable sign of the failure described in section 4.
4. Sub-goal completion rate by retry level. Which sub-goals are failing, and at what retry level do they start failing. Falling sub-goal success at level 3+ is an early warning of windup — the agent is no longer failing on the hard parts; it's failing on things it completed successfully on the first attempt. That's not a task difficulty problem. That's accumulated context degrading reasoning ability.
5. How similar outputs are across reruns at the same retry level. Falling similarity at retry level N versus level 0 signals that the agent's behavior is being shaped by the history of previous attempts rather than the current goal. It's no longer responding to the task. It's responding to the record of its own failure.
6. Terminal state classification. Every session ends somewhere. Classify it: success, graceful failure (agent recognized it couldn't proceed), panic stop (hard timeout or error limit hit), infinite loop detection, or human escalation. The ratio across these categories is your headline reliability metric and your primary SLO input.

The context management patterns that make this instrumentation tractable at scale are covered in context engineering for long-running agents. The short version: structured context with explicit checkpoints makes trajectory logging a natural byproduct of the agent's own state management.

7. Designing Convergent Retry Strategies

The default retry loop in most agent implementations: on failure, append the error to context, retry. This is not a retry strategy. It is guaranteed to exhibit windup for any failure mode that isn't immediately correctable by appending error text — which is most failure modes.

A convergent retry strategy has four structural properties:

Bounded context accumulation

Context from failed attempts is summarized, not appended verbatim. The model on retry N sees: “Previous attempt: tried [action]. Result: [error]. Relevant state: [key facts].” It does not see the full token stream from the failure. This is back-calculation — a reset technique from control theory. The total context size does not grow with each retry.

State isolation between retries

Each retry begins from a known-good checkpoint, not from whatever state the previous attempt left behind. If the agent modified external state before failing — wrote a partial record, sent a partial API call, acquired a lock — you need rollback capability before the retry can be valid. Retrying with dirty external state is like running calculations on corrupted sensor data.

Decreasing scope on retry

Each failed attempt should narrow the target sub-goal rather than re-attempt the full task. Break it down on failure; isolate the failing sub-goal; retry only that sub-goal with a clean context window. This is similar to “gain reduction” in control theory: when you're uncertain, don't give the agent full authority. Narrow the scope until you have evidence the agent can succeed at a broader level again.

Hard depth limits with explicit escalation

The retry counter must have a hard limit. An agent without a depth limit is an unstable system by design — not a reliability risk you manage, but a problem you have built in. Set N, enforce N, escalate at N with structured context: what was attempted, what failed, what state is known-good. Do not let the agent continue past saturation in a degraded state where outputs look plausible but accumulated context has undermined its reasoning. The cost of a visible, explicit failure is almost always lower than silent divergence surfacing three days later in a customer complaint.

The key rule: total context tokens at retry N should stay roughly constant — not grow with N. If your context grows with each retry, you have built a system with no anti-windup protection. You will get windup.

8. Closing the Benchmark-to-Production Gap: An Operational Checklist

The 30–40% benchmark-to-production gap does not close by running more evals. It closes by treating your agent as a control system with known failure modes, and engineering your infrastructure to address those failure modes explicitly.

Before deployment

• Define task completion rate (at depth 0), consistency threshold, and retry depth budget as explicit targets before you ship. Your depth budget is your integrator's saturation point — if you haven't decided where it is, you've designed an unbounded system.
• Run your test set with rephrased inputs — paraphrase tasks, reorder steps, vary surface formatting. If performance drops more than 5% on paraphrased inputs, your agent is sensitive to how tasks are worded before you've even shipped. Production will generate exactly these variations, and each one is a potential windup seed.
• Characterize the failure mode distribution across all four dimensions — and specifically identify which failures are windup-driven (occur at depth 2+ but not at depth 0) versus first-attempt failures. These require different interventions: first-attempt failures are model problems; windup-onset failures are infrastructure problems.
• Implement bounded context accumulation in your retry loop. Audit every code path that could append to context on retry and replace verbatim appends with structured summaries.

In production

• Track the retry depth histogram and how sessions end as your main metrics. Alert when the tail of the retry histogram grows or when the ratio of graceful failures to panic stops shifts. These are your early warning signs of windup.
• Log full session trajectories with retry depth annotations. Windup onset is invisible in aggregate metrics — it only shows up in the per-session trajectory of token accumulation and sub-goal failure sequence. You cannot detect windup post-hoc without a complete record of what the agent accumulated.
• Enforce the depth limit you set. Human review, a simpler model fallback, or a controlled failure — any structured escalation is correct. An agent past its depth limit is past its hard stop; it is in a mathematically unstable state.
• Periodically re-run a consistency sample at the same retry level — same tasks, same level, compared weekly against your baseline. Consistency that falls only at level 2+ while level 0 stays steady is the signature of growing sensitivity to retry history in your system. Investigate before it reaches level 0.

Ongoing

• Treat prompt changes as production deployments. Use a gradual rollout (canary deployment), monitor your key metrics, roll back if the retry depth histogram shifts. Prompt changes are code changes to a live system. Act accordingly.
• Track sub-goal failure rates broken down by retry level. Windup shows up as falling sub-goal success as the retry level increases, not as uniform failure across all levels. If you track only overall sub-goal failure rates, windup is invisible until it's catastrophic — summary metrics stay flat while per-level sub-goal success collapses.
• Review the memory and state architecture for potential windup points. Any system that accumulates context across failures without a structured reset mechanism is a windup risk. Memory is the new state management covers the architectural patterns that make this tractable.

We now have a rigorous scientific framework for agent reliability (Rabanser et al.), we have empirical evidence of the exact failure mechanics (the 7.1× error compounding), and we have a 50-year-old engineering discipline that solved analogous problems in physical systems. The conceptual bridges exist. What's been missing is the engineering culture to apply them — and the willingness to call a windup problem a windup problem, rather than assuming the next model version will fix it.

It won't. Design your systems accordingly.