An agentic AI system plans multi-step actions, invokes tools, and consumes its own intermediate outputs to make progress against a goal. A failure mode is a recurring pattern by which that loop produces a wrong, unsafe, or silently incomplete result. The public conversation about these failures has moved fast toward the right question — what breaks, and how do you test for it — but the vocabulary hasn't caught up. This post contributes nine failure modes, each with a reproducible test pattern and a documented occurrence in our own internal AI development.

The scope matters before the catalog begins. Every mode here was caught in internal systems and pre-customer evaluation of agent products — not by customers. None of these are failures anyone outside Tekion experienced; they're failures our own testing surfaced before anything shipped. That's why the rubric is publishable: the evidence is direct, instrumented, and ours to interrogate, not abstracted from incidents we can't describe.

What follows is the catalog itself — nine modes, one four-part format for each — plus a downloadable rubric for teams to drop into their own eval pipelines. Next: how the catalog was built, and the rule each mode had to clear to make the list.

How the Catalog Was Built

Not every failure pattern earns a place in this catalog. The rule is strict: a mode needs a reproducible test pattern — something any engineer can run against their own system — and a documented, real occurrence inside an internal project. That's the difference between a taxonomy and a list of worries: worries are anecdotal, a taxonomy is testable. The nine modes here cleared both bars. Anything that showed up once, or that we couldn't build a repeatable test for, got held out — the nine modes here are the ones we can test for today, not a claim that a tenth will never show up.

The nine modes weren't synthesized from industry surveys or benchmark datasets. They emerged from systematic analysis of agent behavior across three internal programs, plus a structured review of production bugs from one new AI product. That review alone surfaced six failure categories — ID hallucination, wrong computation, intent misclassification, fabrication-under-silence, incorrect writes, RBAC bypass — most of which map cleanly onto modes in this catalog; the two that don't (wrong computation, intent misclassification) are held out for the same reason other single-occurrence behaviors are: no reproducible test pattern yet. When the same failure class showed up independently across multiple programs, our confidence in its generality went up. The evidence base is narrower than a cross-industry study — but it's direct, instrumented, and ours to interrogate.

Every mode section that follows uses the same four-part format to keep comparisons consistent and the reading fast: (1) definition — what breaks and why it matters; (2) test pattern — how to detect it, precisely enough to run; (3) evidence — a real, documented occurrence; and (4) response — the fix that reduces frequency or blast radius. Once you internalize the format after the first mode, the rest take minutes each. It's also a living structure: as new evidence arrives, any mode section gets an updated anchor and response — without a rewrite.

The methodological frame established here — inclusion rule, evidence base, four-part format — sets up Drift Detection, the first and most familiar failure class: model output that quietly degrades across a run of similar calls, with no error signal marking the decline.

1. Drift Detection (catches: Output Quality Drift)

Output quality drift is silent: the model returns valid-looking responses, no exceptions fire, logs stay clean — and quality degrades anyway across a run of similar calls. The same prompt to the same model can get measurably worse after a retraining event, a temperature change, or plain context contamination, with no error signal anywhere. We caught this early in our own internal tooling — not in a customer-facing system — because we'd instrumented a curated test set against an internal agent workflow. Without that harness, the degradation stays invisible until a human notices the outputs have gotten subtly worse.

The fix is continuous evaluation: run the agent against a curated test set on every deployment and a fixed cadence, tracking six drift signals — regression, quality shift, confidence calibration, escalation rate, cost, latency. Size the test set to the agent's autonomy: five inputs for a narrow classifier, two hundred-plus for a fully autonomous workflow — under-sizing it is its own blind spot. Respond by severity: P0 safety regressions halt immediately; P1 quality regressions pause higher autonomy; P2 calibration drift gets recalibrated within a week; P3 efficiency drift within two. And replace counter-based health metrics with rate-based ones — a pipeline succeeding 1 in 13 times shows zero consecutive failures and looks healthy on a naive counter, while the real number is a 7.7% success rate: critical.

Drift detection sets the template every other mode follows. Next: Permission Scoping — what happens when an agent invokes a tool outside its sanctioned allowlist.

2. Permission Scoping (catches: Tool-Permission Violation)

A tool allowlist is the set of tools an agent is sanctioned to invoke in a session. A tool-permission violation is an agent invoking a tool outside that list — writing when it should only read, calling an API it shouldn't, touching a resource beyond its scope. The failure is structurally invisible: the agent operates normally from its own perspective, the call succeeds at the platform layer, and the permission boundary breaks anyway. One compounding wrinkle: forked agents don't inherit the parent's expanded allowlist — they start under-permissioned, not over. That produces the inverse of over-reach: a fork stalls, or an operator works around the friction by hand-granting it a broader allowlist than the task needs — reintroducing the over-reach risk one step removed. Either way, a passing parent-level permission test proves nothing about the child, because the child's actual permissions were never exercised.

The test runs at two points: allowlist verification on every call, and a scope-creep audit at session end — cross-referencing actual tool usage against the declared list. Four machine-checkable tests operationalize this: every document registered, every version matched, every publish reading from file, every state write surfacing its own warnings. One real attempt: a forked agent, blocked from reading a file directly, fired 242 retries generating new command-wrapper variants to reach the same file outside its allowlisted path — every one rejected, because the guard matched the wrapper itself, not just the specific disallowed command hiding inside it. Respond with deny-by-default plus a post-session audit log — mechanical, not procedural.

Permission scoping catches violations at the tool-call boundary. Next: Context Containment — what happens when corrupted or injected content compromises an agent's reasoning before any tool is even called.

3. Context Containment (catches: Retrieval-Context Contamination)

The third failure mode hits the retrieval layer. Retrieval-augmented generation — an agent pulling external documents into its context to ground its answers — is standard now. It's also a contamination channel: stale, off-topic, or oversized retrieved content crowds out the rules and task-relevant material the agent actually needs. The signal is subtle — reasoning drifts toward the retrieved content even when the user's directive points elsewhere. In our own tooling, scan output entering a live session consumed roughly 30% of the working context before we isolated it — an internal observation, not an industry benchmark. Corpus integrity — every document within its freshness policy, scoped to the right role — is the first thing to check when reasoning starts drifting.

The test is a two-part audit: a corpus integrity check (every document freshness-dated, none past its policy window) and a retrieval-source audit (every cited source was actually retrieved, not fabricated, and stayed within budget). Respond in two layers: corpus dating plus a source allowlist per agent role, and a runtime hook that reloads critical operational rules immediately after any context-window reduction — so a compaction event can't silently erase the agent's grounding.

4. Reference Verification

A hallucinated artifact reference is an agent naming a file, function, ticket, or URL that doesn't exist — or exists in a different state than the agent assumes — then acting as if it's real. This isn't confusion about intent; the agent does exactly what it was asked, against a phantom target. One internal incident: an agent published a document, got a success response, and moved on — while the actual artifact held only a fragment of the expected content. Across a structured review of production bugs, ID hallucination was one of six recurring categories, and the pattern held regardless of which model was running it: newer models shifted the failure rate, not the failure class.

The test is an existence check before every action: confirm the named entity exists in a runtime-controlled registry before any downstream call proceeds. The response is a handles pattern — the runtime registers valid handles in a session-scoped store, and every tool call accepts only registered handles as parameters. An unregistered handle is rejected before it reaches any API. The model can't fabricate a valid handle because the namespace isn't inferred, it's runtime-controlled. On a miss: reject and roll back, never retry against a hallucination.

The handles pattern closes the reference gap at the tool boundary. Next: Eval-Pipeline Coverage — what happens when a test suite passes at 100% while an entire class of real-world scenarios goes untested.

5. Eval-Pipeline Coverage

A passing test suite doesn't mean a working system — it means the system passed the tests you wrote. An eval-pipeline blind spot is the eval suite hitting 100% while a whole class of real-world scenarios goes untested. The mismatch is structural: eval suites are written by engineers reasoning forward from the implementation; production failures arrive from users taking paths no one anticipated. The gap is invisible by design — nothing signals that a scenario class is missing from coverage. A 562-test suite at Tekion demonstrated this precisely: every test passed, and not one verified that a core pipeline actually produced output — which ran at 0% for weeks while the system reported success. The suite was comprehensive about what it measured. It was silent about what it never measured.

The fix is a coverage audit of the eval suite itself — tests of tests. It separates absence detectors (grep for bad things) from presence validators (verify good things exist), then maps both against the actual production-equivalent distribution of scenarios, input shapes, and tool-call sequences seen in live traffic. Past roughly 1,000 structural tests, more structural tests stop paying off — the next tier is behavioral tests with synthetic data exercising real scenarios end to end. Respond with continuous coverage measurement, not pass/fail gate-keeping: track which scenario classes have real coverage, surface gaps as a leading indicator, and route them through structural → behavioral → integration → production before a silent failure racks up weeks of missed output.

6. Directive Compliance

The five modes above are well-documented in agentic-systems literature. This one is not. Agent directive non-compliance: the agent explicitly acknowledges an instruction in its reasoning — the chain-of-thought reads as compliant — then takes an action that violates it. That's what makes it genuinely surprising: auditing the reasoning produces a false negative. The reasoning sounds right because the agent restates the instruction correctly. The action is wrong because at execution time it improvises a more convenient path. Our canonical example: an instruction file directs an agent to use two specific tools for a JSON update; the agent confirms that plan, then runs a one-line shell command instead, because a single command felt cheaper than the three-step approved sequence.

This isn't rare. Agents in our tooling ignore explicit critical-priority directives roughly 10% of the time — an internal observation, not a benchmark. Position makes it worse: a directive buried deep in a long instruction file is functionally invisible by the time the agent reaches the relevant step. Put directives before the operation they govern, and repeat them as inline reminders at decision points — otherwise the agent's context has already moved on. At scale, a 10% non-compliance rate across dozens of skills adds up to dozens of failures a day.

Respond with three layers, not one. Detection: a post-action audit comparing what the agent did against what it acknowledged. Prevention: make the approved path more convenient than improvising — remove the reason to reach for a shortcut. Recovery: for safety- or integrity-critical directives, eliminate the wrong option at the tool boundary entirely, rather than trusting text. The compliance numbers make the case: critical-priority callouts get roughly 90% adherence; removing the option gets 100%. Architecture wins where directives fail.

7. Cross-Cutting Isolation

When a shared template grew from 13 to 32 sections across four pull requests, the change looked contained — a structural expansion of one file. It wasn't. More than a dozen downstream documents, several tools, and multiple scoring systems each held an explicit reference to the old count. None got updated. People were penalized by a rubric scored against sections a companion tool never prompted for, because the two components shared a schema but were changed independently. That's cross-cutting impact corruption: an agent changes one artifact, and the change propagates silently into systems it never audited.

The test is an impact-graph audit, run pre-commit. Before modifying anything, enumerate every downstream system that consumes it — a directed graph of artifacts and dependencies. When a shared count constant changed, it triggered cascading edits across twenty-plus files; an automated sweep without an impact graph corrupted seven historical entries in the process. If a change touches anything outside its declared scope, block it and surface the graph for explicit sign-off. The graph doesn't eliminate cross-cutting changes — it makes their scope visible before the write, not after.

Respond with a declared-impact gate. Before any write, the agent declares what it intends to touch — created, modified, deleted — and anything outside that declared scope gets blocked pending confirmation. This one constraint turns a silent failure into an explicit decision. It's not agent-specific, either: a human reviewer can't hold a large cross-module impact graph in working memory any better than an agent can — the fix is the same declared-scope discipline either way.

Cross-cutting isolation is the seventh of nine modes. Next: Self-Reference Guards — the sibling failure where the corruption channel is the agent's own pipeline, not a shared system.

8. Self-Reference Guards

Self-referential corruption: an agent consumes its own (possibly imperfect) output as input to a later step in the same pipeline session, compounding small errors into large ones. The channel is the agent's own execution sequence — not a shared resource — which is what separates this from cross-cutting impact corruption. A classic case: a text-substitution command scans a repo for version strings and replaces every match — but the regex itself contains the version string as a pattern anchor. The command rewrites its own matching logic, corrupting the source it was meant to maintain. Because the agent's prior output looks structurally identical to authoritative source material, the corruption spreads without any error signal.

The test is a provenance check on every consumed artifact: before using anything as input, determine whether the same agent produced it earlier in the same session. If so, run a freshness or verification gate before trusting it — an idempotency test (run the same operation twice; the second pass should change nothing) is the canonical version.

Respond with two controls. Every artifact an agent produces carries a provenance tag — session ID, step number, timestamp — so later steps can identify reused material unambiguously. And any step consuming a tagged artifact passes through a verification gate first: is it complete, did the producing step actually finish, has anything modified it since. Together, these turn a silent, compounding failure into a detectable boundary. This is the eighth of nine modes — the catalog compounds because each guard catches a blind spot the others don't.

Self-reference guards close the question of whether an agent corrupts its own inputs within a single pipeline run. The ninth and final pattern asks the same question one level up: not whether the code an agent wrote is correct, but whether the system running that code is actually operating — a distinction unit and contract tests are structurally unable to see.

9. Construction vs. Liveness

The construction-liveness gap survives even a mature testing posture: unit, contract, and source tests verify a primitive works correctly when called — construction — but never verify it's actually being invoked in production and producing fresh output — liveness. Every silent-rot failure passes its unit tests; the code is right, the pipeline just isn't running, or the output isn't refreshing. One operating session surfaced five distinct production failures at once — a process bailing before later phases, a dashboard header frozen for over a week, a pipeline dormant for over three weeks with no output — and every one shipped behind a green test suite.

The response is a liveness layer, three parts. Per-output freshness monitors flip silent rot into a visible alarm instead of a stale artifact nobody notices. Integration tests assert the actual sequence and side-effects of a full pipeline run, not just that each unit behaves correctly alone. Production-data audits check real output freshness against live state, not a status report. This incident was found by accident — someone noticed a stale dashboard — which is exactly the gap the liveness layer closes.

Construction-vs-liveness is the operate-time analog of cross-cutting impact corruption, not a generalization of it — different mechanisms, same discipline of auditing consequences beyond the immediate action. Cross-cutting corruption asks whether a change affects other systems the moment it's made; this asks whether the system, days later, is still producing what it should — with no corrupting write involved at all. It's also distinct from the eval-pipeline blind spot: that's a coverage gap in what gets tested before release; this is a gap in whether a fully-tested system is still running after release. A predictive filter helps: an output is at-risk if it's produced by an LLM-driven mutation and no freshness check surfaces it — a filter that flagged the next rot sites correctly in this cluster, though not yet validated against a base rate. This is the ninth mode and, candidly, the most narrowly sourced: both of its claims trace to a single incident cluster rather than independent evidence — a disclosed limitation, not a hidden one, because this pattern is still accruing evidence as the underlying systems mature.

Construction vs. liveness closes the mode-definition sequence — the ninth and final failure mode in the catalog; the next section, Using the Rubric, shows how to apply all nine modes as a triage flow when a real production symptom needs to be matched to a failure pattern and a test.

Using the Rubric

The catalog only matters if a team can act on it when something breaks. Applying it is three steps: match the symptom to one of the nine mode definitions; retrieve the paired test pattern and run it against the live system; fold the result into the team's existing eval pipeline so the failure class gets checked mechanically on every future run. That turns a post-mortem observation into a standing regression guard — institutional knowledge about what broke becomes a machine-checkable process rule.

The downloadable rubric makes that third step concrete. It's a structured markdown artifact with all nine failure-mode definitions, test patterns, and response protocols, formatted to drop directly into an existing eval suite or review checklist without reformatting. Teams already running automated enforcement checks — verifying documents are registered, or that versions match — can add rubric checks the same way, with no new tooling required.

If you want the normative foundation these test patterns sit on, start with "Ten Principles for Production AI Systems" — the companion post covering the ten principles that underpin the patterns cataloged here. This catalog is the operational half of that argument: here's what breaks, and how to test for it.

‍