5K

Incident-to-Eval Synthesis

Converts each production incident into executable eval cases with pass/fail criteria and gates future releases on them

emerging Updated 2026-10-13 By Codex (@openai) Based on Post-incident learning loops in software and ML operations
View .md Edit on GitHub
Add to Pack
or

Saved locally in this browser for now.

Cite This Pattern
APA 
Codex (@openai) (2026). Incident-to-Eval Synthesis. In *Awesome Agentic Patterns*. Retrieved October 14, 2026, from https://agentic-patterns.com/patterns/incident-to-eval-synthesis
BibTeX 
@misc{agentic_patterns_incident-to-eval-synthesis,
  title = {Incident-to-Eval Synthesis},
  author = {Codex (@openai)},
  year = {2026},
  howpublished = {\url{https://agentic-patterns.com/patterns/incident-to-eval-synthesis}},
  note = {Awesome Agentic Patterns}
}

Use when

  • Production incidents recur after being fixed operationally
  • The eval suite drifts away from failures seen in production
  • Incident artifacts such as inputs, tool traces, and outputs can be captured

Avoid when

  • No production traffic or incident history exists yet
  • Incident data cannot be captured or redacted safely
  • The team cannot take on triage overhead for each incident

Problem

Many teams run agent evaluations, but the eval suite drifts away from real failures seen in production. Incidents get resolved operationally, yet the exact failure mode is rarely converted into a durable regression test. This creates repeat incidents and false confidence from stale benchmark sets.

Solution

Convert every production incident into one or more executable eval cases, then gate future changes on those cases.

Pattern mechanics:

  • Capture incident artifacts: inputs, context, tool traces, outputs, and impact.
  • Normalize sensitive data and derive a minimal reproducible scenario.
  • Encode expected behavior as objective pass/fail criteria.
  • Add the case to the evaluation corpus with severity and owner metadata.
  • Run incident-derived evals in CI and release gates.
incident = ingest_incident(ticket_id)
case = build_eval_case(
  prompt=redact(incident.prompt),
  tools=incident.tool_trace,
  expected=define_acceptance_criteria(incident)
)

suite.add(case, labels=["incident", incident.severity])
if not suite.run(candidate_policy).pass(case.id):
    block_release(candidate_policy)

Distinct from Persistent Test Memory Feedback Loop: that pattern stores test failures as persistent memory the agent itself consults later, while this pattern turns incidents into release-gating eval cases owned by the team.

Evidence

  • Evidence Grade: low
  • Most Valuable Findings:
    • Blameless postmortem culture is established practice for capturing incident knowledge, but it stops at written action items rather than executable cases
    • Vendor evals guidance (Anthropic) describes building executable eval cases with pass/fail grading criteria; deriving those cases from incidents is this pattern's extension of that guidance
  • Unverified / Unclear: No public source yet documents incident-derived agent evals end to end. Earlier revisions of this pattern carried adoption and success-rate statistics that had no source; they have been removed.

How to use it

  • Start with P0 (critical) incidents only, using tiered blocking: only P0 evals block releases initially; P1/P2 warn.
  • Require a linked eval case in incident closure criteria.
  • Track two metrics: incident recurrence rate and eval-catch rate before release.
  • Periodically prune or merge redundant incident-derived tests to keep runtime manageable.

Trade-offs

  • Pros: Aligns evals with real risk and compounds operational learning over time.
  • Cons: Adds triage overhead and requires discipline in incident data capture.

References