Prompt Caching via Exact Prefix Preservation

Long-running agent conversations with many tool calls can suffer from quadratic performance degradation:

• Growing JSON payloads: Each iteration sends the entire conversation history to the API
• Expensive re-computation: Without caching, the model re-processes the same static content repeatedly
• ZDR constraints: Zero Data Retention (ZDR) policies prevent server-side state, ruling out previous_response_id optimization
• Configuration changes: Mid-conversation changes (sandbox, tools, working directory) can break cache efficiency

As conversations grow, inference costs and latency increase quadratically without proper caching strategies.

Maintain prompt cache efficiency through exact prefix preservation - always append new messages rather than modifying existing ones, and carefully order messages to maximize cache hits.

Core insight: Prompt caches only work on exact prefix matches. If the first N tokens of a request match a previous request, the cached computation can be reused.

Mechanism: Caching operates at the token level, not message level. The cache checks token-by-token for prefix matches, independent of message boundaries.

Message ordering strategy:

1. Static content first (beginning of prompt - cached across all requests):

   • System message (if server-controlled)
   • Tool definitions (must be in consistent order)
   • Developer instructions
   • User/project instructions

2. Variable content last (end of prompt - changes per request):

   • User message
   • Assistant messages
   • Tool call results (appended iteratively)

Configuration change via insertion:

When configuration changes mid-conversation, insert a new message rather than modifying an existing one:

[Static prefix...]
<sandbox_config_v1>     // Original config message
[Conversation...]

<sandbox changed>
<sandbox_config_v2>     // NEW message inserted
[Conversation continues...]

This preserves the exact prefix of all previous messages, maintaining cache hits.

What breaks cache hits:

• Changing the list of available tools (position-sensitive)
• Reordering messages
• Modifying existing message content
• Changing the model (affects server-side system message)

Provider variations:

• OpenAI: Caching requires an exact rendered-prefix match ("Cache reuse requires the entire rendered prefix to match"). Older models cache implicitly at model-dependent intervals; newer models also support explicit cache breakpoints (up to four cache writes per request), with cached tokens discounted up to 95% (reads at 0.1x the uncached input rate on most models)
• Anthropic: Explicit cache_control breakpoints plus automatic caching, a 5-minute default TTL with a 1-hour TTL option, cache reads at 0.1x the base input price (0.05x on Opus 5.5/Sonnet 5.5, 0.025x on Fable 5.1/Mythos 5.1), and a per-model minimum cacheable prefix length

Stateless design for ZDR:

Avoid previous_response_id to support Zero Data Retention. Instead, rely on prompt caching for linear performance:

Without previous_response_id:
- Quadratic network traffic (send full JSON each time)
- Linear sampling cost (due to prompt caching)

With previous_response_id:
- Linear network traffic
- But violates ZDR (server must store conversation state)

Green = Cached (exact prefix match) | Pink = Recomputed (new tokens)

Prompt construction checklist:

1. Order messages by stability: Static → Variable
2. Never modify existing messages: Always append new ones
3. Keep tool order consistent: Enumerate tools in deterministic order
4. Insert, don't update: For config changes, add new messages

Handling configuration changes:

MCP server considerations:

MCP servers can emit notifications/tools/list_changed to indicate tool list changes. Avoid honoring this mid-conversation as it breaks cache hits. Instead:

• Delay tool refresh until conversation boundary
• Or accept the cache miss as necessary trade-off

Compaction:

The Codex post that documents this pattern also describes a /responses/compact endpoint for auto-compaction: the client posts the conversation and receives a compaction item with opaque encrypted_content that summarizes prior turns while keeping the request stateless, and auto_compact_limit controls when compaction triggers. Compaction trades context for a cache reset — everything after the compacted summary is new tokens.

Implementation sketch:

function buildPrompt(state: ConversationState): Prompt {
  const items: PromptItem[] = [];

  // Static prefix (cached)
  items.push({ role: 'system', content: state.systemMessage });
  items.push({ type: 'tools', tools: state.tools });  // Consistent order!
  items.push({ role: 'developer', content: state.instructions });

  // Variable content (appended)
  items.push(...state.history);

  return { items };
}

function handleConfigChange(
  state: ConversationState,
  newConfig: SandboxConfig
): ConversationState {
  // DON'T: Modify existing permission message
  // DO: Insert new message
  return {
    ...state,
    history: [
      ...state.history,
      {
        role: 'developer',
        content: formatSandboxConfig(newConfig),
      },
    ],
  };
}

Pros:

• Linear sampling cost: Prompt caching makes repeated inference linear rather than quadratic
• ZDR-compatible: Stateless design supports Zero Data Retention policies
• No server state: Avoids previous_response_id complexity
• Simple conceptual model: Exact prefix matching is easy to reason about

Cons:

• Quadratic network traffic: JSON payload size still grows quadratically (only sampling is cached)
• Cache fragility: Mid-conversation changes (tools, model) break prefix matching
• Disciplined ordering required: All static content must come before variable content
• Tool enumeration complexity: Must maintain consistent tool ordering
• MCP server limitations: Dynamic tool changes can cause cache misses

• Unrolling the Codex agent loop | OpenAI Blog (September 11, 2026) — exact-prefix caching under ZDR, insert-don't-modify config changes, tools/list_changed cache misses, and the /responses/compact auto-compaction endpoint
• Prompt Caching Documentation | OpenAI — prefix-match rules, explicit breakpoints on newer models, cache pricing
• Prompt Caching | Anthropic — cache_control breakpoints, TTLs, read pricing, minimum cacheable length
• Codex CLI | GitHub
• Related: Context Window Auto-Compaction