Groundhog Day for AI agents: Prompt caching patterns for using OpenAI models on Amazon Bedrock

Prompt Caching for AI Agents on Amazon Bedrock

When an AI agent talks to a large language model (LLM) it must resend the entire conversation history on every turn—tool definitions, developer instructions, prior tool calls, and their results. This repeated “read‑again” behavior quickly inflates token usage and latency, especially for long sessions. Amazon Bedrock’s prompt caching solves the problem by storing the model’s pre‑filled key‑value (KV) state for any unchanged prefix of the prompt and reusing it on subsequent calls. Below is a practical guide to applying both explicit and implicit caching patterns so your agents stay responsive and cost‑effective.

---

1. What an Agent Sends on Each Turn

| Component | Description | Change Frequency | |-----------|-------------|------------------| | Tool list | JSON definitions of functions the agent can call. | Rare (new release). | | Instructions | System prompt with policies, persona, examples. | Rare (release). | | History | All prior turns: user messages, tool calls, and results. | Grows each turn. | | New turn | Latest user input or tool result. | Every request. |

Without caching, the model re‑processes the entire prefix on every turn, paying full token cost for each repeat.

---

2. How Prompt Caching Works

1. Prefill phase – The model tokenises the input and builds KV vectors for every token.
2. Cache store – Bedrock can persist this KV state when the same token sequence appears again.
3. Reuse – On a later request that starts with the identical prefix, Bedrock loads the saved KV state and only processes the new suffix.

Two modes control where the cache boundary lies:

• Explicit – You mark a `promptcachebreakpoint` in the payload. Ideal for static sections like tool lists and developer instructions.
• Implicit – Bedrock automatically determines the longest reusable prefix. Useful for growing conversation histories where the exact breakpoint isn’t known in advance.

Both can be combined: explicit for the immutable start, implicit for the evolving middle.

---

3. Caching the Static Prefix (Explicit Mode)

```python KEY = "support-agent:v7:gpt-6-astra" INSTRUCTIONS = "Your full support policies and examples..."

resp = client.responses.create( model="us.openai.gpt-6-astra", promptcachekey=KEY, promptcacheoptions={"mode": "explicit", "ttl": "30m"}, tools=TOOLS, # static tool definitions input=[ { "role": "developer", "content": [{ "type": "inputtext", "text": INSTRUCTIONS, "promptcache_breakpoint": {"mode": "explicit"} }] }, {"role": "user", "content": task} ] ) ```

• Key groups requests that share the same cached prefix.
• The breakpoint tells Bedrock to stop caching after the instructions block, so only the user task is charged each turn.
• The cached prefix must be at least 1,024 tokens to qualify for storage.

Tip: Ensure the serialized text of tools and instructions is deterministic (same ordering, whitespace, and quoting) so the cache hit works reliably.

---

4. Extending Caching Through the Conversation (Implicit Mode)

When the conversation grows, you can let Bedrock manage the cache boundary automatically:

```python def askwithimplicitcache(history): return client.responses.create( model="us.openai.gpt-6-astra", promptcachekey=KEY, promptcache_options={"mode": "implicit", "ttl": "30m"}, tools=TOOLS, input=history # full turn list )

history = [ {"role": "developer", "content": INSTRUCTIONS}, {"role": "user", "content": task} ]

Subsequent calls add new turns to `history`.

resp = askwithimplicit_cache(history + [{"role": "assistant", "content": "..."}]) ```

Bedrock detects the longest unchanged prefix (typically the static tools + instructions) and reuses its KV state, charging only for the new turn’s tokens. This pattern works across sub‑agents: each helper can share the same `promptcachekey` to start from the same cached base while maintaining its own evolving history.

---

5. Practical Recommendations

| Situation | Recommended Cache Strategy | |-----------|-----------------------------| | Static agent definition (tools, policies) | Use explicit caching for the entire block; set a TTL matching your release cycle. | Long multi‑turn sessions | Combine explicit (static prefix) with implicit (dynamic conversation) to keep the cache boundary optimal. | Spawning helper agents | Share the same `promptcachekey` for the common prefix; each helper maintains its own history after the breakpoint. | Frequent minor variations (date, user ID) | Keep variable data after the cached region; otherwise the cache will miss.

Monitoring cache hit rates via Bedrock metrics helps you fine‑tune TTL values and identify when a prefix has grown beyond the 1,024‑token minimum.

---

6. Closing Thoughts

Prompt caching transforms the “Groundhog Day” experience of AI agents into a smooth, cost‑aware workflow. By explicitly marking immutable sections and allowing Bedrock to implicitly manage the rest, you reduce token spend, lower latency, and keep agents consistent across sessions and helper instances. Apply these patterns early in your agent design to reap the performance and budget benefits throughout the lifecycle of your applications.