Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang

Introduction

In early September 2026 Cloudflare discovered that the GLM‑5.3 model hosted on Workers AI was leaking snippets of prior users’ responses when handling exceptionally long prompts. The issue stemmed from a misconfiguration in the open‑source inference engine SGLang, which caused its key‑value (KV) cache to be shared across concurrent requests. Although only 155 responses were affected, the incident highlighted the challenges of multi‑tenant isolation in server‑less AI deployments.

How the bug manifested

• Trigger condition – Prompts longer than 681,280 tokens caused the inference engine to spill over into a shared KV cache.
• Impact – Responses occasionally contained text generated for a different customer’s request that had been processed moments earlier.
• Scope – The problem was limited to the GLM‑5.3 model; no other Workers AI models exhibited the same behavior.
• Timeline – The bug was reported on September 8, reproduced on September 9, and the model was taken offline within 40 minutes. A patched version of SGLang was deployed later that day, and service was restored on September 9.

Remediation steps with SGLang

1. Configuration hardening – The SGLang deployment was updated to enforce request‑scoped KV namespaces, ensuring that each inference request receives an isolated cache instance.
2. Cache eviction policy – A stricter time‑to‑live (TTL) was introduced for KV entries, preventing stale data from persisting long enough to be accessed by subsequent requests.
3. Runtime isolation – The engine now spawns a lightweight sandbox per request, guaranteeing that memory and KV resources cannot be inadvertently shared.
4. Upstream collaboration – Cloudflare worked closely with the SGLang maintainers to merge these changes into the main codebase and to add automated tests that simulate extreme‑length prompts.
5. Monitoring enhancements – New observability hooks were added to flag any KV access patterns that exceed normal usage thresholds, enabling faster detection of similar anomalies.

Lessons learned and next steps

• Multi‑tenant safety – Even well‑tested open‑source components can behave unexpectedly under edge‑case workloads. Rigorous isolation testing must be part of any AI inference stack.
• Rapid incident response – The coordinated disclosure process, combined with Cloudflare’s internal incident playbook, allowed the team to contain the exposure within an hour of confirmation.
• Continuous hardening – Ongoing audits of SGLang configurations and regular fuzzing of prompt lengths are now part of the standard operational cadence for Workers AI.
• Customer communication – A transparent post‑mortem was shared with affected customers, and Cloudflare offered a dedicated support channel for any follow‑up questions.

Conclusion

The cross‑request KV cache exposure in the GLM‑5.3 model was a brief but instructive episode that reinforced the importance of strict tenant isolation in server‑less AI services. By tightening SGLang’s configuration, improving cache eviction, and enhancing monitoring, Cloudflare has restored confidence in Workers AI and set a higher bar for future model deployments.