GLM-5.3 on Workers AI: What the KV Cache Tenant Isolation Leak Means for Your LLM Security Checklist

By James Hartley, IT Insights — Security desk. Editorial contact: contact@itinsights.com

Prompt text surfacing inside somebody else's model output is a failure mode that firewalls, WAF rules and secrets scanners will never see, because the memory involved is genuinely shared by design. Cloudflare's post-incident report on @cf/zai-org/glm-5.3 documents exactly that scenario: a cross-request KV cache leak inside a production inference fleet. The affected service is GLM-5.3 on Workers AI, the exposure window ran from August 27 to September 9, and 155 responses carried data from other customers — every one of them produced from a prompt longer than 681,280 tokens.

The write-up was published on October 26, 2026 as a coordinated disclosure alongside a post from the SGLang project, the open-source inference engine Cloudflare runs underneath the model. For security and platform owners, the value is not the news itself but the mechanics: this is a rare, fully documented example of how tenant isolation breaks in disaggregated LLM serving, what evidence exposes it, and which controls stop it.

Incident snapshot for triage reporting

Field	Detail
Service involved	GLM-5.3 / GLM 5.3 served through Workers AI as @cf/zai-org/glm-5.3
Exposure class	Cross-request KV cache read: fragments of other customers' prompts and partial generations appeared inside responses
Window	Present from launch on August 27 until restoration on September 9
Volume	155 responses, all for prompts above 681,280 tokens
Reporter and channel	Gadi Evron, security researcher at Knostic, submitted through HackerOne on September 8, 2026
Other models	None affected; Cloudflare states no other Workers AI model reached the rejection path, and other catalogued models such as GLM 5.3 Flash and DeepSeek V4 sit outside the bug's blast radius
Exploitation evidence	None found; once Cloudflare's own testing and the researcher's activity were excluded, the flagged requests matched ordinary long-context use in timing and volume
Current state	Remediated — corrected configuration, patched engine, upstream fixes shipped in v0.5.20, v0.5.21 and v0.5.22

From report to restoration: the response path

Z.ai's GLM family is a text generation model aimed at agentic coding workflows, and it is one of many models available through Workers AI. Evron's HackerOne report stated that very long prompts sent to @cf/zai-org/glm-5.3 sometimes returned text belonging to other users' sessions. Cloudflare reproduced the behaviour the next morning and took the model offline within 40 minutes of declaring an incident; a corrected configuration plus a patched inference engine then went out, and service was restored the same day, with the model back in public use on September 9.

Confirming that no other model hosted on Workers AI was affected mattered as much as the fix itself, because it let the provider scope the incident to a single deployment rather than the whole serving platform.

Two overlapping issues, one leak path

The cross-tenant exposure was the product of two overlapping issues that only became dangerous together. Neither one alone would have leaked a byte.

Defect one: capacity that did not match the advertised window

GLM 5.3 was deployed with a 1,048,576-token context window, while the KV cache pool on the prefill GPUs only had room for 681,280 tokens. Cloudflare knew about the mismatch before launch. The expectation was that the engine would simply return an error for any prompt it could not process, and because pre-launch long-context benchmarking returned successful HTTP status codes and a streaming body with no error frames, the gap was judged benign. The real testing hole was what was never measured: response codes and throughput were validated at long contexts, but model output and coherence were not.

Defect two: a rejected request treated as an accepted one

The second failure sat in how SGLang handled over-length requests in disaggregated prefill/decode mode. A prompt that did not fit the pool was internally marked as rejected and swapped for a one-token placeholder, but instead of aborting the request and returning a client error, prefill generated the KV cache for that placeholder and transferred it to the decode node. The decode node received a "transfer complete" message and assumed the pool pages had been overwritten with the new prompt's cache; in reality most of the previous state stayed intact. Generation then ran over stale cache pages that still held earlier requests' data.

Architecture context: why shared KV pages are a trust boundary

Workers AI runs GLM 5.3 with SGLang in a disaggregated layout. One pool of prefill GPUs processes the prompt and builds the model's KV cache; a second pool of decode GPUs generates the response, and the cache crosses between them over remote direct memory access (RDMA) via Mooncake.

Inside the GPU, the KV cache is a shared pool of fixed-size pages. A request receives pages when it is admitted and releases them when it finishes. The pages are not currently zeroed between requests — the next request that lands on a page simply overwrites it. That design is common among inference engines and is necessary for performance, but it pushes ownership tracking onto the engine. Ownership has to account for shared use inside a single node (offload and token generation touch the same memory with different lifetimes) and for a reference that is technically shared with the remote prefill node, which holds a simultaneous mutable reference to the page.

The critical operational point for defenders: pool size is not a configuration value. SGLang sizes it at startup from whatever GPU memory remains after model weights load, while the advertised context length is a separate setting. Nothing forces the two numbers to agree. Checking that they do is the operator's responsibility, and the chosen pool size is visible in the startup logs and as a metric.

Step by step: how a rejected request still answered the client

On a single-node deployment the rejection path was harmless: the placeholder ran, the batch finished, and the client saw an error. In disaggregated mode, the admission queues on both roles checked whether a request was finished rather than whether it had been rejected, and a rejected request still reported itself as live.

1. Bootstrap. The router informed the decode node of the prompt length (8 pages in the accompanying diagram), and decode reserved 8 KV pages for the request.
2. Prefill admission. The prefill admission queue admitted anything whose finished() flag was False. A rejected request stored its status in to_finish rather than finished_reason, so the flag stayed False and the request was admitted anyway.
3. Prefill forward pass. Rejection had already substituted the one-token placeholder, so prefill ran a trivial forward pass on that single token.
4. KV transfer. Prefill pushed one page of KV to decode and signalled that the transfer was complete.
5. Decode admission. The decode admission queue used the same finished() test and therefore accepted the handoff.
6. Decode attention. Generation began, and attention read all 8 reserved pages. Only page P0 had been written by this request; pages P1 to P7 still contained what earlier requests had left behind — other users' prompts and partial generations.
7. Response. The client received HTTP 200 instead of HTTP 400. prompt_tokens matched the long prompt and the reply read as coherent. It arrived faster than an 8-page prefill should take, and it carried content from other users' KV pages.

Every request above the threshold behaved this way. The underlying bug had already been reported to SGLang by another user as issue #30233 on July 6, 2026, and Cloudflare only found that report while investigating this incident. Issue classes like this had been flagged to the SGLang team since v0.5.10; the security incident accelerated work already under way. The patch landed in SGLang v0.5.20, with follow-ups in v0.5.21 and v0.5.22, and is available to every other operator dependent on SGLang.

GLM 5.3 was the only model hit. Every other disaggregated deployment on Workers AI runs KV pools larger than its context length on both prefill and decode, so no request could reach the rejection path. The engine fix and the hardening described below have been applied to every model served with SGLang, and deployment checks now require them for every new deployment.

Disclosure timeline (all times UTC)

When	What happened
Aug 27, 2026	GLM 5.3 went live on Workers AI with a prefill KV pool smaller than its context window; the exposure was present from the first request.
Sep 8, 2026 - 23:11	Report arrives through HackerOne.
Sep 9, 2026 - 08:00	Verification of the HackerOne report begins.
Sep 9, 2026 - 13:22	Security incident declared; Product Security reproduced the leak with two independent accounts inside 25 minutes.
Sep 9, 2026 - 14:00	Scope confirmed, GLM 5.3 set to private, requests return HTTP 400, status page updated.
Sep 9, 2026 - 14:35	Root cause identified as a prefill pool smaller than the context length; a canary prefill node with a larger pool was confirmed not susceptible.
Sep 9, 2026 - 14:52	All prefill nodes rolled to the corrected configuration.
Sep 9, 2026 - 19:04	Full deployment rebuilt with the corrected configuration plus a Cloudflare-authored engine patch that returns the original error for requests the prefill scheduler rejects; validated against the researcher's reproduction script.
Sep 9, 2026 - 19:40	Model restored to public.
Sep 10, 2026 - 23:30	Meeting with the SGLang maintainers (RadixArk CTO and core engineers) to agree on upstream fixes.
Sep 11-19, 2026	Additional hardening rolled out to all production SGLang deployments.
Sep 16-Oct 12, 2026	Upstream merges the admission fix, per-transfer checksums and abort-ACK preservation, and turns deferred KV release on by default.
Oct 2, 2026	SGLang v0.5.21 released.
Oct 19, 2026	SGLang v0.5.22 released.
Oct 26, 2026	Coordinated publication with the researcher and RadixArk.

Remediation gate: what had to be true before public access returned

Cloudflare disabled the model first, then worked the problem in three stages before restoring access:

• Configuration fix — make the pool bigger than the window. Aligning the GLM 5.3 configuration with the GLM 5.2 configuration freed enough GPU memory that the prefill KV pool now holds about 2.04 million tokens, roughly twice the 1,048,576-token context length, while the model's context length remains 1,048,576 tokens. Any prompt that fits the context window now fits the pool, so a valid request never reaches the rejection path.
• Engine fix — make the rejection path safe anyway. Pool size alone is a fragile dependency: changes to quantization, tensor parallelism, CUDA graph sizes or a future model version shift the pool in ways that are easy to miss, and other scheduler states could still reject a request. The patched engine retires a request carrying a pending rejection at the disaggregation admission queue, with its original error, before any bootstrap, allocation or KV transfer happens. That approach was proposed upstream as #38704 and also had the decode node verify that the number of transferred tokens matches the number it reserved. The equivalent change was already on SGLang main as #36988, merged September 6 for a related failure path but not yet released, and maintainers merged #38935, which rejects these requests earlier still.
• Verification — prove it before switching back on. Both changes were validated against the researcher's reproduction script and against Cloudflare's own canary-marker tests before the model was restored.

Hardening after the incident: the wider bug class

This leak belongs to a family of KV cache memory-safety problems Cloudflare has been working for several months, where a page is freed, reused or read while something else still has a claim on it. Two causes recur.

The first is page lifetime inside one node. Several tasks on a decode node can hold the same page simultaneously — token generation, offload to host memory, the KV transfer receiver, and the abort and error paths. If one frees the page while another is still using it, a new request can be handed that page mid-read or mid-write. The earlier fixes for decode offload (#24226, #30899) and for double release on grammar errors (#30937) were of this kind.

The second is a race across nodes. When a request is aborted while prefill is still writing KV to decode over RDMA, writes already sent cannot be recalled; if decode frees those pages immediately, another request can receive them and the late writes land in that request's cache. Cloudflare contributed proposed fixes (#31357, #38961) and a reproduction script (#32564). Both causes produce corrupted output or one user's data in another user's response with no error raised. The difference here was determinism: a rejected request went through the handoff every time rather than by timing accident, and decode never learned that prefill had rejected it.

Since the incident the provider has extended deferred release so prefill's source pages are held as well, and so every failure type is covered rather than aborts only (proposed upstream as #38961, superseded by #40645, #41402 and #41404); deployed bounded Mooncake transfer batches (#32758, from the SGLang community) so no single transfer call can outlast the release timeout; changed the load balancer so a client disconnect no longer aborts a request mid-transfer — it holds the prefill connection until prefill completes and the decode connection until the first token, then closes both; and added a mandatory check to every disaggregated deployment change covering pool size against context length, deferred release on both roles, transfer batch bounds inside the release timeout, and all required patches applied. All of these changes are live in disaggregated production deployments.

Upstream changes operators should adopt

Cloudflare met the SGLang maintainers at RadixArk on September 10, the day after the incident, and agreed that RadixArk would investigate the reported race conditions, build the integrity-checking idea upstream and write reproduction tests for this bug class. The following landed on SGLang main, and all but one are included in v0.5.22; operators running disaggregated SGLang should move to v0.5.22 or backport these patches.

Category	Pull request	Release	Function
Incident	#36988	v0.5.20	Retires aborted prefill requests before their result reaches decode, closing this incident's exposure.
Incident	#38935	v0.5.21	Rejects such requests at admission, ahead of any bootstrap, allocation or KV transfer; follows the approach in Cloudflare's draft patch, #38704.
Race	#39461, #41612	v0.5.21, v0.5.22	sgl-router cancels both halves of a request on client disconnect or prefill failure.
Race	#37454, #41023	v0.5.20, v0.5.21	Enables deferred KV release only on backends that support it, then makes it the default.
Race	#40645, #41402	v0.5.21	Repairs cases where the drain confirmation was lost so releases fire on confirmation instead of waiting out the timeout.
Race	#41404	v0.5.21	Holds pages on every transfer failure, not only decode-initiated aborts.
Race	#41618	v0.5.21	Fixes a regression in the abort notification path.
Race	#43719	v0.5.22	A failed Mooncake send no longer reports its writes as drained.
Race	#37077	v0.5.22	Tags abort messages per request so a reused ID cannot consume a stale confirmation; adds MORI support.
Race	#36612	v0.5.20	Shares the prefill-to-decode failure notification across Mooncake, NIXL and MORI.
Race	#32758	v0.5.20	Optional cap on Mooncake transfer batch size so one timed-out call cannot leave an unbounded number of writes in flight.
Lifetime	#30899	v0.5.21	Decode waits for an in-progress host-memory offload before retracting a request and freeing its pages. Authored by Cloudflare.
Lifetime	#43562	v0.5.22	Keeps tensor-parallel ranks in agreement when a host-memory cache restore fails.
Observability	#39500	v0.5.21	Adds --disaggregation-enable-kv-checksum so decode verifies a checksum of the transferred KV before generating; cleanup in #39729.
Observability	#38596, #40642	v0.5.20, v0.5.21	Fixes for SGLang's KV canary, which detects and logs potential corruption.
Observability	#41128	v0.5.22	Metrics for deferred release: count, duration, and whether each hold ended on drain confirmation or timeout.

Still in progress upstream: #39729, additional cleanup for --disaggregation-enable-kv-checksum (merged, awaiting release); zeroing KV pages on release, drafted as proposal #46435; extending KV checksum validation from the prefill-to-decode handoff to every decode step; and broader model and feature support for KV Canary and KV Checksum.

Common misconceptions this incident disproves

• "A 200 response proves the request was served correctly." Here the client got HTTP 200 instead of HTTP 400, prompt_tokens matched the long prompt and the reply was coherent. Success status said nothing about where the tokens came from.
• "Advertised context length equals available cache capacity." The pool is sized at startup from leftover GPU memory; the context length is a separate configuration value. Nothing forces the two to agree.
• "Rejection paths are safe because rejected requests are marked as finished." Both admission queues checked finished(), and a rejected request kept its status in to_finish, so it stayed live and moved through the handoff.
• "Single-node testing predicts disaggregated behaviour." On one node the placeholder ran, the batch finished and the client received an error; only the split prefill/decode topology turned the same code path into a leak.
• "A big enough pool means the bug class is closed." Pool size alone is fragile — quantization, tensor parallelism, CUDA graph sizes and model version changes move it, and other scheduler states can reject a request.
• "No exploitation evidence means no exposure." The requests looked like ordinary long-context use in timing and volume, yet 155 responses still crossed tenants.
• "Updating the engine is all an operator needs." Version currency matters, but so do deployment-time checks, validation against reproduction scripts, and observability such as KV checksums and canaries.

Pre-flight checklist for disaggregated inference deployments

Condensed from Cloudflare's remediation and recommendations for operators running SGLang in disaggregated prefill/decode mode:

• Read the selected KV pool size from the engine's startup log or metric on every prefill and decode node, and compare it with the context length after any change to weights, quantization, parallelism or memory fraction.
• If the pool is the smaller number, lower the context length so it fits; do not rely on a proxy to police the limit.
• Confirm every deployment gate actually asserts the pool-versus-context relationship rather than inferring it from response codes.
• Validate model output and coherence at long contexts, not only HTTP status and throughput.
• Run cross-tenant isolation tests with canary markers and two independent accounts, using prompts above the pool threshold.
• Update SGLang dependencies to v0.5.22 or later, or backport the incident and race-condition patches.
• Turn on deferred KV release on both roles, bound transfer batches inside the release timeout, and enable --disaggregation-enable-kv-checksum where supported.
• Ensure client disconnects do not abort requests mid-transfer, and that rejection is honoured at admission before bootstrap, allocation or KV transfer.
• Require all of the above as mandatory checks for every new disaggregated deployment, not as a one-off audit.

FAQ

Was customer data actually leaked, and how much?

Yes. Between August 27 and September 9, a small number of customers using the GLM 5.3 Workers AI model received responses containing data from other customers' requests to the same model shortly before theirs. In total 155 responses were affected, all to prompts longer than 681,280 tokens.

Which deployments were in scope?

Only GLM 5.3. Cloudflare confirmed that no other model hosted on Workers AI was affected, and that every other disaggregated deployment has KV pools larger than its context length on both prefill and decode, so no request could reach the rejection path.

Because the affected model was @cf/zai-org/glm-5.3, the disclosure lists the rest of the catalogue — including GLM 5.3 Flash and DeepSeek V4 — as outside the failure path.

Why did nobody notice for roughly two weeks in August and into September?

The failure was silent by construction. Over-length requests still produced HTTP 200, a coherent reply and correct token accounting; the only tell-tale sign was that answers arrived faster than an 8-page prefill should take, and that they contained text from other users' KV pages.

Are self-hosted SGLang fleets at risk too?

Any operator running SGLang in disaggregated prefill/decode mode should assume the class applies until measured otherwise. The patch for this issue was released in SGLang v0.5.20 with follow-ups in v0.5.21 and v0.5.22, and is available to all other operators dependent on SGLang; updating to v0.5.22 or later is the recommendation.

Is this a prompt-injection or model-behaviour problem?

No. It is a memory-ownership and lifecycle problem in the serving layer: pages are not zeroed between requests, ownership is shared between offload, generation and the remote prefill reference, and a rejected request was never aborted at the point of admission.

What should buyers of hosted inference ask their vendors?

Whether their serving stack is disaggregated; how KV pool size is validated against context length and where that check is enforced; whether KV checksum verification and canary tests exist; how rejection, abort and client-disconnect paths are handled; and what the vendor's incident timeline and notification obligations look like.

Action recommendations for security and platform owners

1. Treat GPU KV cache pools as a multi-tenant data store: they belong in your data-flow diagrams, threat models and breach-scoping runbooks.
2. Add a measurable invariant to every inference rollout — pool capacity must be at least the context length — and fail the deploy when it is not.
3. Extend long-context test suites beyond status codes and latency; assert on output content and coherence, and use canary markers to detect cross-session reads.
4. Pin an inference-engine version policy with a security patch SLA, and track upstream releases such as v0.5.20, v0.5.21 and v0.5.22 rather than feature releases only.
5. Instrument ownership failures: enable deferred-release metrics, KV canary logging and KV checksum verification, and alert on drain-versus-timeout patterns.
6. Make reproduction of externally reported bugs a gate for restoration, using independent accounts and the reporter's script.
7. Review your coordinated-disclosure playbook against this case: report intake, 40 minutes from incident declaration to takedown, restoration the same day, and publication 47 days later with the researcher and the upstream maintainer.
8. Re-run this checklist after every change to quantization, parallelism, model weights or memory fraction — those are the moments when pool size drifts silently.

The durable lesson for anyone running or buying hosted LLM inference is that isolation is not a property of the model, the API key or the network edge — it is a property of memory bookkeeping in the engine underneath. The disclosures, patches and metrics now available around GLM-5.3 give security teams a concrete template: measure the pool, enforce the invariant, prove rejection actually aborts, and validate output rather than status codes. Do that and a silent cross-tenant leak becomes a blocked deployment in review rather than 155 responses you learn about weeks later.