EN PT-BR
← Voltar ao feed
high technology cloud services artificial intelligence

Cloudflare Workers AI GLM-5.3 cross-tenant KV cache leak via SGLang rejection-path bug exposes 155 responses

1 fonte(s) · 1 atualização(ões) · publicado 26/10/2026 19:01 · coletado há 6 dias

Cloudflare disclosed a tenant isolation failure in the GLM-5.3 model (@cf/zai-org/glm-5.3) on Workers AI, reported by researcher Gadi Evron (Knostic) through HackerOne on September 8, 2026. Between August 27 and September 9, 155 responses to prompts longer than 681,280 tokens contained data from other customers' recent requests. Cloudflare found no evidence of deliberate exploitation, and no other Workers AI model was affected. The model was taken offline within 40 minutes of the incident being declared and restored on September 9 after a configuration fix and an engine patch. The disclosure was coordinated with SGLang maintainers at RadixArk.

Detalhes técnicos

  • Two overlapping issues caused the leak. First, a misconfiguration: GLM-5.3 was deployed with a 1,048,576-token context window, but the prefill GPU KV cache pool held only 681,280 tokens. Cloudflare knew of the mismatch but assumed the engine would return an error. Pre-launch tests checked only HTTP status codes and performance, not output coherence.
  • Second, an SGLang bug in disaggregated prefill/decode mode. Over-length prompts were marked rejected and replaced with a one-token placeholder, but the admission queues checked finished(), which stayed False for rejected requests. Prefill therefore transferred one KV page to decode, and decode reserved the full set of pages (e.g. 8) and signaled completion.
  • Decode then attended over stale, unzeroed pages still holding other users' prompts and partial generations. The client received HTTP 200 instead of 400 with a coherent reply. KV pages are not zeroed between requests, so the engine must track page ownership.
  • The bug was reported upstream as SGLang issue #30233 on July 6, 2026. It is one of a class of KV cache memory-safety bugs involving page lifetime on a single node and abort races across nodes over RDMA (Mooncake).
  • Cloudflare's fix made the prefill pool hold about 2.04 million tokens, matching its GLM 5.2 configuration, and patched the engine to retire rejected requests at the disaggregation admission queue before any bootstrap, allocation or KV transfer. Upstream fixes include #36988, #38935, and #38704 (Cloudflare's proposal).
  • Further hardening: deferred KV release on both prefill and decode roles, bounded Mooncake transfer batches, a load balancer that no longer aborts mid-transfer on client disconnect, and --disaggregation-enable-kv-checksum and KV canary checks. Cloudflare is also prototyping page-level ownership tags and formal verification of the KV transfer protocol.

Remediação

  • Update SGLang to v0.5.22 or later, or backport the relevant patches. The admission fix shipped in v0.5.20, with follow-ups in v0.5.21 and v0.5.22.
  • Compare the KV pool size with the context length on every prefill and decode node after any change to weights, quantization, parallelism or memory fraction. Lower the context length if the pool is smaller.
  • Run deferred KV release on both roles and bound Mooncake transfer batches.
  • Enable KV checksum and canary checks where the model supports them.
  • Configure load balancers to avoid unnecessary aborts before the first token.

Linha do tempo

  1. blog.cloudflare.com ↗ · pub. 26/10/2026 19:01 · coletado 26/10 20:00 primeiro registro

    Cloudflare Workers AI GLM-5.3 cross-tenant KV cache leak via SGLang rejection-path bug exposes 155 responses

    ler a análise desta fonte

    Cloudflare disclosed a tenant isolation failure in the GLM-5.3 model (@cf/zai-org/glm-5.3) on Workers AI, reported by researcher Gadi Evron (Knostic) through HackerOne on September 8, 2026. Between August 27 and September 9, 155 responses to prompts longer than 681,280 tokens contained data from other customers' recent requests. Cloudflare found no evidence of deliberate exploitation, and no other Workers AI model was affected. The model was taken offline within 40 minutes of the incident being declared and restored on September 9 after a configuration fix and an engine patch. The disclosure was coordinated with SGLang maintainers at RadixArk.

    Technical Details

    • Two overlapping issues caused the leak. First, a misconfiguration: GLM-5.3 was deployed with a 1,048,576-token context window, but the prefill GPU KV cache pool held only 681,280 tokens. Cloudflare knew of the mismatch but assumed the engine would return an error. Pre-launch tests checked only HTTP status codes and performance, not output coherence.
    • Second, an SGLang bug in disaggregated prefill/decode mode. Over-length prompts were marked rejected and replaced with a one-token placeholder, but the admission queues checked finished(), which stayed False for rejected requests. Prefill therefore transferred one KV page to decode, and decode reserved the full set of pages (e.g. 8) and signaled completion.
    • Decode then attended over stale, unzeroed pages still holding other users' prompts and partial generations. The client received HTTP 200 instead of 400 with a coherent reply. KV pages are not zeroed between requests, so the engine must track page ownership.
    • The bug was reported upstream as SGLang issue #30233 on July 6, 2026. It is one of a class of KV cache memory-safety bugs involving page lifetime on a single node and abort races across nodes over RDMA (Mooncake).
    • Cloudflare's fix made the prefill pool hold about 2.04 million tokens, matching its GLM 5.2 configuration, and patched the engine to retire rejected requests at the disaggregation admission queue before any bootstrap, allocation or KV transfer. Upstream fixes include #36988, #38935, and #38704 (Cloudflare's proposal).
    • Further hardening: deferred KV release on both prefill and decode roles, bounded Mooncake transfer batches, a load balancer that no longer aborts mid-transfer on client disconnect, and --disaggregation-enable-kv-checksum and KV canary checks. Cloudflare is also prototyping page-level ownership tags and formal verification of the KV transfer protocol.

    Remediation

    • Update SGLang to v0.5.22 or later, or backport the relevant patches. The admission fix shipped in v0.5.20, with follow-ups in v0.5.21 and v0.5.22.
    • Compare the KV pool size with the context length on every prefill and decode node after any change to weights, quantization, parallelism or memory fraction. Lower the context length if the pool is smaller.
    • Run deferred KV release on both roles and bound Mooncake transfer batches.
    • Enable KV checksum and canary checks where the model supports them.
    • Configure load balancers to avoid unnecessary aborts before the first token.