Why Efficient LLM Serving Matters

Serving frontier open models such as Moonshot's Kimi K-series and Z.ai's GLM is a challenge: they're large, long-context, and use mixture-of-experts architecture. On GPUs, memory fills up fast—often the KV cache, not weights, becomes the bottleneck. Cloudflare's Workers AI tackles this with three techniques: quantizing the KV cache, compressing model weights, and protecting the shared cache. This insight explores each method, its trade-offs, and the engineering mindset behind making AI inference cheaper and faster.

If you're building AI-native infrastructure, understanding these optimizations is crucial. For a broader look at infrastructure resilience, check our Q1 2026 Internet Disruption Report.

GPU server rack with multiple accelerators for high-performance AI inference Algorithm Concept Visual

Technique 1: Quantizing the KV Cache

The KV cache stores attention keys and values for every processed token, enabling long conversations. By default, it's 16-bit (BF16). Cloudflare stores it in 8-bit floating point (FP8), halving memory usage. For Kimi K2.6, context capacity jumps from ~686K to ~1.37M tokens.

Performance Trade-offs:

  • At any concurrency, BF16 is a few percent faster per token.
  • BF16 runs out of memory at 32 concurrent requests; FP8 continues to 64, reaching 2,192 tokens/sec—41% higher than BF16's peak, at ~30% lower cost per token.

Accuracy Impact: Across benchmarks (GSM8K, ARC, MMLU), FP8 and BF16 caches are indistinguishable—differences are within noise.

Code Example (Conceptual):

# Example: Configuring KV cache quantization in SGLang for production
import sglang as sgl

llm = sgl.Engine(
    model_path="/models/kimi-k2.6",
    kv_cache_dtype="fp8_e4m3",  # Use FP8 for cache to save memory
    mem_fraction_static=0.8,     # Allocate 80% GPU memory to cache/weights
    enable_mixed_dtype=True,     # Allow different dtypes for prefill/decode
)

# Prefill uses BF16 for compute-bound phase, decode uses FP8 for memory savings
llm.set_cache_dtype(phase="prefill", dtype="bf16")
llm.set_cache_dtype(phase="decode", dtype="fp8_e4m3")

Developer monitoring KV cache usage and model performance on a dashboard Dev Environment Setup

Technique 2: Compressing Model Weights

For GLM 5.2, weights are compressed from FP8 to INT4, shrinking the checkpoint from 705GB to 421GB (40% smaller). This reduces per-GPU memory from ~88GB to ~52GB, freeing space for ~1.18M tokens of KV cache.

Speed Gains:

  • Decode becomes faster because it's memory-bandwidth-bound; INT4 reduces data movement.
  • At 1 concurrent request, GLM decode jumps from 60 to 92 tokens/sec (+55%).
  • Prefill is compute-bound; INT4 requires expansion, making it slower (8,660 vs 10,160 tok/s). Cloudflare uses disaggregated pools to apply INT4 for decode and FP8 for prefill.

Accuracy: INT4 and FP8 are indistinguishable across all benchmarks, with differences under 0.8 points.

Network diagram showing distributed inference nodes and cache sharing Development Concept Image

Technique 3: Protecting a Shared KV Cache

With more requests sharing the same GPU, cache integrity becomes critical. Cloudflare implements a tagging system: each physical cache page gets a tag that changes on reallocation, and requests verify their expected pages before reading. If mismatch, the request is aborted to prevent data corruption.

Overhead:

  • Throughput impact: under 1% across concurrency levels.
  • p95 latency impact: under 1%.
  • The check runs as a separate batch, avoiding GPU race conditions.

Limitations and Caveats

  • Quantization trade-offs: While accuracy holds on tested benchmarks, edge cases may reveal subtle differences. Always validate on your own data.
  • Hardware dependence: FP8 and INT4 support varies by GPU architecture; Blackwell introduces NVFP4, which may further improve efficiency.
  • Complexity: These techniques require deep expertise in inference frameworks like SGLang and may not be necessary for smaller models.

Next Steps for Learning

  • Dive into SGLang's source code to understand paged attention and cache management.
  • Experiment with different quantization levels (FP8, INT4, NVFP4) and measure the impact on your specific workloads.
  • Explore how disaggregated prefill/decode architectures can optimize your serving stack.

Conclusion

Efficient LLM serving is about making smart trade-offs between memory, speed, and accuracy. By quantizing the KV cache and model weights, and ensuring cache safety, Cloudflare delivers high-performance inference at lower cost. This engineering mindset—measuring, validating, and iterating—is key to building scalable AI infrastructure.

For more on building privacy-aware AI systems, see our guide on AI-native data classification.

This content was drafted using AI tools based on reliable sources, and has been reviewed by our editorial team before publication. It is not intended to replace professional advice.