Cloudflare has published an efficiency methodology on its official blog to deliver large open models at high speed with Workers AI.
The targets are long-text Mixture-of-Experts (MoE) models with high capabilities and demanding memory requirements, such as Moonshot's Kimi series and Z.ai's GLM. It is a commentary with actual measurements to ensure that accuracy is not compromised and that more requests are handled at a lower cost.
In this article, we will organize the three published methods and the points that companies and developers want to keep in mind.
The official description can be found on the Cloudflare Blog (https://blog.cloudflare.com/smaller-faster-safer-models/).
What's hard?
Workers AI turns inference on GPUs located in Cloudflare data centers around the world to deliver models closer to the user.
On the other hand, large and long pulse models such as Kimi and GLM are prone to bottlenecks due to GPU memory constraints. Not only the weights, but the KV cache to stretch the conversation squeezes the memory.
Cloudflare has previously published designs that use GPUs efficiently by separating inference prefills (context loading) and decodes (token generation). On top of that, there are three layers of content.
- Quantization of KV cache (FP8)
- Model weight compression (INT4)
- Shared Cache Integrity Verification
Method 1: KV cache to FP8
When the model generates a sentence, it saves the attention key (K) and value (V) of the token that has already been processed in the KV cache. In the Long Pulse model, this cache tends to fill the GPU memory before the weights.
The size is usually halved by holding the place with BF16 and holding it with FP8 (e4m3). Kimi K2.6 is said to have roughly doubled the context it can hold from approximately 686,000 tokens to approximately 1.37 million tokens.
The point is "how many requests can be placed in memory at the same time" rather than the one-shot speed itself. While BF16 ran out of memory at around 32 concurrently, FP8 was able to grow to 64, with peak throughput about 41% higher and cost per token about 30% lower.
Even on the benchmark, the difference between BF16 and FP8 is almost in the error range in GSM8K, MMLU, etc. It is positioned to increase the capacity without changing the response quality.
Method 2: Compress weight to INT4
The other memory consumption is the model weight. GLM 5.2 compresses weights from FP8 to INT4 and reduces checkpoints from 705GB to 421GB by approximately 40%.
In 8-way tensor parallel, the memory per GPU drops from about 88GB to about 52GB, creating about 1.18 million tokens worth of KV cache space on the same hardware.
since decode "reads weights from memory every time", it is easy to become bandwidth-limited, and token generation is faster when the amount of data is reduced. An example of up to about +55% faster is also shown for low concurrency.
On the other hand, since prefill is computationally fast and INT4 is more expensive and slower to deploy, Cloudflare's policy is to use decode as INT4 and prefill as FP8. The accuracy difference is also described as within 0.8 points on each bench.
Method 3: Shared Cache Integrity Verification
Both of the above have the effect of "placing more requests on one GPU". This increases the risk that hundreds of requests will read and write to the same physical KV cache page.
accurate management of paged attention, continuous batching, and cache reuse is a prerequisite, and even rare mistakes cannot be ignored in large traffic.
Therefore, Cloudflare tags the physical cache page with a change every time it is reassigned, and records the page and tag that each request expects. It has a mechanism to match before decode, and to interrupt the request before returning incorrect data if there is a mismatch.
Overhead is generally less than 1% of throughput and tail latency. The verification is not embedded in the attention kernel but in separate batches, and there are almost zero-cost routes for unnecessary deployments.
What companies and developers need to think about
This is a story about "making huge models cheaper and faster", but it also plays a role in the design judgment of the AI introduction side.
- Look at not only the model name, but also the inference-based memory design and quantization policy
- Decide first what the bottleneck is in terms of long term, number of concurrent users, and cost
- separate applications on the premise that the optimal conditions are different in prefill and decode
- Include validation costs for correctness in the design for shared caching and batch processing
- Confirm the basis for maintaining accuracy (bench/internal evaluation) with the vendor or self-operated
Even when AI is placed on internal systems and web services, not only "which model" but also "how to distribute and monitor" divides the results and costs.
AI adoption and infrastructure design
The release of Cloudflare is a practical insight to provide a stable frontier-grade open model on edge GPUs. In the future, the policy will continue to expand FP8KV, verify NVFP4 weights for Blackwell, and keep integrity checks on.
Makoto Tejima supports the implementation of the latest AI models, including existing system integration, authority and monitoring, and cost design.
Please feel free to contact us if you have any questions about how to proceed, such as internal AI, agent implementation, or adding AI to existing web systems.
Recap
Cloudflare has published three methods to efficiently run large and long pulse models like Kimi and GLM with Workers AI: FP8 KV cache, INT4 weight compression, and shared cache integrity verification.
The common goal is to loosen memory constraints while maintaining accuracy, increasing simultaneous processing and cost efficiency. Even with the introduction of AI, model selection and the design of the inference base must be viewed as a set.
If you have any questions about system development using AI, deployment design that balances cost and stability, quotes, or how to proceed, please do not hesitate to contact us.
