The open-source llm-d project shipped its 1.0 release on Tuesday, October 20, 2026, which its maintainers call the most important version since the effort began. llm-d is a Kubernetes-native layer that sits in front of model servers such as vLLM and SGLang and decides where each inference request should run. It was started in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, is now a Cloud Native Computing Foundation Sandbox project, and says its contributor base now exceeds 500 people spread across more than 25 organizations.
Earlier releases focused on performance features: routing requests to replicas that already hold the relevant prompt prefix in cache, offloading the key-value cache to cheaper memory, separating the compute-heavy prefill phase from the memory-heavy decode phase, and spreading large mixture-of-experts models across many accelerators. Version 1.0 targets what the project calls Day 2 operations, the work of keeping a shared inference service stable for many customers over time.
The main additions are operational. A new flow-control system groups traffic by tenant and priority, protecting premium workloads during spikes while batch jobs wait, and it can shed load gradually instead of failing. The router now runs with a hot standby, and a component called DisaggregatedSet treats prefill and decode replicas as a single unit so both can be upgraded in place without dropping requests. The configuration API has graduated to a stable v1. For agents, session-aware routing keeps a multi-turn conversation on the replica that holds its cache, and support for Responses API state means an agent does not have to resend its full history.
Google, which published its own post the same day, added an asynchronous processor that pulls latency-tolerant jobs from queues such as Pub/Sub or Redis, plus time-slicing on TPUs and GPUs that fills idle gaps during reinforcement learning. Google says that lifted accelerator duty cycles from roughly 40% to above 70%. It also reports that on an agentic coding workload on Ironwood TPUs, cache-aware routing with host-memory offload cut median time to first token by more than 85% and raised throughput 2.4 times compared with a plain Kubernetes Service, and that cache hits on prompt prefixes for its Gemini platform doubled, rising from 35% to 70%.
The project lists production users across vendors. According to the llm-d team, AWS reports up to 70% more tokens per second on SageMaker HyperPod, Tesla and Red Hat measured three times the output throughput for Llama 3.1 70B on AMD MI300X GPUs, and IBM Research runs as many as 3,000 coding agents at once on a cluster of 544 NVIDIA H100s. DigitalOcean, Oracle Cloud Infrastructure, Mistral AI and JetBrains are also named. The stack runs on NVIDIA, AMD and Intel GPUs as well as Google TPUs, and Red Hat ships it as part of Red Hat AI.
The release matters because inference, not training, is becoming the largest share of AI compute spending, and agent workloads reuse huge amounts of context. A shared, vendor-neutral scheduler could reduce lock-in to any single chip supplier or proprietary serving platform.
All performance figures come from llm-d's founders and adopters, measured on their own workloads, and none has been independently benchmarked. The 1.0 label signals API stability, but operators will still need to judge how mature individual hardware paths are, and support for NVIDIA's upcoming Vera Rubin platform is still in progress.