4 November 2026 · 3 min read
llm-d 1.0 Released to Cut AI Inference Costs with Multi-Tenant Controls
The CNCF-governed project llm-d reaches 1.0, adding multi-tenant routing, high availability, and session-aware caching to help developers run large language model inference more efficiently on shared infrastructure.
The llm-d project released version 1.0 of its open source AI inference software on 4 November 2026 [1]. Founded by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, the project is now governed by the Cloud Native Computing Foundation (CNCF) and has drawn contributions from more than 500 developers across over 25 organisations.
llm-d focuses on the inference stage of the AI stack – the point where trained models generate responses for users and applications. As companies deploy chatbots, coding tools and agentic systems that make repeated calls on memory and compute, inference has become a growing cost centre. Version 1.0 is the first full production release, designed to help organisations run large language models more efficiently across shared infrastructure.
What’s new in 1.0
The release introduces several features aimed at production deployments [1]:
- Multi-tenant controls to manage competing workloads on the same hardware.
- High-availability routing and rolling upgrades that allow maintenance without service interruption.
- A stable configuration interface for standardising deployments across different clouds and accelerators.
- Session-aware routing, which keeps multi-turn conversations close to cached context so systems can avoid recomputing the same information.
llm-d supports a range of model servers, including vLLM and SGLang.
Cost benchmarks
Project backers shared two examples of cost savings. IBM Research used open models with llm-d in work on quantum error-correction hardware and reported inference costs 11.5 times lower than an all-proprietary setup. That deployment recorded a prefix-cache hit rate of about 98% and a blended serving cost of roughly $0.11 per million tokens [1].
In another test, researchers deployed the open-weight GLM 5.2 model (approximately 753 billion parameters) on widely used NVIDIA GPUs. On benchmark agentic workloads with hundreds of parallel sessions, the system produced more than 6.6 million output tokens per minute at peak and served up to 3,000 concurrent coding agents without pre-emptions. According to figures released by the project, that setup would have delivered a per-token cost five to ten times lower than equivalent commercial API pricing at current cloud rental rates. The biggest savings appeared on input-heavy workloads, common in software agents that repeatedly submit long prompts and context windows [1].
Industry backing
Beyond the founding companies, quoted supporters include AMD, DigitalOcean, JetBrains, Mistral and Roblox, each describing live or planned use across their own AI services and infrastructure [1].
Chris Aniszczyk, CTO of CNCF, said the release helps organisations meet the shift to AI inference as a core cloud native workload, bringing together open source inference capabilities with the operational requirements needed to scale agentic workloads. Brian Stevens, SVP and AI CTO at Red Hat, noted that as organisations move into full-scale production, the bottleneck shifts from generating answers to managing economics, reliability and infrastructure demands, and that llm-d 1.0 delivers a standardised, hardened core for distributed inference. Priya Nagpurkar, VP of AI Native Systems at IBM Research, highlighted the importance of open technologies like llm-d for improving performance, lowering costs and accelerating innovation as agentic workloads become more prevalent [1].
What it means for developers
For developers and builders, llm-d 1.0 offers a production-grade, open source layer for serving large language models. The multi-tenant controls and session-aware routing directly address the cost and latency challenges of running agentic systems that reuse long context windows. The stable configuration interface and support for multiple model servers mean teams can standardise deployments across on-premises and cloud GPUs without being locked into a single serving engine. The reported cost reductions – up to 11.5 times versus proprietary setups – suggest that adopting llm-d could significantly lower the per-token expense of inference-heavy applications. As an open source project under the CNCF, it also provides a community-driven foundation that can be integrated into existing cloud native toolchains.
Sources
- llm-d
- inference
- open source
- cncf
- ai costs
Researched and drafted with AI from the sources above.
Learn to build with AI
Hands-on tutorials on Cursor, Claude Code, AI agents and more.