7 November 2026 · 3 min read
Buoyant Releases Kinference: Open Source Control Plane for vLLM on Kubernetes
Buoyant has released Kinference, an Apache 2.0 control plane that adds policy-driven routing, KV-cache-aware load balancing, and per-tenant metrics to vLLM inference on Kubernetes.
Buoyant, the company behind Linkerd, announced the general availability of Kinference on 5 November 2026 [2]. The initial release is v0.1.0-alpha.7, licensed under Apache 2.0 and written in Rust [1]. Kinference is an inference control plane that sits in front of a fleet of vLLM workers on Kubernetes, applying routing policies, token rate limits, KV-cache-aware load balancing, and per-tenant metrics in a single layer [1][2].
What Kinference does
Kinference turns Kubernetes plus vLLM into a multi-tenant inference platform. It acts as the control plane while vLLM remains the data plane [1]. The system watches vLLM KV-cache events and engine metrics, using NVIDIA’s dynamo-kv-router crate to process cache events [1]. Configuration is managed through five Kubernetes Custom Resource Definitions (CRDs): GatewayBackend, InferencePolicy, InferencePool, InferenceProvider, and InferenceWorkerSet [1].
For platform teams, Kinference provides guardrails and controls around performance, cost, and compliance. It can track GPU usage per tenant, group, or team and attribute costs accordingly [2]. It also supports bursting to external cloud providers when local GPU pools are saturated or when policies dictate [1][2].
Performance benchmarks
Buoyant published self-reported benchmarks comparing Kinference against standard Kubernetes Service routing (kube-proxy). Each test used three vLLM 0.31.0 pods, each on a single L40S GPU, running the Qwen3.8-27B-FP8 model. Load was generated with GuideLLM 0.8.1 [1].
On a shared-prefix workload, median time-to-first-token (TTFT) dropped from 6.88 seconds to 0.31 seconds, a reduction of about 95%. On multiturn traffic, median TTFT fell from 1.70 seconds to 0.59 seconds (about 65%). On a control workload with no shared prefix, there was no significant difference (6.92 s vs 6.83 s) [1].
Peak goodput—requests per minute meeting a TTFT service-level objective—rose from 14.0 to 103.0 on shared-prefix traffic (about 7.4×) and from 23.8 to 82.7 on multiturn traffic (about 3.5×). The Kubernetes Service baseline never met the SLO at any tested concurrency [1].
These numbers compare Kinference against random routing, not against other cache-aware routers such as llm-d or NVIDIA Dynamo. Buoyant notes that the SLOs used (8 s and 4 s) suit offline agent workloads rather than interactive chat [1].
Policy-driven routing
Kinference’s InferencePolicy CRD lets teams write declarative rules that decide how each request is routed or rejected. Rules can match on tenant, requested model, GPU pool state, and semantic tags. Five example patterns are documented: routing simple prompts to a cheaper model, bursting to cloud when local queues are long, keeping PII-tagged requests on-premises, admitting batch requests only when GPU cache usage is below 70%, and aliasing a stable model name to several backing models [1].
Most policies end with minimise: PredictedTTFT. Kinference predicts the time-to-first-token for each candidate endpoint by considering KV-cache overlap and queue length, then routes to the fastest option that satisfies the policy [1]. Policies can be stored in git and validated offline with the kinfctl CLI [1].
Deployment and current limitations
Installation requires Gateway API v1.5.1 CRDs, the Buoyant gateway Helm chart, and the Kinference Helm chart [1]. The project ships with Prometheus metrics and Grafana dashboards, including per-tenant measurements of TTFT, inter-token latency, end-to-end latency, and cached versus uncached prompt tokens [1].
As an alpha release, several features are not yet available: rate limiting, classifiers, and Anthropic API support are listed as not shipped [1].
What it means for developers
For platform engineers, Kinference adds a governance layer that has been missing from self-hosted inference on Kubernetes. It provides the declarative controls needed to offer an internal inference platform that balances developer flexibility with cost and compliance boundaries [2]. For developers consuming inference, the immediate benefit is lower latency on workloads with shared prefixes or multi-turn conversations, without changing how they call the model.
Kinference does not replace vLLM or other inference engines; it complements them by handling routing and policy enforcement. Its focus on policy, tenant accounting, and provider fallback sets it apart from other open source routers that prioritise raw serving performance and disaggregation [1].
Sources
- Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM
- Buoyant Announces Kinference: An Open Source Inference Control Plane for Kubernetes - AIwire
- Buoyant Unveils Open Source Control Plane for AI Workloads Running on Kubernetes
- kinference
- kubernetes
- vllm
- inference
- open source
- buoyant
Researched and drafted with AI from the sources above.
Learn to build with AI
Hands-on tutorials on Cursor, Claude Code, AI agents and more.