A research team around MIT's Han Lab has open-sourced Evolving Infra, an LLM serving system built specifically for agent traffic. The mit-han-lab/EvolvingInfra repository appeared on GitHub on November 10, 2026 under the MIT licence, a day after the paper "Evolving Infra: Agent-Adaptive LLM Serving at Scale" (arXiv 2611.12304) was submitted. The authors are Hao Kang, Weili Xu, Ziyang Li, Zheng Du, Hanchen Li, Tian Xia, Song Bian, Tushar Krishna, Huizi Mao, Ion Stoica and Song Han, and the project's README lists Georgia Tech, the University of Illinois Urbana-Champaign, UC Berkeley, NVIDIA, OpenAI and MIT as affiliations.

The starting point is that coding, math and computer-use agents now generate a large share of inference demand, and they behave differently from chat. They fire long chains of requests, spawn sub-agents and periodically compact their history. Today's prefill-decode disaggregated servers split GPUs into fixed prefill and decode pools and manage the KV cache with coarse signals, which the authors say leads to load imbalance and to evicting cached context that an agent was about to reuse.

Evolving Infra changes three things. First, it separates compute allocation from KV cache storage: every worker can run either the prefill or the decode leg of a request, and cached context lives in a host-memory pool that all nodes can read, so a node can change role in seconds without a restart. Second, a planner uses a cost model that accounts for sub-agents and context management to adjust the prefill-to-decode split while the service runs. Third, an agent-aware distributed KV manager tracks which agent branches still reference which context. It evicts first what no branch needs, such as histories replaced by compaction or the context of finished sub-agents, and keeps what agents will reuse.

The headline numbers come from the authors' own experiments. Serving GLM-5.2 and Kimi K2.6 for Claude Code, Putnam math and OSWorld computer-use workloads on GB300 GPUs, they report 1.10 to 2.73 times the throughput of Dynamo, NVIDIA's production serving framework, 81 to 95 percent less lost KV reuse, mean end-to-end latency lower by up to 4.1 times and P95 time to first token lower by up to 14 times. A phase switch takes 6.2 seconds on average in their tests, against 754 seconds for relaunching a Dynamo node. When the workload shifted from math to computer-use agents, the system adapted its split online and reached 2.1 times Dynamo's throughput.

The code is substantial rather than a paper artefact. The engine and Rust router are derived from SGLang and keep their Apache 2.0 licences, the KV directory builds on Dynamo's indexer, and a runtime image targets H200, B300 and GB300 machines. Unusually, the repository is designed to be operated by coding agents: deployment, harness connection, benchmark reproduction and diagnosis are written as skills that tools such as Claude Code or Codex follow step by step. Recorded agent traces are published on Hugging Face so others can replay the benchmarks.

For teams running agent fleets on their own GPUs, the work targets a real cost driver. Agent sessions reuse large contexts across many turns, so keeping the right KV cache resident and rebalancing prefill and decode capacity as the mix changes translates directly into latency and hardware efficiency.

The results still need outside confirmation. All figures are the authors' measurements on their chosen workloads and clusters, the paper has not yet been peer reviewed, and no independent benchmark or press analysis was found at the time of writing. Reproducing the paper's Claude Code result needs nine GPU nodes for roughly two hours, which puts verification out of reach for most readers. The repository had 54 GitHub stars on November 16.

References