A new reproducibility study suggests that prefix caching can change the path taken by an LLM agent even when the model, decoding settings, seed and request order are held fixed. The effect grew substantially as the tested model weights were quantized more aggressively, turning what is normally treated as a serving optimization into a hidden source of experimental state.
The paper, submitted to arXiv on September 4 and currently under review for IEEE Access, does not show that caching makes models less accurate on average. Its narrower finding is more operationally important for evaluation and debugging: under the tested conditions, cache state can make an otherwise identical run follow a different token trajectory, and that difference becomes easier to trigger at lower weight precision.
The serving layer changed the trajectory
The study ran an 80-episode, multi-turn tool-use workload across two serving engines and four weight formats. Requests were serial, batch size was one, greedy decoding was used, temperature was zero and every request used seed 42. The key-value cache itself stayed at 16-bit precision, so the variable under study was weight quantization rather than KV-cache quantization.
With cache reuse disabled, repeated runs were bit-identical across all tested configurations: 0 of 800 episodes diverged. When cached and recomputed execution were compared, 36.2% of agent episodes changed trajectory at 16-bit weight precision and 75.0% did so at four-bit precision. The paper also reports a controlled cache-state experiment in which the cached and recompute paths were each individually reproducible on 40 of 40 items, yet differed from each other on 14.
That distinction matters. The result is not evidence that the serving stack is randomly unstable. It indicates that the stack can be deterministic conditional on state that is not represented in the request itself.
A hidden state variable can defeat a “deterministic” replay
The paper found an additional server-level prompt-cache layer in one engine that materially affected repeated runs. With that layer active, inserting a cache-disabled pass between two cache-enabled passes increased divergence from 38.8% to 77.5%. With the layer disabled, the same ordering change left divergence at 1.2%.
The authors are unusually explicit about measurement problems they found during internal review. Their public artifact records an answer-extraction bug that originally distorted mathematics accuracy and an execution-order confound that affected the first engine comparison. The corrected analysis is regenerated from preserved raw logs rather than from new inference. That disclosure improves auditability, but it does not turn the study into an independent replication.
A separate vLLM issue provides useful corroborating context. In January, a user reported different outputs between a prefix-cache miss and subsequent cache hits on an AMD MI355X system. That issue concerns different hardware, a different model and a different software build, so it should not be treated as a reproduction of the new paper. It does show that cache-path differences have surfaced independently in production-oriented serving software.
The result is about reproducibility, not average quality
The paper's single-turn bridge found correctness flips in both directions while aggregate accuracy stayed roughly stable. This is important because a headline such as “prefix caching hurts accuracy” would overstate the evidence.
For agent evaluation, however, unchanged average accuracy is not enough. A trajectory can change tool choices, intermediate state, external calls and failure timing while leaving the final benchmark score nearly unchanged. Two benchmark runs can therefore appear statistically similar while exercising different execution paths.
The study also has clear limits. It covers open-weight models from 7B to 14B parameters, one consumer RTX 4090, single-tenant serving and English workloads. It does not establish the same divergence rates for frontier models, hosted APIs, production multi-tenancy or different accelerators. The agent benchmark was also difficult for the tested models, so the paper explicitly avoids using it to claim an effect on agent task success.
Evaluation provenance should include serving state
Aipolix's practical conclusion is that reproducibility metadata for LLM systems should extend below the model API. Recording model name, seed, temperature and prompt is not sufficient when a serving optimization carries state across requests.
For release gates, benchmark pipelines and incident replay, cache configuration should be treated like a test-environment parameter. At minimum, teams should record the serving engine and version, whether prefix or prompt caching is enabled, and any observable cache exposure. Where a comparison must be repeatable, the runner should reset or otherwise control cache state at run boundaries.
There is a second implication for quantized deployments. Lower precision is often introduced to reduce memory and serving cost, while evaluation still assumes that identical inputs define comparable trials. This study suggests those two decisions are not independent. If quantization makes small cache-path perturbations more likely to cross token decision boundaries, then a quantized release should be validated under the exact serving configuration that will be used in production, not only against a cache-neutral offline baseline.
That is a stronger engineering requirement than simply adding another benchmark score. It turns serving-state provenance into part of the evidence needed to reproduce an agent failure or to defend a model comparison.
The artifact makes the claim inspectable
The accompanying repository includes the measurement harness, raw per-request logs, analysis scripts, pinned engine versions, model checksums and launch flags. The author states that every reported number and figure can be regenerated from those logs. The repository also preserves the corrected and original scoring outputs from the internal-review fixes.
That level of artifact disclosure makes the paper more useful to practitioners even before peer review, because teams can inspect the exact configuration assumptions or repeat the measurement on another stack. The remaining question is external validity: whether the magnitude survives different models, accelerators, batching regimes and multi-tenant cache behavior.
For now, the defensible takeaway is narrower and actionable. Prefix caching should not automatically be treated as a transparent implementation detail in reproducibility-sensitive LLM systems, and quantization may make its behavioral footprint substantially larger.