FLEET is a new test-time decoding method that tries to fix a simple but expensive weakness in repeated LLM sampling: each new completion normally forgets what earlier completions already tried. The authors argue that this makes pass@k-style inference waste compute on semantically similar or already-failed branches. Their alternative adds memory to generation and turns repeated sampling into a structured search over high-uncertainty decision points.
In the paper's headline result, FLEET raises LiveCodeBench Pass@32 from 59.9% to 66.2% on the evaluated setup and reports reaching the repeated-sampling baseline accuracy with roughly one third of the samples, which the authors summarize as a 3× speedup. The result is interesting because it targets inference-time efficiency rather than a larger model or more training.
The problem is not diversity everywhere
Temperature sampling adds randomness at every token. That can help a model escape a bad high-probability branch, but it also injects noise into low-uncertainty parts of a solution where the model was already stable.
FLEET tries to separate those cases. It monitors entropy and varentropy in the model's internal states to identify likely branching points. A VectorDSU structure clusters similar hidden states so the system can remember which actions were already explored and how those trajectories scored.
The search then uses reward information from completed trajectories in a pUCT-like process. Instead of blindly increasing temperature, FLEET penalizes previously explored actions that look less useful and redirects later generations toward alternatives. In the evaluated configuration, generation remains greedy and deterministic once those penalties are applied.
That design changes the role of repeated inference. The system is no longer drawing independent lottery tickets from the same distribution; it is spending each new attempt using evidence from earlier attempts.
The coding result is meaningful, but narrow
The paper evaluates FLEET on GSM8K and a filtered LiveCodeBench set using Llama 3.2 3B with 32 trajectories per problem. Under ground-truth verification, the coding benchmark moves from 59.9% to 66.2% Pass@32, resolving 147 of 222 tasks instead of 133. GSM8K improves only slightly because the baseline is already close to saturation.
The authors also study an outcome reward model as a more realistic substitute for ground-truth verification. That experiment is important because production systems rarely know immediately whether a generated answer is objectively correct. The results show that search quality and final answer selection become partly dependent on the quality of the reward signal.
That is a key limitation, not a footnote. A search algorithm can explore better candidates and still choose the wrong final one if its evaluator is poorly calibrated.
The strongest fit is self-hosted inference
The official repository is installable as the fleet-search package and includes examples for Transformers, nnsight and Ray. But FLEET is not a transparent drop-in replacement for ordinary hosted APIs.
The implementation requires white-box access to model hidden states. That makes the method much more immediately relevant to teams serving open-weight models or operating their own inference stack than to developers who only receive text or standard token probabilities from a closed API.
For coding agents, the most attractive environment is one with a cheap, deterministic verifier: tests, compilation, static checks or another objective task score. In that setting, failed completions are not just discarded. Their trajectory information can steer the next attempt away from the same bad branch.
This suggests a different way to think about inference-time scaling. The useful resource is not only the number of samples. It is whether later samples learn anything from the failures that consumed the earlier budget.
The 3× claim should not be generalized yet
FLEET is an arXiv preprint, not a peer-reviewed production benchmark. The reported experiments are narrow: one 3B model, two benchmark families, and a LiveCodeBench subset of 222 relatively easy tasks. The repository also explicitly notes the requirement for hidden-state access.
So the 3× figure should be read as an experimental sample-efficiency result in the tested configuration, not as a general promise of 3× lower latency or infrastructure cost across models and workloads.
The next evidence that would materially strengthen the result is cross-model replication on larger open models, harder coding sets, wall-clock measurements on real serving stacks, and tests with imperfect verifiers. If those results hold, FLEET would point to a practical shift in test-time scaling: from paying for more independent guesses to building inference systems that remember which guesses already failed.