A new arXiv preprint finds that some large language models can correctly recognize that a question is fundamentally unknowable yet still take a confident action when the same question is wrapped in a professional-looking evidence panel. Across 12 frontier models, the author reports commitment rising from 6.5% with the bare question to 54.0% as increasingly authoritative-looking evidence is added.
The result matters for agents that act on dashboards, retrieved documents, market feeds or analyst-style reports. The paper argues that the failure is not simply bad factual knowledge or poor probability calibration. It is a separate action gate: the model can know that evidence cannot resolve the outcome and still behave as if the presentation itself justifies a decision.
Fabricated evidence can trigger nearly the same action as real data
The study constructs questions whose outcomes are provably unpredictable from the information available to the model, then varies the context around them. In one manipulation, the author compares genuine market data with a panel in which every displayed number is fabricated.
According to the preprint, the fully fabricated panel raises commitment from 24.5% without a panel to 36.8%. Genuine market data produces 37.6%, a difference the study reports as statistically indistinguishable. The implication is deliberately narrow: in the tested setup, the authority and structure of the presentation can be enough to push some models toward action even when the evidence has no information about the unknowable outcome.
The effect is not universal. The paper reports that three models are strongly susceptible, several largely refuse to commit, and others commit regardless of context. That heterogeneity is important because it prevents a simple claim that all LLM agents share the same failure mode.
The models often know the question cannot be answered
A matched set of answerable questions helps separate capability from action. The author reports that the same models answer those questions at near-perfect accuracy when the information actually determines the result.
The study also asks models to classify whether a question is knowable before taking action. They identify irreducible uncertainty roughly 90% of the time, then commit on only 0.4% of cases they have explicitly labeled unknowable. That suggests the system can express the right epistemic judgment when the workflow gives it a dedicated place to do so.
Self-reported belief probabilities do not explain the large behavioral shift either. The paper says stated probabilities move little while commitment changes sharply as the panel becomes more authoritative. This is why the author frames the problem as a disconnect between knowing and acting rather than a conventional calibration error.
A small training intervention fixes the gate, but only under some formats
The paper tests whether the action gate can be trained separately. Supervised fine-tuning of a 3B model on 540 synthetic examples, mainly simple dice, coin, jar and timer tasks, reduces commitment to zero on the original evaluation and transfers to three unseen domains across the reported runs.
That result is promising but not robust to every deployment constraint. The preprint reports that rigid output formats can remove the reasoning space the model used to refuse appropriately. In one ablation, the tuned model commits on all 48 unknowable cases when forced into a restrictive response format.
For agent builders, this is a useful warning. A safety behavior learned in free-form reasoning may disappear when a production API requires a short schema, fixed JSON field or binary decision. The interface around the model becomes part of the safety mechanism.
The release is unusually reproducible for a fresh preprint
The work is preliminary and no peer-review status was verified. It is also authored by a single independent researcher, so the results need external replication across additional tasks and agent frameworks.
At the same time, the release signal is stronger than a typical early paper. The official GitHub repository includes code, data, preregistration material and cached model outputs. The arXiv record states that the numerical results can be recomputed from the released artifacts, reducing dependence on inaccessible API calls for basic verification.
The paper is also candid about limits. Only the equity and crypto tasks have sealed or resolved outcomes that directly validate unpredictability. Sports and weather lack the same outcome structure, and the author notes that the weather panel contains real forecast signal, making that domain a weaker instrument for the core claim.
Why this matters for production agents
Many production agents consume information that looks authoritative by design: dashboards, retrieval results, database records, generated summaries and tool outputs. The study suggests that teams should test not only whether an agent can state uncertainty, but whether that uncertainty actually blocks action.
A practical control is to make the action decision explicit. The agent can first classify whether available evidence is sufficient for the requested action, record that judgment, and only then proceed to a tool call or recommendation. The paper's knowability-first result suggests this separation can materially reduce unjustified commitment, although it should not be treated as a universal guarantee.
Evals should also include fabricated-but-plausible evidence, not merely missing data. A system may behave safely when information is absent yet become overconfident when irrelevant information is formatted like a high-quality source. That is especially relevant for retrieval-augmented agents, financial assistants and operational systems that combine model reasoning with structured feeds.
The broader contribution of the preprint is a sharper definition of agent calibration. It is not enough for a model to assign reasonable probabilities or verbally acknowledge uncertainty. A production agent also needs a reliable policy for when uncertainty should prevent action, and that policy has to survive the exact tool and output constraints used in deployment.