A new benchmark accepted for CIKM 2026 asks a harder question than whether a model can retrieve the right passage: can it follow a professional manual correctly from beginning to end when one early mistake can corrupt everything that follows?
Researchers from Manulife introduced Tasks over Application Manuals (TAM), built around two rule-heavy tasks: ICD-10-CM clinical coding and U.S. federal sentencing-guideline calculations. The benchmark uses human-validated targets and manuals with thousands of pages and tens of thousands of rules and cross-references.
Across GPT-5 baselines using single-pass RAG, agentic RAG, ReAct-style tool use and an agent harness, the best exact-match result was 1% on ICD-10-CM coding and 15.5% on sentencing. The authors explicitly present these as initial prompting-based baselines, not an exhaustive test of every specialized system.
The result does not show that GPT-5 cannot reason in general. It shows that techniques which look capable on shorter retrieval and multi-hop tasks can still fail badly when correctness depends on maintaining a globally consistent state across dozens of dependent decisions.
Retrieval was only one step
TAM requires a system to identify the applicable rule, follow cross-references, preserve earlier commitments, apply exceptions in the right order, revisit earlier choices when new constraints appear and stop only when the complete procedure has been executed.
For ICD-10-CM, the benchmark uses a 1,304-page Alphabetic Index, a 1,942-page Tabular List and official coding guidelines. Its dataset contains 1,000 cases sampled from 38,332 eligible encounters. The legal side contains 200 human-reviewed cases using annual Title 18 archives and U.S. Sentencing Guidelines manuals from 2021 through 2025.
This exposes a weakness ordinary RAG evaluation can hide. A model may retrieve a relevant rule and make a locally plausible decision, yet still produce an invalid final answer because an earlier branch was wrong or a later mandatory constraint was missed.
Among 50 saved non-exact ReAct trajectories, 38 showed global inconsistency as the first visible divergence. Seven stopped before completing the procedure and five missed a required element.
An agent harness did not solve the state problem
The paper also tests a LangChain Deep Agents baseline. Work is split into staged workers and reusable skills; each worker receives the case materials, relevant manual text and prior notes needed for its step.
That resembles many production agent designs: decompose the task, constrain each worker's context and pass structured intermediate results forward.
But decomposition alone did not provide procedural integrity. If the system does not preserve the right global state, validate accumulated constraints and backtrack when an earlier choice becomes inconsistent, dividing the work among agents can simply distribute the same procedural error.
Long context, retrieval and agent orchestration address different problems. None automatically supplies end-to-end procedural correctness.
Stronger evidence, with important limits
TAM is grounded in real professional manuals, has human-validated labels, and publishes the benchmark data, code, prompts, worker configuration and navigation tools. The paper has been accepted for the 35th ACM International Conference on Information and Knowledge Management, CIKM 2026.
The limitations matter. TAM covers only two domains with exact outputs and does not represent workflows where judgment is intentionally subjective. The study compares representative prompting-based approaches rather than every specialized or fine-tuned system.
There is also an operational complication: 35 of 200 sentencing cases ended early for the agent harness because retrieved statutory text triggered provider-side content filtering. The benchmark therefore exposes dependencies on provider policy and tool execution as well as reasoning architecture.
The clinical gap is not explained away by label disagreement. The strongest partial ICD result reached only 37% recall and 50.1% primary-diagnosis accuracy.
Why this matters for enterprise agents
Aipolix's analysis is that TAM targets a reliability gap that matters directly to enterprise automation. Insurance, financial controls, compliance, procurement, healthcare administration, tax and regulated approvals contain long procedures in which distributed rules must be applied in the right order and state must survive across many decisions.
An agent can sound reasonable at every individual step and still be wrong at the process level.
Evaluation therefore needs more than retrieval accuracy or a persuasive final answer. Teams need trajectory checks: which rules were active, which constraints remained unresolved, whether a later decision invalidated an earlier one, whether mandatory steps were skipped and whether the system knew when to backtrack.
The architecture implication is a reliability stack built around explicit state, constraint tracking, verification checkpoints and reversible decisions—not only larger context windows or more retrieval calls.
TAM is not a permanent verdict on frontier models. It is a new benchmark with two domains, and specialized methods may improve rapidly. That is why the public release matters: the dataset is on Hugging Face and the implementation is on GitHub, allowing other teams to test structured state machines, search, domain tools and verification methods.
For now, the warning is practical: strong short-horizon reasoning scores are not sufficient evidence that an agent can safely execute a long professional procedure.
The hard part is not only finding the rule. It is remaining consistent with every rule that still matters after the twentieth decision.