CordisBench puts a narrower but highly practical question in front of teams building dynamic AI-agent infrastructure: when an agent changes the components around itself, can the model reliably predict what those changes will leave behind? The new benchmark finds that models can often identify which components are affected while becoming much less reliable at predicting final state, reasoning across cleanup orders, or choosing minimal reconfiguration steps as interactions accumulate.
The important engineering result is not simply that models make mistakes. CordisBench also builds a finite reference semantics that matches the Cordis runtime on every observation and action outcome used to score 528 executable questions. For this restricted class of lifecycle changes, the mechanical consequence is computable exactly. That turns part of agent-harness governance from a prompting problem into a verification problem.
The benchmark isolates lifecycle reasoning inside mutable harnesses
CordisBench was submitted to arXiv on September 1 and releases both code and a frozen dataset. It contains 1,200 structured questions generated from 240 systems. The tasks cover localization of affected components, final-state prediction for a specified teardown order, conditions that hold across all or some permitted orders, and reconfiguration actions that are executed against the runtime.
The benchmark uses two settings. A compact formal setting makes the dependency and state-transition rules explicit. A Cordis-native setting compiles related lifecycle patterns into executable plugins against Cordis 4.0.0-rc.7. The authors vary the number of relevant interactions from 2 to 32 while keeping the question form and scoring rule fixed within each task.
The evaluated models are Gemini 3.7 Flash, GPT-5.6 Luna and DeepSeek V4 Flash at low reasoning effort. This is a deliberately limited model set, but the benchmark design is useful because answers are deterministically scored and reconfiguration plans are actually executed rather than judged from prose.
Knowing what changes is easier than knowing what remains
The clearest reported pattern is a separation between localization and consequence prediction. GPT-5.6 Luna, for example, remains near ceiling when identifying affected components while its formal reachable-condition score falls from 91.7% to 14.1% as interaction count rises. Its executed reconfiguration success in the Cordis-native setting falls from 62.5% to 25.0%.
Some of the degradation has identifiable confounds. Twenty-nine Gemini responses hit the original 8,192-token output limit, and raising that limit materially improves several large-instance scores. The authors also run a fixed-schedule diagnostic to avoid confusing a larger dependency structure with simply having more teardown orders. The remaining decline is therefore more informative than a single headline benchmark number, but it still belongs to this controlled task family.
Additional reasoning can recover performance. On a balanced 16-interaction subset, GPT-5.6 Luna's Cordis-native prediction rises from 31.2% with no reasoning to 85.4% at medium effort, while executed reconfiguration rises from 0% to 50%. The trade-off is about 2,967 reasoning tokens per question on average at that setting.
Some lifecycle decisions should not consume model reasoning at all
CordisBench's strongest systems result is that the finite reference semantics agrees with Cordis execution on all 528 executable questions for every observation and action outcome used in scoring. In other words, within the benchmark's restricted lifecycle model, extra language-model reasoning is being spent on consequences that software can calculate exactly.
That distinction matters for production agent systems. A model is useful for selecting goals, interpreting ambiguous intent and choosing among policies where the environment cannot be fully formalized. Dependency withdrawal, cleanup ordering and restoration of explicitly represented state are different. When those mechanics are known, asking the model to predict them creates an avoidable reliability and inference-cost surface.
The practical design principle is to move deterministic lifecycle consequences below the model boundary. The agent can propose a harness mutation, but a runtime verifier should compute the affected dependency cone, allowed cleanup schedules, resulting state constraints and whether the requested reconfiguration satisfies them before execution.
Harness evolution needs two different release gates
This paper arrives hours after HarnessDev, another September 1 preprint that Aipolix has covered, showed a broader problem: harness improvements selected on visible feedback do not always transfer to hidden tasks or different executor models. CordisBench identifies a complementary layer.
The two results should not be merged into one generic warning about autonomous agents. They point to two different assurance mechanisms. Semantic changes to an agent harness still need empirical promotion tests because their effect on downstream task performance cannot generally be calculated. Mechanical lifecycle changes should additionally pass deterministic verification whenever dependencies and cleanup effects are explicit enough to model.
That creates a more precise release architecture. First, validate whether a new harness version actually improves the intended workload under hidden tasks and production executors. Second, independently verify that the version cannot leave the runtime in an invalid or unintended state when components are added, removed or torn down in different legal orders. Model-based evaluation and formal lifecycle verification solve different failure classes.
The result is useful, but deliberately narrow
CordisBench is a preprint, not independently replicated evidence of production failure rates. Its largest 24- and 32-interaction cases are stress tests rather than estimates of typical deployed harnesses. The Cordis-native tasks use a controlled set of teardown orders and focus on dependency-driven removal plus restore-on-cleanup effects. They do not model irreversible external actions, failures, hot module replacement or the full feedback loops available to a production agent.
The three-model evaluation also leaves open how stronger models, tool access, execution feedback and retries would change the picture. Gemini is near ceiling on several Cordis-native tasks, and some large-instance results are sensitive to output limits.
Those limitations do not erase the main engineering lesson. They sharpen it. CordisBench does not show that language models cannot manage dynamic harnesses. It shows that teams should first ask which parts of harness management require probabilistic reasoning at all. When a state transition is mechanically derivable, the safer and cheaper architecture is to compute or verify it, then reserve model reasoning for the parts that are genuinely uncertain.