Most AI governance programs spend most of their energy on prevention: better prompts, tighter policies, narrower permissions, stronger reviews.

That is necessary. It is not sufficient.

Once an agent can change state, call tools, spend money, update configuration, modify code or trigger other systems, governance also needs a plan for the moment prevention fails. The important question becomes operational: can the runtime stop the action, determine what it touched, and recover the affected state?

This is Part 6 of the ten-part series AI Governance in an AI-Native Software Development Company. Part 5, AI Agents Need Contracts, Not Better Prompts, defined the contract that says what an agent is allowed to do. This article assumes something less comfortable: a system may still act outside that contract, or may behave badly while technically staying inside it.

That is where containment begins.

Prevention reduces probability. Containment limits consequence.

Containment is a runtime capability

Containment is often treated as part of incident response. That is too late.

If the first serious discussion about how to stop an agent happens during an incident, the organization is already improvising. A governed runtime needs containment before the first failure, because autonomous systems can act faster and fan out more widely than human operators.

One mistaken interpretation can become dozens of tool calls. One configuration change can propagate through dependent services. One permission can expose a much larger action surface than its name suggests. One automated workflow can continue retrying long after the original decision should have been reconsidered.

A useful containment design begins with three questions:

  1. Can we stop the action without depending on the agent's cooperation?
  2. Can we identify, with evidence, what was affected?
  3. Can we restore the affected state to a known-good baseline?

Those three capabilities can be summarized as STOP, SCOPE, RECOVER.

A system missing any of them is not fully governed. It may be observable, but observation after impact is not containment.

STOP: the system must be able to take authority away

A real stop mechanism cannot depend on the same agent deciding to obey it.

The control path should be independent from the execution path. It should also be scoped. Stopping one agent, one tool, one tenant or one capability is very different from shutting down an entire platform.

Useful stop primitives include:

  • pause when the situation is ambiguous and work may resume;
  • scope down when the system should continue with fewer tools or privileges;
  • quarantine when outputs may continue to be produced but must not reach production;
  • kill when execution and authority must end immediately.

For distributed agent systems, credential revocation is often one of the strongest stop mechanisms. Short-lived credentials, renewable leases and narrowly scoped tokens create chokepoints outside the model's reasoning loop.

This changes the design question. Instead of asking whether the agent will remember to stop, ask whether the infrastructure can remove its ability to act.

SCOPE: blast radius must be known before the incident

Containment depends on understanding impact.

For a human operator, "this tool can edit configuration" may be enough context. For an autonomous system, that description is incomplete. The runtime needs to know how far the action can propagate.

Blast radius can be represented through operational dimensions such as:

  • number of records, services, tenants or users affected;
  • downstream systems that may inherit the change;
  • reversibility of the action;
  • persistence of the effect;
  • visibility of the effect to monitoring and operators.

The exact scoring model matters less than making the boundary explicit.

A low-radius action may proceed automatically. A medium-radius action may require richer evidence or staged execution. A high-radius action may require signed human approval. Some catastrophic actions may simply remain outside the agent's available capabilities.

This is where governance becomes state-aware. The same requested action can be acceptable in one context and prohibited in another because the possible consequence is different.

RECOVER: agent-authored change must be reversible

Engineering teams already understand rollback for code and deployments. Agent-authored changes complicate the picture because the artifact that changed may not be the only thing that matters.

A code commit may be reversible while the prompt, skill, routing rule or policy bundle that produced it has already changed. An agent may perform many individually reversible mutations whose combined effect is difficult to reconstruct.

The containment rule should be simple:

  • if an agent can author a change, the platform should know how to undo it;
  • if an agent can promote a change, the platform should know how to demote it;
  • if an agent can fan out an action, recovery must account for that fan-out.

Rollback also needs rehearsal. A rollback path that exists only in documentation is an assumption, not a control.

The recovery action should produce the same quality of evidence as the forward action: who triggered it, what it reversed, which version was restored, and which systems were confirmed healthy afterward.

Behavior canaries matter more than host health

Traditional canaries are designed to tell you whether software is alive.

Agent systems need another kind of signal: whether behavior is still within expectations.

A model, prompt, skill, retrieval source or tool schema can change behavior without causing an HTTP error or a latency spike. The host can be healthy while the agent's decisions are drifting.

Behavior canaries should therefore watch things such as:

  • refusal rate;
  • escalation rate;
  • tool-call distribution;
  • unexpected access patterns;
  • output-schema violations;
  • changes in typical blast-radius classes;
  • shifts in evidence quality.

The purpose is not to create a perfect behavioral baseline. It is to make important behavioral change visible before it becomes a production incident.

Containment debt is real operational debt

Teams accumulate containment debt whenever they bypass controls temporarily and fail to restore them.

Examples are familiar: a rollback process that has not been exercised for months, an emergency permission that became permanent, a kill switch nobody has tested, credentials that never rotate, or an exception that exists only in someone's memory.

Each shortcut reduces the organization's reaction margin.

Containment debt is dangerous because it often stays invisible during normal operation. Its cost appears in incident duration, uncertainty and recovery time.

The practical response is to treat containment controls as operational assets with owners, tests and review dates.

A practical implementation order

Containment does not require a new platform before any progress is possible.

Start with the highest-consequence autonomous capability and make three things explicit: how to stop it, how to calculate its possible impact, and how to restore state after it acts.

Then:

  1. implement an out-of-band stop path;
  2. classify blast radius on tools and skills;
  3. connect high-radius actions to stronger approval requirements;
  4. build and rehearse rollback for one class of agent-authored change;
  5. add behavioral canaries to model, prompt, skill and tool changes;
  6. exercise stop, scope and recovery outside incidents;
  7. record every exception and give it an expiry path.

This sequence is intentionally operational. Governance improves when the controls can be exercised, not when they merely exist in architecture diagrams.

Closing

The goal of AI governance is not to prove that agents will never make the wrong move.

That standard is impossible.

The more useful goal is to make failure bounded.

A governed system can revoke authority, calculate impact and recover state. It does not rely on the model being perfect, the prompt being interpreted exactly as intended, or every dependency behaving normally.

That is why containment expands the amount of autonomy an organization can safely allow. When the blast is small, the stop is fast and recovery is routine, teams can take measured risks without pretending that prevention will always work.

The important question is no longer only:

"Can we trust this agent?"

It is also:

"Can we contain it when that trust turns out to be wrong?"

That second question can be answered with architecture, controls and evidence.

Next in the series: Traceability: Who (or What) Wrote This Line of Code?

Originally published by Reza Arani on Medium on June 1, 2026. Adapted for Aipolix as Part 6 of the AI Governance in an AI-Native Software Development Company series.