Why agent incidents are different
An ordinary SaaS incident usually asks a familiar set of questions: who logged in, what changed, what data moved, and how fast can we contain it. Agentic AI adds two ugly variables. First, the system may chain multiple tools and decisions before a human even notices. Second, the reasoning path isn't just application logic. It includes prompts, memory, retrieved context, approval rules, and external tool responses.
That changes the logging burden immediately. If your records stop at API success and error codes, you're blind to the actual incident path. And if you're blind to the path, the post-incident review collapses into hand-waving.
Control principle: log enough to replay the decision chain, not just enough to prove the service was running.
The detection signals you should monitor first
Don't overcomplicate the first version of your playbook. Start with the signals that reveal containment-worthy behaviour fast.
- Unexpected tool invocation: the agent calls a tool outside its normal workflow or outside its approved allowlist.
- Privilege anomalies: token use, role changes, or access requests exceed the expected scope for that agent.
- Runaway loop behaviour: repeated retries, escalating prompts, mass actions, or unusual transaction volume.
- External communication anomalies: outbound messages, file transfers, or third-party calls outside policy windows.
- Memory drift: the agent begins reusing stale, poisoned, or inappropriate context across sessions.
These signals should route into the same operating rhythm as security alerts. Ticket, severity, owner, decision, timeline. The mistake is treating agent incidents as a research problem. In production they are an operations problem.
Who can approve shutdown
This needs to be boringly clear. A surprising number of teams still have no written answer to who may suspend an agent, revoke its credentials, or disable its orchestration workflow. That is not a documentation gap. It is a control failure.
| Decision | Primary authority | Fallback authority | Required evidence afterwards |
|---|---|---|---|
| Pause workflow | Service owner or SOC lead | On-call platform lead | Alert source, timestamp, reason code |
| Revoke credentials | IAM or security engineering | Privileged access admin | Credential list, revocation log, affected systems |
| Disable tool access | Platform owner | Runtime administrator | Tool inventory, allowlist delta, rollback record |
| Declare material incident | Incident commander | CISO delegate | Severity rationale, stakeholder notifications |
| Return to service | Business owner + security sign-off | Risk committee delegate | Root cause, remediation, retest results |
The business owner should not have veto power over emergency containment. That creates dangerous delay. They do, however, need to be in the recovery decision, because restoring an agent to service is a business risk acceptance event.
What to log during normal operations so incidents are survivable
The time to decide your log schema is before the incident. At minimum, retain:
- Prompt and response events: request payload, model identifier, timestamp, and session or run ID.
- Tool-call records: tool name, parameters, result code, destination system, and execution duration.
- Memory interactions: what context was retrieved, written, updated, or deleted.
- Approval events: who approved, what threshold was triggered, and how long the workflow paused.
- Configuration state: model version, policy rules, allowlist version, and identity scope at the time of execution.
One warning: over-collecting without access controls is its own problem. Prompt and memory logs can contain sensitive material. Security teams need retention rules, role-based access, and a clear decision on what is masked or segregated.
Evidence preservation checklist
When an incident starts, teams instinctively focus on stopping the harm. Correct. But evidence evaporates quickly in agentic systems, especially where session memory and ephemeral containers are involved. Your preservation checklist should capture:
- Run IDs, session IDs, tenant IDs, and timestamps.
- Raw prompt chain and retrieved context snapshots where policy permits.
- Tool-call sequence with parameters, outputs, and network destinations.
- Approval events, rejections, overrides, and operator chat or ticket notes.
- Runtime configuration, credential scope, and policy version in force at the time.
Think of it like a flight recorder. You don't need every millisecond of noise forever. You do need the sequence that explains how the system crossed from normal execution into unsafe behaviour.
What the post-incident review must decide
A decent post-incident review for agentic AI should answer five questions:
- Was the action within the approved autonomy boundary or outside it?
- Did the control fail because the rule was missing, the rule was bypassed, or the signal was ignored?
- Was the shutdown path technically available and operationally clear?
- Which artefacts now need updating: inventory, risk register, vendor review, runbook, approval matrix?
- Should the agent return to service at all?
Harsh truth: some agent use cases are not ready for production, and the post-incident decision should be decommission, not retest. Teams hate hearing that. Auditors tend to respect it.
FAQ
Do we need a separate incident response playbook for AI agents?
You need an extension to the existing playbook, not a totally separate universe. Reuse your incident command structure but add agent-specific logging, evidence, and shutdown paths.
Who should have emergency shutdown authority?
Someone operationally on point: the service owner, SOC lead, or platform on-call role. Business approval should not be a precondition for emergency containment.
What is the most common logging gap?
Teams log the API event but not the tool-call sequence, approval chain, memory retrieval, or configuration state. That makes root-cause analysis far harder than it needs to be.