Originally drafted 30 September 2026. First published 7 October 2026; revised for publication.
Consider an illustrative service failure. A customer-service assistant retrieves an outdated policy note and tells a customer that a claim is ineligible. The agent sends the response. The error is discovered only after the customer complains, by which time the same note has informed dozens of other replies. The immediate temptation is to adjust a prompt or replace a document. But the operational failure is wider: nobody had a defined way to identify affected cases, stop the system, preserve the evidence, correct the decision or determine whether the knowledge source and review process were still safe to use.
That is an AI incident. An AI incident is an operating signal: when an AI-supported workflow creates or could create harm, an organisation needs a rehearsed way to contain it, correct people’s outcomes and learn before repetition turns a local defect into institutional risk. A feedback button and a vendor support ticket do not meet that need.
This matters as teams move into customer service, document processing, research, fraud review, case triage and internal decision support. The systems combine sources, retrieval, prompts, permissions, interfaces and people. A response model has to cover that chain.
Failure becomes different when it affects a person or decision
Models can produce incorrect, incomplete or inappropriate outputs without automatically creating an operational incident. A bad draft discarded by an experienced employee may be useful evaluation evidence but causes no external harm. The threshold changes when an output influences a payment, eligibility decision, customer communication, safety action, reputation, rights or access to a service. It also changes when a system exposes sensitive data, bypasses an approved control, or repeatedly consumes staff time to correct hidden mistakes.
NIST’s AI Risk Management Framework is voluntary guidance intended to help organisations manage AI risk. Its companion Generative AI Profile, published on 26 July 2024, identifies risks including confabulation, information integrity, data privacy and harmful bias. It does not prescribe a universal incident taxonomy. Its value here is the discipline it reinforces: risks must be governed, mapped, measured and managed in their deployment context rather than inferred from a model’s general capability.
Classify the consequence, not the embarrassment. An odd response in a low-risk internal brainstorming tool needs a proportionate response. A confident but unsupported answer given to a patient, borrower, applicant or supplier needs an accountable one. Teams should classify incidents by the credible effect on people, service delivery, legal or contractual commitments, security and data exposure, as well as by how many cases may be affected.
This avoids two opposite failures. The first is treating every imperfect output as a crisis, which overwhelms teams and teaches staff to ignore alerts. The second is treating a repeated pattern as mere “model behaviour”, after it has started influencing real outcomes. The threshold should be defined by the service owner before launch, with a route for frontline staff and affected users to report concerns monitoring cannot detect.
The AI system is a chain of accountable components
When an assistant makes a harmful recommendation, the model may not be the primary cause. The relevant source may be obsolete. A retrieval rule may have prioritised the wrong document. A policy owner may not have published the change. A tool permission may have allowed an action without review. A person may have been given too little time or context to detect the error. Treating the incident as a single prompt defect can conceal the actual control failure.
A response record should therefore identify the full path: intended task; model and version where available; instructions; source material or retrieved records; tools called; permissions used; output delivered; human review or override; and final service outcome. This is not a demand to store every interaction indefinitely. Logging should use purpose limitation and data minimisation, with retention, access and disclosure rules set for the relevant jurisdiction and service. It is a requirement to preserve enough evidence to investigate a consequential event without relying on memory or vendor dashboards.
Traceability is not the same as explanation. A log can show that an approved document was retrieved; it cannot prove that the document was correct, current or suitable for the user’s circumstances. A model may provide a rationale that sounds persuasive while failing to reflect how the output was generated. Operational traceability is more modest and more useful: it records what system components did, what evidence was presented, who was authorised to act and what happened next.
For organisations working across paper records, spreadsheets, messaging and formal platforms, this should include the last mile. If a local officer changes a case after an AI recommendation, can the central team see it? If a system is unavailable and a manual workaround is used, can the final state be reconciled? If staff work in more than one language, are recurring errors visible rather than hidden in an English-only monitoring view? An incident process that only observes the central application will miss where failures emerge.
Containment has to be designed before the launch
A production AI system needs more than an “off” switch. The right intervention depends on the workflow. Teams may need to pause a retrieval collection, revoke a tool permission, disable an automated response, route a defined class of cases to human review, roll back a prompt or model version, or remove a compromised integration. Each action has consequences: a sudden shutdown can create a backlog; continued operation can extend harm.
The UK National Cyber Security Centre’s guidelines for secure AI system development are written for a security context, but their lifecycle framing is useful beyond cybersecurity. Secure design needs ownership, risk management, testing and incident processes rather than a single end-stage check. AI response design should be treated similarly: a service team needs named authority to make a safe intervention, clear criteria for escalation and a tested way to restore a controlled service.
Contain the risky function, not necessarily the whole organisation. If an assistant is returning outdated policy guidance, the immediate action may be to remove the affected corpus and require human approval for that request type. If an automated classification is producing unsafe false positives, stop that class from triggering downstream action while preserving the rest of the service. Broad shutdown can be right in severe cases, but a granular design makes it possible to respond quickly without creating a second avoidable failure.
This requires practical preparation. Maintain an inventory of live AI-supported workflows, their owners, authorised actions, source systems and support contacts. Keep version and configuration records adequate to identify what changed. Give frontline teams a route to flag a suspected problem without diagnosing model behaviour. Run a short simulation before an important deployment: introduce an obsolete source, a missing identifier, an adversarial instruction or a failed external API, then measure whether the team can detect, contain and recover.
Correction is part of accountability
Many response plans end with a technical fix. That is insufficient where people have received an incorrect answer, been delayed, been denied a service or had their data exposed. The institution must decide who needs to be contacted, which decisions should be reviewed, what correction or remedy is appropriate and how a person can contest an outcome. The answer depends on the sector, law, contractual commitments and severity of the event; it cannot be replaced by a generic model disclaimer.
A correction process should identify the affected population rather than wait for complaints. If an error came from a versioned knowledge base, use interaction and case records to determine which outputs relied on it. If a system made a recommendation rather than a final decision, review whether the human approval genuinely intervened or simply confirmed it. If no adequate audit evidence exists, that is itself a control finding and should limit the system’s future authority.
Human oversight must include correction authority. Staff cannot be said to supervise a system if they can flag a problem but cannot pause a decision, amend a record, inform an affected person or obtain a timely escalation. This is particularly relevant in decentralised public services and distributed commercial operations, where the people closest to the service failure may not control the central platform. Give them visible escalation paths and a defined time in which the accountable owner must respond.
Local context also shapes what credible redress looks like. A correction route that assumes personal email, reliable connectivity or familiarity with a formal complaint process will miss people using shared devices, assisted channels or local-language support. Design routes that work through the actual service ecosystem while protecting privacy and preventing intermediaries from taking control of an account or claim.
Incident data should improve the system, not just close the case
The OECD AI Incidents and Hazards Monitor catalogues reported AI incidents and hazards to support learning about real-world impacts. Its records are not an exhaustive census or an independent adjudication of every reported event. An external registry cannot replace an organisation’s own operational evidence. Its underlying lesson is useful: incidents become more valuable when patterns can be examined, classified and used to improve controls rather than being treated as isolated anecdotes.
Internally, a short review should ask what failed, why the existing control did not prevent or detect it, how many cases were affected, what was corrected and whether the system’s authorised scope should change. Avoid a performative blame exercise. The aim is to learn whether the issue lies in data stewardship, retrieval, model behaviour, tool permissions, interface design, user training, supervision or a policy that has not been translated into operational rules.
Useful measures are time to detect and contain; affected cases; corrections; recurrence by workflow; source-retrieval failures; overrides; and unresolved escalations. Track them by language, channel or user group where relevant and lawful, so that a system that appears to improve overall performance does not impose more correction work on a particular group.
Xelius supports organisations to map AI-supported workflows, establish evidence and escalation paths, evaluate systems and build practical governance into software and data delivery. A sensible starting point is one live or proposed use case where the output can influence a consequential communication or decision, then running a contained incident simulation before expanding its authority.
The mature question is not whether a team can eliminate every AI error. No complex service can make that promise. It is whether the organisation can recognise a meaningful failure, stop it at the right boundary, correct the outcome and use the evidence to make the next decision safer. That is what turns AI governance from a policy document into an operating capability.