What breaks when insurers try to use AI on claims and underwriting documents without governance?

Without governance, AI systems can misread document type, miss key fields, or act on stale and incomplete content. In claims, that creates fraud blind spots and slow handling. In underwriting, it can lead to poor risk decisions, missed controls, and re-keying work that erodes efficiency. The failure is not the model alone, but the absence of certified data inputs.

Why This Matters for Security Teams

Insurance workflows often look document-heavy rather than security-heavy, which is why governance gaps are easy to miss until an AI system is already influencing decisions. Claims and underwriting documents can contain signatures, policy terms, exclusions, medical information, financial details, and third-party data. If model inputs are not classified, versioned, and approved, AI can accelerate the wrong work just as efficiently as the right work. That is a control failure, not merely a quality issue.

Security and risk teams should treat document governance as part of decision integrity. Current guidance suggests mapping the full document lifecycle, from ingestion to retention, to controls for access, provenance, and auditability. The NIST Cybersecurity Framework 2.0 remains useful here because it ties governance, protection, detection, and recovery into one operational view. In practice, many insurers encounter the problem only after an AI-assisted workflow has already surfaced inconsistent claim decisions or underwriting exceptions rather than through intentional control testing.

How It Works in Practice

Governance has to sit between the document repository and the model, not just around the model itself. That means the organisation needs a trusted intake path, document classification rules, source-of-truth mapping, and explicit permissions over which files can be used for which purpose. In claims, an adjuster note, a scanned repair invoice, and an external fraud file should not be treated as equivalent evidence. In underwriting, a broker submission, an appraisal, and a prior renewal file may each have different authority and freshness requirements.

Operationally, strong programmes usually combine four controls:

• Document provenance checks so the AI can identify the source, owner, and version of each input.
• Data minimisation so the model sees only the fields needed for the task, not the whole file by default.
• Human review thresholds for exceptions, low-confidence extractions, and policy-impacting outputs.
• Audit logging that preserves what was ingested, what was ignored, and who approved the use case.

For AI-specific risk, teams should also consider prompt injection in uploaded documents, model hallucination during extraction, and poisoning through contaminated historical files. The NIST AI Risk Management Framework and MITRE ATLAS both reinforce that the control problem spans data integrity, model behaviour, and adversarial manipulation, not only accuracy. Where insurers use retrieval-augmented generation, the retrieval layer must be governed as tightly as the model because stale or privileged documents can shape the answer without obvious warning. For a deeper control baseline, the OWASP Agentic AI guidance is useful when AI systems can take action on documents rather than merely summarise them. These controls tend to break down in legacy claims platforms with shared file stores and inconsistent metadata because the system cannot reliably prove which document version was used.

Related guidance from NIST AI Risk Management Framework and MITRE ATLAS helps teams separate model risk from document risk, which is essential when AI is used to support financial decisions.

Common Variations and Edge Cases

Tighter document governance often increases operating overhead, requiring organisations to balance speed against assurance. That tradeoff is real in insurance, especially where intake volumes are high and document formats vary widely across brokers, loss adjusters, and regional entities. Best practice is evolving, but current guidance suggests that not every document needs the same level of scrutiny; the classification should depend on whether the file is advisory, evidentiary, or decision-bearing.

Edge cases matter. A scanned PDF with embedded handwriting is not just a poor OCR input, it may be a legal record that requires retention and traceability. A third-party document feed may be technically available yet contractually restricted, which creates a governance issue even if the model can parse it. If a workflow uses agentic AI, the question is not only whether the model can read the document, but whether it is authorised to act on the extracted meaning. That is where ai governance begins to overlap with identity and privilege management.

For regulated environments, governance should also cover privacy boundaries, cross-border data handling, and evidence preservation. There is no universal standard for this yet, so insurers should align controls to the sensitivity of the document class and the impact of the decision being supported. The OWASP AI Security and Privacy Guide and CISA materials are useful references when defining practical safeguards for live insurance workflows.

Related resources from NHI Mgmt Group

• What breaks when AI agents use session-based micropayments without governance?
• What breaks when AI agents use MCP without stronger governance?
• How should organisations use AI agents in access reviews without losing governance control?
• How should organisations use AI in IAM without weakening governance?