Why do AI systems complicate traditional risk management in software delivery?

Why This Matters for Security Teams

AI changes the risk profile of software delivery because the system under review is no longer just application code. It includes model artefacts, prompts, retrieval sources, orchestration logic, connectors, and the permissions that let an AI system act on behalf of a user or service. That widens the attack surface and makes classic change management, secure coding, and test gates necessary but insufficient. The NIST Cybersecurity Framework 2.0 is still useful, but it must be applied across the full AI lifecycle rather than only at release time.

Security teams often miss that AI risk is not limited to malicious code insertion. Prompt injection, training data contamination, model drift, unsafe tool use, and weak identity boundaries for agents can all create outcomes that look like ordinary business logic failures until they become security incidents. Current guidance suggests treating AI delivery as a socio-technical control problem: governance, data integrity, access control, monitoring, and human oversight all need to work together.

In practice, many security teams encounter AI-related risk only after a model has already exposed data, taken an unsafe action, or been fed untrusted content through an integration path rather than through intentional threat modeling.

How It Works in Practice

Traditional software delivery assumes that code review, dependency scanning, testing, and change approval will catch most material risk. With AI systems, those controls remain important, but they do not cover several failure modes that emerge after deployment. A model can behave differently depending on the prompt, retrieved context, or upstream data source. An agent can also call tools, create tickets, query systems, or trigger workflows, which means its effective privilege matters as much as its code path.

Operationally, teams need to manage the AI pipeline as a set of linked trust boundaries. That includes model provenance, dataset lineage, prompt handling, connector permissions, and output validation. NIST’s AI risk guidance and the NIST Cybersecurity Framework 2.0 both support this broader view, especially where governance and monitoring need to extend beyond software release gates. MITRE ATT&CK-style thinking is also useful for mapping attack paths, even though the techniques look different from classic web or endpoint compromise.

• Validate training and retrieval data sources before they reach production workflows.
• Restrict connector scopes so agents only reach the systems they actually need.
• Log prompts, tool calls, and model outputs for review and incident response.
• Apply human approval for high-impact actions and sensitive data access.
• Test for prompt injection, data leakage, and unsafe model behaviour during release planning.

Where AI is embedded in CI/CD, software delivery also inherits model update risk, vendor dependency risk, and shifting behaviour from external services. Best practice is evolving toward continuous evaluation rather than one-time validation. These controls tend to break down in highly automated pipelines with weak asset inventory because teams lose track of which model, prompt, or connector version actually made the decision.

Common Variations and Edge Cases

Tighter AI governance often increases delivery overhead, requiring organisations to balance faster experimentation against stronger control over data, model changes, and execution authority. That tradeoff is especially visible when product teams want rapid model updates but security teams need traceability and rollback options.

There is no universal standard for this yet, so the right level of control depends on the use case. A low-risk internal summarisation feature does not need the same review depth as an agent that can approve refunds, alter customer records, or access secrets. In regulated environments, current guidance suggests aligning AI controls to business impact and data sensitivity, then setting stricter gates where the model can influence financial, safety, or identity decisions. The NIST Cybersecurity Framework 2.0 remains useful for framing governance, protection, detection, response, and recovery, but AI-specific threat modeling should be layered on top.

Edge cases also matter. Retrieval-augmented systems can leak sensitive context even when the model itself is not fine-tuned on private data. Open-weight models can be deployed safely, but they still need integrity checks, update control, and usage monitoring. Agentic systems add another wrinkle because the security question is not only what the model says, but what it is allowed to do. In practice, the hardest failures occur when AI output is treated as authoritative without validation at the exact point where it influences code, access, or production decisions.

Related resources from NHI Mgmt Group

• Why do AI agents complicate traditional model risk management in financial services?
• Why do AI systems complicate GDPR compliance more than traditional software applications?
• When does AI create more governance risk than traditional data systems?
• Why do AI systems complicate traditional data security controls?