Why do changing codebases make AI security evaluation harder?

Why This Matters for Security Teams

Changing codebases turn AI security evaluation into a moving target. A finding that was valid on Monday may be fixed on Tuesday, while a seemingly safe branch can introduce a new prompt injection path, tool misuse issue, or data exposure in a later commit. The real challenge is not just detection, but traceability across versions, branches, model prompts, and orchestration logic. That is why AI security work increasingly aligns with the NIST AI Risk Management Framework, where governance and continuous monitoring matter as much as initial testing.

Practitioners often underestimate how much the surrounding application changes the security posture of the model. Inference-time controls, tool permissions, retrieval sources, and agent workflows can shift even when the underlying LLM remains the same. If evaluation is not tied to code provenance, commit history, and release artefacts, teams can end up re-validating fixed issues while missing regressions introduced elsewhere in the stack. In practice, many security teams encounter the risk only after a release has already reintroduced the flaw, rather than through intentional change tracking.

How It Works in Practice

Effective evaluation treats the model, prompts, tools, and application code as a single security boundary. That means every test result should be linked to a specific commit, configuration, dependency set, and runtime path. Current guidance suggests treating AI security assessment more like software assurance with model-specific controls, rather than a one-time red-team exercise. The evaluator needs to know whether a failure came from the model, the retrieval layer, the prompt template, the agent policy, or a downstream tool.

A practical workflow usually includes version-aware scanning, regression testing, and change-diff review. For example, teams can compare the behavior of a release candidate against the prior baseline, then re-run the same abuse cases after any change to prompts, system instructions, fine-tuning data, retrieval corpus, or tool scopes. The CSA MAESTRO agentic AI threat modeling framework is useful here because it encourages structured analysis of agent actions, trust boundaries, and failure paths.

• Track evaluation results by commit hash, model version, and environment.
• Re-test known issues after every prompt, policy, or dependency change.
• Separate model behavior from application-layer behavior in the findings.
• Record whether a weakness is patched, mitigated, or still exploitable.
• Keep tool permissions and retrieval sources under change control.

Where agentic systems are involved, the evaluation must also include authorization scope and task chaining. A small code change can alter which tools an agent can call, what context it receives, or how it escalates to higher-risk actions. Anthropic’s Anthropic Project Glasswing reflects the growing need for controlled, research-led evaluation of agent behavior across changing environments. These controls tend to break down when teams ship frequent prompt or toolchain updates without preserving a reproducible test baseline, because the evaluation target changes faster than the review cycle.

Common Variations and Edge Cases

Tighter evaluation often increases release friction, requiring organisations to balance confidence against delivery speed. That tradeoff is especially visible in fast-moving CI/CD pipelines, where the security team may not have the time to re-run every scenario on every commit. Best practice is evolving toward risk-based regression coverage: critical workflows, privileged actions, external tool calls, and sensitive data paths get tested more frequently than low-impact features.

There is no universal standard for this yet, but one common pattern is to prioritise the parts of the system most likely to change security posture. A prompt update may be low risk in one application and high risk in another if it affects retrieval or tool routing. Similarly, fine-tuning changes can be more dangerous than code refactors because they alter model behavior without obvious source-code diff signals. Security teams should also be alert to version drift in evaluation datasets, since stale test cases can create false confidence when the attack surface has evolved.

For organisations building agentic or retrieval-augmented systems, evaluation should account for both direct model output and the actions the system can take after the output. That is where AI security overlaps with NHI governance: the agent may not be a human identity, but it still acts with delegated authority and must be constrained like any other privileged actor. In a changing codebase, that delegation is often the first thing to drift.

Related resources from NHI Mgmt Group

• Why do large codebases make AI-assisted security review harder?
• Why do standing privileges make AI agent security harder?
• Why does AI make patch management harder for identity and security teams?
• Why do AI systems make shared responsibility harder than cloud security did?