What should organisations do when AI moves into code generation and software delivery?

Why This Matters for Security Teams

When AI starts generating code, scripts, infrastructure changes, or deployment logic, the risk is no longer limited to code quality. The control problem expands into software supply chain integrity, approval boundaries, and the trustworthiness of automated actions. A model can introduce insecure patterns, select risky dependencies, or produce code that looks valid but bypasses organisational standards. That makes governance of the development pipeline as important as code review itself.

This is also where identity and privilege become operationally relevant. If an AI agent can open pull requests, trigger builds, or call deployment tools, it effectively needs a governed identity, scoped permissions, and traceable approvals. Current guidance from the NIST Cybersecurity Framework 2.0 supports treating these risks as part of broader governance, protection, and detection functions rather than as a pure developer productivity issue. The mistake many teams make is assuming the code scan at the end of the pipeline will catch a trust failure that was introduced at the start. In practice, many security teams encounter AI-assisted build risk only after a compromised dependency or unauthorised release path has already been exercised.

How It Works in Practice

Organisations should treat AI-assisted development as a controlled workflow, not an open-ended productivity layer. That means defining where AI is allowed to assist, what inputs it can see, what outputs require human review, and which actions are forbidden without additional approval. The most effective controls are usually upstream of production, because downstream scanning cannot reliably prove that a generated artefact was built from approved instructions or trusted sources.

Practical governance usually includes:

• Reviewing prompts, system instructions, and tool permissions for AI coding assistants and agents.
• Restricting package installation, dependency updates, and repository writes to approved identities and build contexts.
• Checking provenance for code, artefacts, and external libraries before merge or release.
• Using signed commits, tamper-evident logs, and approval records for changes created or modified by AI.
• Separating assistance from execution, so an AI can suggest code without directly promoting it to production.

For software supply chain integrity, NIST Secure Software Development Framework is useful because it emphasises secure build practices, provenance, and ongoing trust in the delivery pipeline. Where AI systems also have access to repositories, CI/CD tools, or cloud controls, those access paths should be treated like privileged pathways and monitored accordingly. If the AI is using retrieval, external tools, or deployment automation, the organisation should also validate whether those interactions are logged, approved, and reversible. This is where identity governance becomes central: every automated action should be attributable to a named workload, service principal, or agent identity with a defined purpose.

These controls tend to break down when development is highly distributed, the pipeline is heavily automated, and teams rely on unmanaged third-party AI tools because provenance and approval boundaries become opaque.

Common Variations and Edge Cases

Tighter AI governance often increases delivery overhead, requiring organisations to balance developer speed against assurance and auditability. That tradeoff is real, especially where teams are shipping frequently or experimenting with multiple AI assistants at once. Best practice is evolving, and there is no universal standard for this yet, so control design should reflect the organisation’s risk tolerance and release criticality rather than a blanket prohibition.

One common edge case is “suggestion-only” AI tooling. Even if the system cannot deploy code directly, it can still introduce insecure patterns, license issues, or hidden dependencies that enter the repository through human approval. Another is agentic automation that spans code generation, test execution, and release orchestration. In those environments, the issue is not just code correctness but the authority chain behind each step. For threat modelling, MITRE ATT&CK remains useful for understanding how attackers abuse valid accounts, software supply chain trust, and deployment tooling, while OWASP guidance for LLM applications helps teams think about prompt injection, tool misuse, and output trust. Where AI is used to change production systems in regulated environments, current guidance suggests treating those workflows as high assurance paths, with stronger review and rollback expectations. A practical rule is simple: if the AI can influence release outcomes, it needs controls that are closer to production change management than to ordinary coding assistance.

Related resources from NHI Mgmt Group

• What is the difference between code review and access review in AI-generated software?
• Should organisations allow AI tools that can generate attack code?
• How can organisations detect AI-generated passwords in source code?
• How can organisations reduce trust sprawl in software delivery?