AI agent governance is lagging engineering teams’ rapid shift

At a glance

What this is: This is a conference recap showing how AI is reshaping engineering teams and exposing new guardrail and accountability gaps around AI-driven work.

Why it matters: It matters because IAM, NHI, and governance teams now have to account for AI systems that influence code, customer interactions, and operational decisions without inheriting human control assumptions.

👉 Read WorkOS's recap of the Enterprise Ready Conference 2025 CTO panel on AI in engineering

Context

AI is changing engineering workflows faster than many governance programmes were designed to handle. The issue is not just productivity, but that non-deterministic systems are being trusted with customer-facing and code-producing work that still depends on identity, approval, and accountability controls.

For IAM and security teams, the core question is where existing controls assume a human operator, a predictable workflow, or a stable review window. Once AI is drafting code, answering customers, and shaping what users see, those assumptions need to be rechecked across human, NHI, and autonomous identity programmes.

Key questions

Q: How should security teams govern AI tools that help write and review code?

A: Treat AI-assisted development as an identity and control problem, not only an engineering productivity issue. Keep generation, review, and approval separated, limit the data and repositories AI tools can access, and preserve a human owner for any change that affects production behavior or customer trust.

Q: Why do AI-assisted engineering workflows complicate identity governance?

A: Because they extend access beyond a single human user into tools that can read context, draft changes, and shape operational decisions. Even when the system is not autonomous, it can still widen the delegated access surface and weaken accountability if approval paths are unclear.

Q: What do teams get wrong about AI-generated documentation and code review?

A: They often assume documentation or review output is proof of oversight. In practice, if AI generates the work and another AI validates it, the process can become a closed loop unless a separate human applies challenge, context, and responsibility for the final decision.

Q: What should organisations do before AI systems influence customer-facing content?

A: Define escalation thresholds, review ownership, and incident handling for any AI output that can affect brand trust or regulatory exposure. Customer-facing AI should be governed like any other externally visible control point, with clear traceability and a named human accountable for outcomes.

Technical breakdown

Non-deterministic output changes the control model

Large language models are not deterministic components, so the same input can produce different outputs across runs. That matters because traditional application controls assume repeatable behavior, clear test expectations, and bounded failure modes. When an AI system sits inside a delivery workflow, the relevant question is not whether it usually works, but what happens when it produces incorrect, inconsistent, or unsafe output once in every hundred requests. That is a governance problem as much as a software problem, because downstream reviewers may trust the output shape while missing the variability hidden inside it.

Practical implication: require explicit failure handling, review gates, and rollback paths wherever AI output can influence code or customer interactions.

AI-assisted engineering changes the identity surface

When product managers, interns, and engineers use AI tools to create code, summaries, and migrations, the identity surface expands beyond the traditional developer account. The system consuming data, generating output, and initiating actions becomes part of the operating model, even if a human remains nominally responsible. This is not autonomous identity in the strict sense unless the system independently selects actions, tools, and timing. But it is still an NHI governance issue because the tool chain may act with delegated access, broad context, and limited visibility into who actually approved the resulting change.

Practical implication: map which AI tools can read repositories, tickets, or production context, then constrain their delegated access and auditability.

Closed-loop automation still needs human accountability

The panel’s warning about AI generating pull request descriptions while AI tools also review them points to a control loop that can lose meaningful human challenge. In security terms, that is a review integrity issue. Even if the workflow is not fully autonomous, it can still become opaque if the same model family or adjacent tools prepare, summarize, and validate work without independent scrutiny. That weakens the assurance value of code review, documentation, and testing unless a human actually owns the final decision and the system preserves a traceable approval chain.

Practical implication: keep independent human review for changes that affect production behavior, access scope, or customer-facing responses.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI engineering workflows create an identity governance problem before they create an autonomy problem. The article is not describing fully autonomous actors, but it does show that AI tools are increasingly embedded in code creation, documentation, and operational decisions. That means existing IAM and NHI controls now have to govern delegated access, review integrity, and traceability across a wider set of machine-mediated actions. Practitioners should treat this as an expansion of the identity surface, not just a productivity story.

Closed-loop review is the most overlooked failure mode in AI-assisted delivery. When AI writes summaries, AI reviews code, and humans skim the result, the control chain can appear intact while independent verification quietly disappears. That is a governance weakness, not a tooling feature. The implication is that review authority must remain distinct from generation authority, or the process stops proving what it claims to prove.

Answer engine optimization is becoming an identity and trust problem, not only a marketing one. If AI systems now determine what customers see, then content structure, metadata, and brand representation become part of the trust boundary. That links human content governance to machine-mediated delivery paths and makes ownership of externally visible answers part of the broader access model. Practitioners should treat brand-facing AI outputs as governed identity touchpoints.

The engineering org is adopting machine assistance faster than governance can classify the actors involved. Some of the systems described here are advanced NHI patterns, while others are human workflows augmented by AI. The discipline now is to distinguish between tools that assist a human and systems that begin to make independent runtime decisions, because the governance response is different in each case. Practitioners should classify by behavior, not by AI branding.

From our research:

• The average estimated time to remediate a leaked secret is 27 days, despite 75% of organisations expressing strong confidence in their secrets management capabilities, according to The State of Secrets in AppSec.
• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.
• That gap reinforces why teams should read NHI Lifecycle Management Guide when AI-assisted delivery starts to touch secret handling and delegated access.

What this signals

Closed-loop AI review is the next governance blind spot. When the same class of tooling helps create, summarise, and validate work, the control objective shifts from speed to assurance. Teams should look for places where a human still needs to own the final decision, especially when AI-generated changes can affect production code, access scope, or customer communications.

The broader signal for identity programmes is that AI tools are becoming part of the workflow surface long before they become fully autonomous actors. That means access reviews, delegated trust, and audit trails must cover machine-mediated action chains as well as human sign-off. Security teams should prepare now for a world where the control failure is not obvious misuse, but the gradual disappearance of independent challenge.

For practitioners

• Classify every AI-enabled workflow by actor type Separate human-assisted tooling from non-human identity use cases and from systems that make independent runtime decisions. That classification determines whether IAM, NHI, or autonomous governance patterns should apply to the workflow.
• Preserve independent review for production-impacting changes Do not allow the same AI layer to generate and effectively validate the same change without independent human challenge. Keep approval authority separate from content generation for code, access, and customer-facing responses.
• Map delegated data access for AI tools Identify which repositories, tickets, logs, and operational systems AI tools can read or influence. Reduce the scope of delegated access to the minimum needed and log the context they consume.
• Define trust boundaries for brand-facing AI outputs Treat generated customer responses, support replies, and AI-curated content as governed outputs. Set clear escalation paths, review triggers, and incident ownership when an AI system can misrepresent the organisation.

Key takeaways

• AI is already changing engineering governance because machine-assisted workflows expand the identity surface before they become fully autonomous.
• The most dangerous failure mode is a closed review loop where AI generates and effectively validates its own work without independent human challenge.
• IAM and NHI teams should classify AI-enabled delivery paths by actor behavior, then narrow delegated access and preserve accountable review.

Key terms

• Machine-Mediated Workflow: A workflow where software helps create, route, summarise, or approve work that would otherwise be handled directly by a person. The identity concern is not just efficiency, but where delegated access, review responsibility, and accountability sit when machines shape the outcome.
• Closed Review Loop: A control pattern where the same or similar AI systems generate work and then validate it without meaningful independent human challenge. It can look like oversight while actually reducing assurance, because the reviewer no longer provides an external check on the original output.
• Delegated Access: Access granted to a tool or system so it can read data, act on resources, or influence workflows on a human's behalf. In AI environments, delegated access must be tightly scoped because the system may process more context than the user intended.
• Brand-Facing AI Output: Any AI-generated response, summary, or recommendation that can be seen by customers or the public. These outputs sit inside the trust boundary because errors, hallucinations, or inconsistent tone can create regulatory, reputational, and governance risk.

Deepen your knowledge

AI-assisted engineering governance is a core topic in our NHI Foundation Level course, the industry's only accredited NHI security programme. If your team is starting to govern machine-mediated delivery paths, it is worth exploring.

This post draws on content published by WorkOS: CTO panel on how AI is transforming engineering teams. Read the original.