AI coding agent security benchmarks expose a correctness gap

At a glance

What this is: This benchmark compares popular AI coding agent and model combinations on functional and security correctness across real-world tasks, and shows that high code quality does not translate into secure output.

Why it matters: It matters because organisations adopting coding agents need controls for model selection, task scoping, and review, or they will treat insecure automation as acceptable engineering output.

By the numbers:

• The benchmark scores 26 agent and model combinations on real-world tasks.
• 84.9%.
• 29%.
• The benchmark draws 200 tasks from 108 open-source Python projects spanning 77 CWE vulnerability classes.

👉 Read Endor Labs' benchmark on AI coding agent security and functional correctness

Context

AI coding agents are now being evaluated on two different axes that enterprise teams should not collapse into one: whether the generated code works, and whether it is secure. A benchmark that reports both functional and security correctness is useful because production risk often appears only after code passes basic tests. For IAM, PAM, and NHI programmes, the same pattern applies to agentic systems that can generate, modify, or deploy code on behalf of teams.

The governance gap is that productivity tooling is often adopted faster than security evaluation. When an agent can produce apparently correct output while still failing security tests, the issue is not just model quality, it is trust boundaries around delegated software work. That makes this topic relevant to agent identity, approval scope, and human review controls, especially where coding agents sit inside software delivery pipelines.

Key questions

Q: What fails when AI coding agents are judged only by functional correctness?

A: Security failures stay hidden when teams reward only working output. An agent can pass functional tests while still introducing insecure validation, unsafe dependencies, or vulnerable patterns. That is why release decisions need a second gate for security correctness, owned by security or platform engineering rather than only the development team.

Q: Why do AI agents make non-human identity governance harder?

A: AI agents make governance harder because they can request tools, act autonomously, and change behaviour across sessions while still relying on machine credentials. That increases the number of access paths security teams must supervise. The result is a stronger need for task-scoped access, explicit ownership, and continuous monitoring of what the agent can reach.

Q: How do security teams know whether an AI agent is operating safely?

A: Security teams know an AI agent is operating safely when its permissions, invoked tools, and accessed data remain consistent with the approved use case over time. Useful signals include restricted data exposure, unchanged guardrails, and a stable identity path. If any of those drift, the agent should be re-reviewed before it expands further.

Q: How should security teams govern AI-generated code in production environments?

A: Security teams should treat AI-generated code as normal production code with extra provenance risk. Require architectural review, test coverage, static analysis, and approval before merge. Then bind the agent and the build pipeline to least privilege, short-lived credentials, and complete audit logging so implementation speed does not outrun control.

Technical breakdown

Functional correctness and secure correctness are not the same thing

Functional correctness measures whether generated code passes the task requirements and test suite. Secure correctness adds a second filter: whether the same output avoids known vulnerabilities, unsafe patterns, or policy violations. In AI coding agents, these metrics often diverge because a model can infer the intended software behaviour while still introducing insecure defaults, weak validation, or dangerous dependency handling. That divergence is what makes benchmark design matter. A leaderboard that only rewards working code creates false confidence, because the security defects remain hidden until review, testing, or exploitation catches them.

Practical implication: evaluate coding agents against both functional and security criteria before allowing them into delivery pipelines.

Why benchmark harness design shapes what teams learn

A benchmark is only as useful as its task design and anti-cheating controls. Real-world tasks drawn from open-source projects help reduce the gap between lab scoring and production relevance, while prompt hardening, workspace sanitisation, and automated cheating detection reduce score inflation. Those safeguards matter because an agent can appear strong if it is accidentally optimised for benchmark artefacts rather than genuine software work. For security teams, the architectural question is not whether a leaderboard exists, but whether its conditions resemble the environments where code is actually written, reviewed, and merged.

Practical implication: demand benchmark methods that reflect production workflows, not simplified lab conditions.

AI coding agents behave like delegated development identities

An AI coding agent is not just a model that suggests text. In practice it acts as a delegated runtime entity that can create files, edit code, and sometimes trigger downstream tooling. That makes it relevant to identity governance because the agent inherits some of the privileges, trust assumptions, and audit gaps normally associated with service accounts or automation identities. If a coding agent can change code faster than humans can inspect it, then access scope, approval boundaries, and traceability become governance controls, not administrative details. The security problem is therefore as much about delegated identity as it is about code generation.

Practical implication: treat coding agents as governed runtime identities with explicit privileges, logs, and approval boundaries.

Threat narrative

Attacker objective: The attacker objective is to get insecure code accepted into the software supply chain by exploiting trust in agent-generated output.

1. Entry occurs when a coding agent is trusted inside the development workflow and given access to codebases, prompts, and tool execution.
2. Escalation happens when insecure but functional code is merged because review processes focus on correctness more than security context.
3. Impact follows when insecure changes reach production, expanding attack surface through weak validation, unsafe dependencies, or vulnerable logic.

NHI Mgmt Group analysis

Security correctness is now the governing metric for AI coding agents. A tool that produces working code but fails security tests should not be treated as deployable automation. The benchmark makes visible a recurring mistake in AI adoption: organisations optimise for developer velocity first and assume security can be layered on later. That assumption breaks when insecure output is already in the pipeline. Practitioners should treat secure correctness as a release criterion, not a post-hoc review step.

AI coding agents create a delegated identity problem, not just an AI quality problem. When an agent can write and modify code inside trusted environments, it behaves like a non-human identity with tool access and workflow permissions. That puts the topic in the same governance family as service accounts and automation identities, even though the underlying technology is different. The named concept here is development identity drift: the point at which agent privileges and review expectations no longer match the trust the organisation has placed in the system. Teams need to close that drift before it becomes normalised.

Benchmark leadership can mask security underperformance. A high functional score creates a misleading signal if the same model performs poorly on secure output. That is a governance issue for procurement, engineering leadership, and risk teams because model selection based on productivity alone can increase downstream remediation cost. The right question is not which agent writes the best code, but which one can operate within acceptable security boundaries. Practitioners should align selection criteria to both SDLC risk and identity governance.

Anti-cheating controls should now be standard in AI evaluation pipelines. Prompt hardening, workspace sanitisation, and automated cheating detection are not niche benchmark details, they are the difference between a meaningful assessment and a manipulated one. This matters beyond research because many enterprise evaluations repeat the same weakness: they measure output quality without constraining the agent's environment. Security teams should assume evaluation environments themselves need control design, especially when agentic systems can influence their own success conditions.

OWASP NHI thinking belongs in coding-agent governance. Even when the subject is software generation rather than classic machine identity, the control problem is the same: constrain standing access, bound session scope, and audit every privileged action. That makes the article relevant to NHI governance as well as AI security. Practitioners should extend identity controls to agentic development workflows instead of treating them as mere productivity add-ons.

What this signals

AI coding agents are moving from experimental assistants to governed delivery participants, which means security teams should start modelling them as delegated identities with bounded privileges. The practical shift is toward secure-correctness gates, review logging, and access scoping that can survive audit scrutiny. For broader control design, the NIST Cybersecurity Framework 2.0 and the security control patterns in NIST Cybersecurity Framework 2.0 and NIST SP 800-53 Rev 5 Security and Privacy Controls both reinforce the need for explicit protection and monitoring of automated workflow access.

Development identity drift: this is the point where an agent's effective permissions and the organisation's review model stop matching. Once that happens, teams are no longer governing a model, they are governing an operational identity path that can move code faster than humans can inspect it. The response is to align identity, logging, and release controls around agent actions rather than around developer intent.

Enterprises should expect evaluation infrastructure for AI coding agents to become part of the control surface, not just a research concern. Prompt hardening, sandboxing, and anti-cheating checks are signals that the assessment itself is trustworthy. Where agentic systems reach into secrets, repositories, or deployment paths, identity governance and software supply chain controls now overlap in the same decision point.

For practitioners

• Define secure-correctness gate criteria Require coding agents to pass both functional tests and security tests before code can move beyond review. Make the security gate explicit in pull request policy so teams do not confuse successful compilation with acceptable risk.
• Assign least-privilege access to coding agents Limit agent access to the repositories, secrets, and build tools required for a specific task, and revoke broad permissions by default. Treat the agent like a governed non-human identity, not an informal helper.
• Instrument benchmark-style evaluation in your SDLC Test AI coding agents in an environment that mirrors production workflows, including workspace sanitisation, prompt hardening, and cheat detection. That gives engineering and security leaders a more realistic view of operational risk.
• Separate productivity scoring from release approval Use developer experience metrics to measure speed, but use a different control owner and checklist for secure output approval. That separation reduces pressure to merge unsafe code because it happened to be correct.

Key takeaways

• AI coding agents create risk when functional success is mistaken for secure output, because working code can still embed exploitable flaws.
• Benchmarks that combine real tasks, security scoring, and anti-cheating safeguards give practitioners a more realistic view of agent readiness.
• Treat coding agents as governed non-human identities with least privilege, audit trails, and security release gates before they reach production.

Key terms

• Secure Correctness: Secure correctness is the property of code that both satisfies the task and avoids introducing known security weaknesses. In AI coding agents, it matters because a result can be functionally correct while still being unsafe to ship.
• Development Identity Drift: Development identity drift is the gap between the privileges an AI agent effectively uses and the controls the organisation thinks it has in place. It appears when delegated automation acts with broader or less visible access than intended, creating audit and containment problems.
• Agentic workflow: An agentic workflow is a sequence of tasks executed by an AI agent with some level of tool access and decision authority. In security terms, the workflow matters because it can span multiple systems, identities, and permissions, which makes attribution and revocation harder than with ordinary automation.

What's in the full report

Endor Labs' full article covers the operational detail this post intentionally leaves for the source:

• The full benchmark table with per-agent functional and security scores across 26 agent and model combinations.
• The methodology behind the 200-task workload and the 108 open-source Python projects used for testing.
• The anti-cheating design choices, including prompt hardening, workspace sanitisation, and automated cheating detection.
• The separate research and blog resources linked from the benchmark page for teams that want to evaluate agent security in more depth.

👉 Endor Labs' full benchmark page includes the scoring table, methodology, and linked research resources.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security and identity practitioners apply bounded access and lifecycle controls to automated systems and agentic workflows.