DIY AI security systems fail without the harness around the model

At a glance

What this is: This analysis says AI coding agents can generate useful remediation PRs, but DIY AppSec workflows fail when teams try to operate them as a complete security system.

Why it matters: It matters because IAM, AppSec, and platform teams need to govern the workflow around the agent, including validation, prioritisation, and tool integration, not just the model prompt.

By the numbers:

• 95% of enterprise GenAI pilots deliver no measurable impact to the bottom line, according to MIT's 2025 State of AI in Business study cited by Nullify.
• Tools bought from external vendors succeed roughly twice as often as the ones built internally, according to MIT's 2025 State of AI in Business study cited by Nullify.

Context

AI security automation is the problem space here, not the model itself. A task can be delegated to an agent, but a system has to decide what matters, validate what it changes, and keep operating as the environment shifts. That distinction is why a polished demo can still mask a weak programme.

The identity angle is real even when the article is framed as AppSec. Once an AI agent is allowed to triage findings, open pull requests, and touch multiple repositories, it becomes part of the control plane. AI agent authorisation is no longer a theoretical issue when the agent is doing work that previously sat with humans.

Key questions

Q: What breaks when AI agent harness controls are not governed like production security controls?

A: The agent can still stay within its technical permissions and yet behave unsafely because prompts, tool routing, retrieval, and escalation logic are unconstrained. That creates a control gap where intent is translated into action without a reliable enforcement layer. The practical failure is not just misuse, but ambiguous boundaries that make unsafe execution possible without obvious policy violation.

Q: When should teams prioritise buying an AI security system over building one internally?

A: Teams should prioritise buy when the workflow must scale across many repositories, multiple telemetry sources, and ongoing model change. A build can work for a narrow use case, but operating the full harness usually becomes the expensive part. If the business needs repeatability, coverage, and forecastable cost, the build path needs a very strong justification.

Q: What are the signs that an AI remediation pipeline is failing?

A: Look for repeated PRs on low-value findings, conflicting severity signals, manual triage after the agent has already acted, and unresolved issues that never make it into the pipeline. Those signs show the agent is consuming noisy telemetry rather than improving security decisions. A healthy workflow should reduce review burden, not add another queue for engineers to manage.

Q: How should security teams govern AI agents that can read tickets and open pull requests?

A: Treat the agent as a governed non-human identity with separate permissions for reading, analysing, and writing. Require explicit approval before code changes are created or merged, log every context source the agent used, and limit the ticket types that can invoke it. The goal is to preserve accountability while still reducing routine triage effort.

Technical breakdown

Why a remediation PR is not a security system

A remediation pull request is a single output, not an operating model. The article's core point is that a useful task-doer can still fail at the system level because the surrounding controls are missing: signal quality, validation, deduplication, prioritisation, and merge follow-through. In practice, the weak point is the orchestration layer between scanner output and developer workflow. If the agent receives noisy findings, it will generate confident fixes for low-value issues and waste human review time. Practical implication: treat the harness as the security product, not the model prompt. See the AI Coding Agents Security Guide.

Practical implication: Measure the workflow as an end-to-end control, not as isolated agent output.

How tooling drift turns AI remediation into an integration problem

The article shows that most DIY builds are not just about model output. They are stitched together from scanners, cloud context, runtime data, secrets detection, and sometimes an ASPM layer, each with its own API, data model, and severity scale. That creates a correlation problem before any fix is even written. If one vulnerability appears three times with three IDs and different severities, the agent has to guess which signal to trust. Practical implication: without a normalised evidence layer, the agent becomes a consumer of conflicting security telemetry rather than a reliable decision engine. See the Agentic AI Security Guide.

Practical implication: Normalise security signals before letting an agent act on them.

Why cost and coverage decide whether the workflow scales

This article separates the demo from the estate. A one-off fix in a single repository is easy; running the same pattern every day across thousands of repositories, with validation, routing, and follow-up, is a different operating problem. The ceiling is also defined by coverage. If scanners miss app-contextual secrets, authorization flaws, or other findings, the agent never sees them. Practical implication: assess whether the workflow improves security coverage and operating cost together, because either dimension alone gives a misleading picture. See the AI Agent Identity Security Buyer's Guide.

Practical implication: Audit coverage and unit economics together before scaling the workflow.

Breaches seen in the wild

• Amazon Q Developer extension compromise 2025: An over-scoped CI token let an attacker ship a prompt telling Amazon Q's AI coding agent to wipe files and AWS resources (CVE-2025-8217).
• PocketOS database deletion incident: An AI coding agent found an over-privileged Railway API token in the codebase and deleted PocketOS production data and backups in nine seconds.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI security automation now lives or dies on the harness, not the model. The article's strongest point is that a useful coding agent can still sit inside a brittle operating model when routing, validation, deduplication, and follow-up are bolted on afterwards. That is an AI governance problem, not just a tooling preference. Named concept: harness debt is the accumulation of missing orchestration, validation, and control logic around model output, and it becomes the hidden failure mode of DIY security automation. Practitioners should treat the control layer as the thing that determines whether the system scales.

The real comparison is not agent versus human, but task versus system. A task can be completed by a model, yet an enterprise control process has to decide what to fix, whether the fix is real, and whether the issue deserves human time. That distinction matters for AI governance because it changes where accountability sits: in the workflow design, not in the prompt. Teams should evaluate whether their current setup can sustain decision quality across the estate, not merely produce a convincing artifact.

DIY AI remediation becomes an identity and access problem as soon as the agent can act across multiple repositories and tools. Once an agent opens pull requests, consumes scanner output, and touches cloud or CI context, its authority has to be bounded like any other delegated capability. That makes AI agent authorisation directly relevant, because the question is no longer whether the agent can write code, but what it is allowed to decide and change. Practitioners should govern the agent as a delegated actor, not as a script with a chat interface.

Coverage gaps matter more than demo quality. The article rightly notes that scanners cannot remediate what they never detect, which means the security outcome is bounded by discovery quality before the model ever enters the loop. That shifts the programme question from “Can the agent fix findings?” to “What classes of risk are invisible to this pipeline?” Teams should use this as a reminder that automation cannot compensate for incomplete telemetry or narrow detection scope.

Buy versus build is really a question of who owns operational complexity. The article's market signal is that the difficult part of AI security is not access to frontier models, but the continuous work of keeping the system aligned with changing models, changing attacks, and changing codebases. That pushes the category toward managed control planes and away from isolated proof-of-concept workflows. Practitioners should judge vendors and internal builds by the resilience of the operating model they create, not by the elegance of a single generated PR.

From our research library:

• The average estimated time to remediate a leaked secret is 27 days, despite 75% of organisations expressing strong confidence in their secrets management capabilities, according to the State of Secrets in AppSec.
• The average time to mitigate a leaked secret is 36 hours, highlighting the operational burden of manual remediation processes, according to the 2024 State of Secrets Management Survey.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Harness debt: the hidden risk in DIY AI security is not the model's capability but the accumulation of missing routing, validation, and follow-through around it. Once those controls are split across scanners, chat prompts, and human review, the programme starts to behave like a toolchain instead of a governed system. Teams should watch for any workflow where the agent is doing visible work while control quality quietly degrades.

The more repositories and telemetry sources you connect, the more the workflow becomes an identity and governance problem. If an AI agent can act on findings across the estate, the programme needs delegated authority, bounded access, and a reliable audit path. AI agent authorisation is the right lens when the agent starts to look less like automation and more like a privileged participant in the security process.

For practitioners

• Define the control boundary around the agent Specify exactly what the AI agent may read, change, and approve. Bound its access to repositories, CI context, and ticketing systems so it cannot wander beyond the remediation workflow you intended.
• Measure the harness, not just the output Track validation pass rate, deduplication quality, false-positive suppression, and merge completion as programme metrics. A PR that looks good but consumes review time is not a net security win.
• Normalise findings before the agent sees them Correlate scanner, reachability, cloud, and secrets data into one evidence layer so the agent is not making decisions from conflicting severities or duplicate issue records.
• Test coverage against missed finding classes Look for app-contextual secrets, authorization flaws, and other issues that your current scanners fail to surface. If discovery is incomplete, the agent only automates the visible slice of risk.
• Compare build and buy on operating cost Estimate the ongoing cost of maintaining integrations, validation logic, routing rules, and model updates before you decide the workflow is worth owning internally.

Key takeaways

• The article argues that the core risk is not whether AI agents can write fixes, but whether the surrounding workflow can validate, prioritise, and finish them reliably.
• The evidence in the piece points to a scale problem, not a demo problem, because the same process that works on one repository struggles when repeated across an estate.
• The practical implication is to govern AI remediation as a control system with scoped authority, normalised telemetry, and measurable follow-through.

Related NHI Mgmt Group resources: AI Coding Agents Security Guide · AI Agent Identity Security Buyer's Guide · AI Agent Authorisation Guide · AI Agent Observability, Audit and Incident Response Guide

Key terms

• AI Remediation: AI remediation is the use of machine learning and contextual analysis to improve how security teams prioritise, validate, and automate fixes. It is not just auto-patching. In practice, it helps reduce noise, judge risk more accurately, and choose safer remediation paths across software delivery workflows.
• Toolchain drift: Toolchain drift is the gradual mismatch between the build environment a project expects and the tools actually present on the host. In embedded Linux, it can change command semantics, syscall paths, and privilege simulation outcomes, creating hard-to-diagnose failures without altering source code.
• Delegated Remediation: Delegated remediation is the transfer of incident-response action from a human operator to a software identity that can propose or execute fixes. The key governance issue is not the quality of the recommendation, but whether the delegated actor has the authority to change state, and whether that authority is reversible and auditable.
• Signal Normalization: Signal normalization is the process of converting security findings from different tools into one common structure. It lets teams compare alerts, ownership, severity, and context across IAM logs, scanners, cloud events, and pipeline outputs without manually reconciling every source.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and the control decisions that govern delegated access. It is designed for practitioners who need to connect identity governance to the broader security operating model.