TL;DR: Scale, not demos, is the bottleneck in AI remediation, and Nullify says the workflow breaks when validation, deduplication, prioritisation, reachability, and follow-through must repeat across an estate. Security teams need governance that turns agent output into measurable completion, not another queue.
Editorial analysis by NHI Mgmt Group, based on content published by Nullify: “DIY AI Product Security: Buy vs Build - A Task Is Not a System”.
By the numbers:
- 95% of enterprise GenAI pilots deliver no measurable impact to the bottom line, according to MIT's 2025 State of AI in Business study cited by Nullify.
- Tools bought from external vendors succeed roughly twice as often as the ones built internally, according to MIT's 2025 State of AI in Business study cited by Nullify.
Key questions
Q: When should teams prioritise buying an AI security system over building one internally?
A: Teams should prioritise buy when the workflow must scale across many repositories, multiple telemetry sources, and ongoing model change.
Q: What are the signs that an AI remediation pipeline is failing?
A: Look for repeated PRs on low-value findings, conflicting severity signals, manual triage after the agent has already acted, and unresolved issues that never make it into the pipeline.
Q: What breaks when AI agent harness controls are not governed like production security controls?
A: The agent can still stay within its technical permissions and yet behave unsafely because prompts, tool routing, retrieval, and escalation logic are unconstrained.
Practitioner guidance
- Define the control boundary around the agent Specify exactly what the AI agent may read, change, and approve.
- Measure the harness, not just the output Track validation pass rate, deduplication quality, false-positive suppression, and merge completion as programme metrics.
- Normalise findings before the agent sees them Correlate scanner, reachability, cloud, and secrets data into one evidence layer so the agent is not making decisions from conflicting severities or duplicate issue records.
Bottom line: The article argues that the core risk is not whether AI agents can write fixes, but whether the surrounding workflow can validate, prioritise, and finish them reliably.
Glossary terms: AI Remediation, Toolchain drift, Delegated Remediation, Signal Normalization
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
AI security automation now lives or dies on the harness, not the model. The article's strongest point is that a useful coding agent can still sit inside a brittle operating model when routing, validation, deduplication, and follow-up are bolted on afterwards. That is an AI governance problem, not just a tooling preference. Named concept: harness debt is the accumulation of missing orchestration, validation, and control logic around model output, and it becomes the hidden failure mode of DIY security automation. Practitioners should treat the control layer as the thing that determines whether the system scales.
A few things that frame the scale:
- The average estimated time to remediate a leaked secret is 27 days, despite 75% of organisations expressing strong confidence in their secrets management capabilities, according to the State of Secrets in AppSec.
- The average time to mitigate a leaked secret is 36 hours, highlighting the operational burden of manual remediation processes, according to the 2024 State of Secrets Management Survey.
A question worth separating out:
Q: How should security teams govern AI agents that can read tickets and open pull requests?
A: Treat the agent as a governed non-human identity with separate permissions for reading, analysing, and writing. Require explicit approval before code changes are created or merged, log every context source the agent used, and limit the ticket types that can invoke it. The goal is to preserve accountability while still reducing routine triage effort.
👉 Read our full editorial: DIY AI security systems fail without the harness around the model