AI security task automation: where the DIY approach breaks down

TL;DR: AI coding agents can produce convincing remediation pull requests, according to Nullify, but the real gap is the harness around them: validation, deduplication, prioritisation, reachability, and follow-through. The hard part is not prompting a model, it is operating a reliable AI security system across the estate at forecastable cost.

Editorial analysis by NHI Mgmt Group, based on content published by Nullify: “DIY AI Product Security: Buy vs Build - A Task Is Not a System”.

By the numbers:

• 95% of enterprise GenAI pilots deliver no measurable impact to the bottom line, according to MIT's 2025 State of AI in Business study cited by Nullify.
• Tools bought from external vendors succeed roughly twice as often as the ones built internally, according to MIT's 2025 State of AI in Business study cited by Nullify.

Key questions

Q: What breaks when AI agent harness controls are not governed like production security controls?

A: The agent can still stay within its technical permissions and yet behave unsafely because prompts, tool routing, retrieval, and escalation logic are unconstrained.

Q: When should teams prioritise buying an AI security system over building one internally?

A: Teams should prioritise buy when the workflow must scale across many repositories, multiple telemetry sources, and ongoing model change.

Q: What are the signs that an AI remediation pipeline is failing?

A: Look for repeated PRs on low-value findings, conflicting severity signals, manual triage after the agent has already acted, and unresolved issues that never make it into the pipeline.

Practitioner guidance

• Define the control boundary around the agent Specify exactly what the AI agent may read, change, and approve.
• Measure the harness, not just the output Track validation pass rate, deduplication quality, false-positive suppression, and merge completion as programme metrics.
• Normalise findings before the agent sees them Correlate scanner, reachability, cloud, and secrets data into one evidence layer so the agent is not making decisions from conflicting severities or duplicate issue records.

Bottom line: The article argues that the core risk is not whether AI agents can write fixes, but whether the surrounding workflow can validate, prioritise, and finish them reliably.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →