Notifications
Clear all
Topic starter
06/09/2026 9:34 am
TL;DR: Changing only the harness around the same model can create up to a 6x performance gap, showing that context management and orchestration matter more than raw model capability for security operations, according to Crogl. The real security issue is not model access but whether the surrounding control plane can preserve investigative context, avoid repetition, and keep decisions grounded in the right evidence.
NHIMG editorial — based on content published by Crogl: Why Can’t I Just Vibe Code My Own Harness?
Questions worth separating out
Q: What is the biggest risk in letting AI systems drive security investigations?
A: The biggest risk is not model error alone, but uncontrolled context handling.
Q: Why do AI-enabled security tools need governance beyond traditional security controls?
A: AI-enabled tools introduce non-deterministic behaviour, which means security controls must address both the system and the model lifecycle.
Q: How do security teams know if agentic AI controls are failing?
A: The main signs are session drift, repeated retry loops, unauthorized tool calls, and behaviour that diverges from the documented task sequence.
Practitioner guidance
- Define harness governance boundaries Document what data the AI harness may retrieve, retain, summarise, and forward to tools such as the SIEM or case platform.
- Separate retrieval from reasoning state Store investigation state in structured form, such as entity relationships and prior findings, instead of relying on raw transcript history.
- Test against long-form incident workflows Validate the harness on multi-host, multi-source investigations that force repeated tool use, not on short synthetic demos.
What's in the full article
Crogl's full blog covers the operational detail this post intentionally leaves for the source:
- How the harness logic maps investigation state into knowledge-graph updates and retrieval decisions
- Why context limits change once tool calls start returning large result sets from a SIEM or case platform
- Where the article distinguishes demo-ready orchestration from production-grade investigation reliability
- How the author frames the build-versus-buy trade-off for AI-driven security operations
👉 Read Crogl's analysis of why AI harness design matters in security operations →
AI harness scaffolding for SOC work: are your controls keeping up?
Explore further
06/09/2026 9:34 am
Harness governance is now a security control, not a developer convenience. Once AI systems can retrieve data, call tools, and shape investigation paths, the surrounding orchestration becomes part of the control stack. That shifts attention away from model capability and toward state handling, evidence selection, and bounded action. In practice, SOC teams need to assess harness logic with the same seriousness they apply to privileged automation.
A question worth separating out:
Q: How should organisations decide between building and buying an AI harness?
A: Teams should compare lifecycle burden, not just initial feature fit. A real harness must evolve with new attack patterns, new log sources, and changing model behaviour, so the question is whether the organisation can support continuous orchestration tuning. If not, the build becomes a long-term product commitment rather than a weekend project.
👉 Read our full editorial: AI harness scaffolding is the real gap in security operations
