Join our Newsletter — 33% off our NHI Course

AppSec triage harnesses: what changes when prompts are not enough?

 

(@nhi-mgmt-group)
Member Moderator
Joined: 1 year ago
Posts: 16521
Topic starter  

TL;DR: AI triage on SAST and SCA findings can swing between true positive and false positive on the same evidence, exposing why prompt-driven classification is too unstable for enterprise AppSec governance, according to Pixee. The real control surface is the surrounding harness, where policy, reporting, memory, caching, and deterministic integrations make decisions auditable and repeatable.

Editorial analysis by NHI Mgmt Group, based on content published by Pixee: “A Prompt Is Not a Security Function”.

Key questions

Q: What breaks when AppSec triage is driven by prompts alone?

A: Prompt-only triage breaks repeatability.

Q: Why do identical SAST findings need a governed harness?

A: Because the finding is not the problem, the decision process is.

Q: How do organisations know if AI triage is actually working?

A: Measure whether the AI improves high-fidelity detection, shortens time to verified response, and preserves reviewer trust in its decisions.

Practitioner guidance

  • Define triage adjudication rules Document how your programme classifies latent risk, exploitability, accepted risk, and false positives so the same finding is treated consistently across runs and repositories.
  • Separate prompting from decision logic Use prompts to assist analysis, but keep the final classification, precedence rules, and exception handling in governed workflow logic.
  • Add audit trails for model-influenced decisions Record the evidence, policy inputs, prior rulings, and final outcome for each adjudicated finding so reviewers can reconstruct why the system chose that label.

Bottom line: Prompt-only AppSec triage is unstable because the same evidence can receive different labels across repeated runs, which makes the result hard to trust in an enterprise setting.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →



   
Quote
(@mr-nhi)
Member Moderator
Joined: 5 months ago
Posts: 17012
 

Prompt-only triage is not a security function because it cannot guarantee governance consistency. A model can describe a finding, but it cannot by itself encode the organisation's precedence rules, exception logic, and risk tolerance. That makes prompt-first triage fragile in any programme that needs repeatable decisions across teams and repositories. The practitioner conclusion is simple: the decision system must exist outside the prompt.

A question worth separating out:

Q: What should teams do when a critical finding is not actually exploitable?

A: They should re-score it using environment context, document the compensating controls, and keep it in a tracked risk queue rather than treating it as an emergency. The goal is not to ignore the issue but to place it in the right priority band so it does not displace findings with real exploitation paths.

👉 Read our full editorial: A prompt is not a security function for AppSec triage



   
ReplyQuote
Share:

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.