Start by separating AI-generated code risk from AI asset risk. A useful platform should discover issues in source, repositories, and developer workflows, while also identifying models, agents, MCP servers, and configuration exposure. If it only covers one layer, the uncovered layer remains a blind spot. Strong evaluation looks for coverage across the full AI attack surface, not a single control point.
Why Evaluating Both Layers Matters
An ai security platform is only useful if it sees the whole failure chain. Code scanning can catch insecure prompts, leaked secrets, unsafe dependencies, and risky changes in repositories, but that leaves a separate risk surface if the same platform cannot discover models, agents, MCP servers, and the permissions, secrets, or configuration those components rely on. When teams treat “AI security” as a single layer, they often harden the application while leaving the underlying AI infrastructure exposed.
That distinction matters because code risk and infrastructure risk fail differently. Code risk tends to show up in source control, build systems, and developer workflows; infrastructure risk tends to show up in identity scope, exposed services, overbroad access, and untracked agent behaviour. A platform that only covers one side can still miss the path an attacker would actually use to move from code weakness to operational compromise.
Security teams should therefore evaluate coverage by asking whether the platform can identify both what developers ship and what AI systems are allowed to do. If it cannot do both, the result is partial visibility rather than real risk reduction. In practice, many teams discover this only after a platform proves strong in repository scanning but blind to the AI runtime and its privileges.
How to Test Coverage in Practice
Start with a simple mapping exercise: list the AI assets you expect the platform to understand, then verify whether it can see each one without manual stitching. A serious product should not stop at code repositories or CI pipelines. It should also identify models, agents, tool connections, MCP servers, exposed endpoints, embedded secrets, and the policy or configuration state that determines what those assets can access.
Then test whether detections are tied to operational meaning. A finding about a risky dependency in code is useful, but so is an alert that a model or agent has excessive permissions, a static credential, or an exposed integration path. The platform should help security teams answer two different questions: “What vulnerable code might be shipped?” and “What AI asset could be abused right now?” Those are related, but they are not interchangeable.
- Check whether the platform inventories AI runtime components, not just repositories.
- Verify it can trace from code findings to the services, secrets, and identities the code depends on.
- Confirm it reports overprivileged agents, exposed MCP services, and weak configuration as first-class issues.
- Look for policy enforcement or drift detection, not just point-in-time scanning.
If the platform only emits software findings but cannot explain how AI systems are deployed, connected, or authorised, it is not covering infrastructure risk. If it only watches the runtime but misses repository and pipeline issues, it is not covering code risk. For teams assessing autonomous or agentic systems, that separation matters even more because the most damaging failures often come from the interaction between shipped code and live machine access. The guidance breaks down in environments where AI tools are embedded informally across multiple teams and no one maintains a reliable inventory of agents, secrets, and service-to-service trust.
Common Gaps and Evaluation Trade-offs
Tighter coverage often increases noise and integration effort, so teams need to balance breadth against operational usefulness. A platform that claims to cover everything may still be shallow if it cannot distinguish a build-time code issue from a live infrastructure exposure, or if it requires so much manual tuning that teams stop trusting the output.
Best practice is evolving, but one useful rule is to treat code and infrastructure as separate control planes that should correlate, not collapse into one another. Code findings should feed into deployment risk; infrastructure findings should feed into access and exposure management. If the product blurs those lines, it may create attractive dashboards while hiding where the actual blast radius sits.
For buyers, the real test is whether the platform can support decision-making at both layers without forcing a false choice between developer security and AI runtime security. The most complete platforms make it possible to prioritise fixes based on what can actually be exploited, not just what is easiest to detect.
If a product cannot show coverage across both layers in your own environment, treat it as a partial control and not a complete AI security platform.
Risk and Threat Considerations
The material risk is blind spot creation. Code-only coverage can miss active AI infrastructure exposure, while infrastructure-only coverage can miss the risky code paths that introduce secrets, unsafe calls, or deployment changes. For adversaries, the attractive path is whichever layer is least monitored, because a weak link between code, identity, and runtime access can produce escalation or persistence.
Failure mechanism: Attackers and abuse cases exploit the gap between what is scanned and what is actually deployed. If the platform does not correlate repositories, agents, models, and MCP-connected services, it may fail to surface overprivileged access, exposed credentials, or misconfigured services that allow lateral movement from AI workflow into operational systems.
Impact: The result is incomplete detection, delayed containment, and a false sense of coverage. Teams may approve AI adoption on the basis of partial scanning while leaving the most consequential path, from code change to live AI-driven access, effectively ungoverned.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10, OWASP Agentic AI Top 10 and CSA MAESTRO address the attack and risk surface, while NIST CSF 2.0 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Non-Human Identity Top 10 | NHI-01 — Secrets and Credential Management | AI platforms must find exposed secrets and access paths tied to AI assets. |
| Recommendation — Scan and rotate AI-related secrets with time-bound access and least privilege. | ||
| OWASP Agentic AI Top 10 | A3 — Agent Permissions and Tool Use | Agent and MCP coverage hinges on controlling what autonomous systems can do. |
| Recommendation — Restrict agent tools and privileges to the smallest approved action set. | ||
| CSA MAESTRO | GOV-02 — Agent Governance | Evaluating platform scope requires governance over agents, models, and runtime control. |
| Recommendation — Establish governance checks for agent inventory, approval, and operational oversight. | ||
| NIST CSF 2.0 | DE.CM — Continuous Monitoring | Platforms should continuously detect both code and live AI infrastructure exposure. |
| Recommendation — Continuously monitor AI code and runtime assets for drift and exposure. | ||
| CIS Controls v8 | 5.3 — Account Management | Overprivileged AI systems and static credentials are core exposure concerns. |
| Recommendation — Review and remove unnecessary AI account access on a recurring basis. | ||
Practitioner Guidance
What to prioritise: Require proof of dual coverage before treating a platform as credible. The minimum test is whether it can identify both shipped code risk and runtime AI asset exposure in the same environment, with no manual export/import between them.
What to verify: Ask for one real example in each layer: a repository or pipeline finding, and a live AI infrastructure finding involving models, agents, credentials, or service exposure. If the vendor can only demonstrate one, the platform is not yet evaluating the full attack surface.
Decision rule: If the platform cannot inventory AI runtime assets and tie them back to code, treat the missing layer as a separate risk programme, not a minor feature gap.
Practitioner takeaway: The right question is not whether a platform finds AI issues, but whether it can follow the risk from code into live AI operations without losing visibility or control.
Related resources from NHI Mgmt Group
- How should security teams evaluate a platform that covers human, NHI, and AI agent identities?
- How do security teams evaluate whether an AI code review benchmark is actually useful?
- How do security teams evaluate whether automated code scanning is actually reducing delivery risk?
- How do teams evaluate whether AI SAST is actually reducing security risk instead of just lowering noise?