Open source AI security tools are reshaping agent governance

At a glance

What this is: This is an analysis of why open source security tooling is becoming central to AI agent governance, with the key finding that runtime access control matters as much as prompt safety.

Why it matters: It matters because IAM, PAM, and NHI teams now have to govern what autonomous or semi-autonomous systems can touch, prove, and revoke across tools, data, and API access.

By the numbers:

• 53% of MCP servers expose credentials through hard-coded values in configuration files.
• Only 18% of MCP server deployments implement any form of access scoping for tool permissions.

👉 Read Obot's analysis of open source AI security tools for agents and LLMs

Context

AI agent governance is the discipline of controlling what software entities can access, do, and disclose when they act on behalf of users or systems. In this article's context, the risk is not the model alone but the surrounding identity layer: tool permissions, data access, auditability, and revocation for agents and MCP-connected services.

The source article argues that open source security tools can help because they expose detection logic, runtime guardrails, and governance controls for inspection and self-hosting. That matters for NHI programmes because agentic systems increasingly behave like privileged non-human identities, even when teams still describe them as application features rather than governed actors.

The pattern is increasingly typical, not edge-case. Once AI systems can call tools, read files, and trigger actions, the control gap moves from prompt quality to lifecycle governance and least privilege at runtime.

Key questions

Q: How should security teams govern AI agents that call APIs instead of using a UI?

A: Security teams should govern AI agents by treating each callable action as a scoped entitlement, not as a general application login. The key control is to limit which APIs, data sources, and write actions the agent can chain together in one session. That keeps machine-paced behaviour inside a reviewable boundary instead of relying on human-style session assumptions.

Q: Why do AI agents create more risk than traditional automation?

A: AI agents create more risk because they can interpret context, choose actions, and invoke tools autonomously. Traditional automation follows fixed rules, but an agent can be manipulated into using its own authority in unintended ways. That makes permission scope, tool boundaries, and monitoring more important than model accuracy alone.

Q: What breaks when MCP servers are left unscoped?

A: Unscoped MCP servers let agents reach more tools and data than their task requires, which turns a small integration into a broad access path. The result is credential exposure, unauthorized data access, and poor auditability. Access scoping has to be explicit, enforced, and reviewable.

Q: How do security teams balance pre-deployment testing and runtime validation for AI systems?

A: Use pre-deployment testing to find jailbreaks, prompt injection, and leakage before launch, then keep runtime validators in place to block unsafe prompts or outputs during live use. The two controls solve different problems, and both are needed if the system can act on data or touch production tools.

Technical breakdown

Why MCP gateways matter for AI agent access control

Model Context Protocol creates a standard way for agents to connect to tools and data sources, but the protocol itself does not solve governance. The security issue is the trust boundary around the connector: who approved it, what credentials it uses, what permissions it has, and whether each call is authenticated and logged. A gateway can centralize policy enforcement so an agent does not talk directly to every backend or secret store. That shifts control from scattered tool integrations to one auditable choke point.

Practical implication: treat MCP connections as governed access paths, not simple integrations, and require authenticated, logged, revocable control in front of every tool call.

How red-teaming tools find AI security failure modes

Tools such as Garak and PyRIT are designed to probe model and agent behaviour before deployment. They test for jailbreaks, prompt injection, leakage, toxic outputs, and other unsafe responses by generating adversarial inputs at scale and measuring whether the system resists them. This is different from standard application testing because the target is not just code correctness but behavioural resilience under adversarial prompting. In practice, these tools help security teams discover where model behaviour, instruction hierarchy, or tool-use policy breaks down before production traffic does it for them.

Practical implication: add adversarial AI testing to release gates so unsafe prompt and tool-use paths are caught before they reach users.

What runtime validation adds that pre-deployment tests cannot

Guardrails at runtime solve a different problem from pre-launch testing. Even a system that passes red-team review can still receive dangerous prompts, emit sensitive data, or drift into unsafe output during normal operation. Runtime validators inspect inputs and outputs as the interaction happens, which is essential when the system can take actions or expose data after a decision is made. This is where security becomes continuous rather than episodic: policy must stay active while the model is live, not just when it is evaluated.

Practical implication: pair pre-deployment testing with runtime validation so unsafe behaviour is controlled during live sessions, not only detected in review.

Threat narrative

Attacker objective: The objective is to turn agent trust into unauthorized access, data exfiltration, or unsafe downstream actions through the tools the system is allowed to use.

1. Entry occurs when an AI system is connected to tools, files, or APIs through MCP or another agent integration without tight governance over the connector. Escalation follows when the agent can choose actions or retrieve data that exceed the original user intent, creating tool misuse and unauthorized disclosure paths. Impact is realized when the agent reaches sensitive systems or data stores with privileges that were not explicitly bounded for that session.

Breaches seen in the wild

• Meta AI Instagram Account Takeover — 20,225 Instagram accounts hijacked via compromised Meta AI support chatbot with overprivileged access.
• Replit AI Tool Database Deletion — Replit vibe coding AI assistant deletes live production database and creates 4,000 fake user records.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Open source is now a governance requirement for AI agent security, not just a licensing choice. The article is right to frame transparency, self-hosting, and extensibility as core security features. In agentic systems, the question is not whether a tool is elegant but whether security teams can inspect what it does, prove what it logged, and revoke what it touched. That is a control-plane requirement, not a preference.

AI agents are behaving like non-human identities, and that changes the control problem. Once an agent can call tools, read files, and hit APIs, the relevant security question becomes what identity it uses, what scope it has, and how that scope is governed. That aligns agent security with NHI governance, not traditional application security alone, because the access path is identity-driven and session-driven.

Model Context Protocol introduces an identity blast radius unless the gateway is governed. The central risk is not MCP as a protocol but unscoped connectors, unmanaged credentials, and tool permissions that outlive the task. In NHI terms, this is standing privilege in a new form: the agent inherits access to more systems than its runtime intent justifies.

Defense in depth is the correct operating model for AI security because no single control covers the full failure chain. Pre-deployment scanning finds prompt and jailbreak issues, runtime validation limits unsafe outputs, and access governance controls what the agent can touch. The article’s strongest point is that these layers are complementary, and organisations that treat any one of them as sufficient will leave exploitable gaps.

From our research:

• 98% of companies plan to deploy even more AI agents within the next 12 months, despite documented rogue behaviour in 80% of current deployments, according to AI Agents: The New Attack Surface report.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• OWASP NHI Top 10 helps teams turn these findings into a practical control map for agentic risk.

What this signals

Agent governance will converge with NHI lifecycle management. As agents proliferate, teams will need the same operational disciplines they use for service accounts: ownership, scoping, logging, review, and revocation. The difference is that agent behaviour is less deterministic, so access review alone is not enough unless it is paired with runtime enforcement and connector governance.

Model Context Protocol can become an identity blast radius if connector sprawl is left unmanaged. The practical signal for practitioners is to look at every tool integration as a permission boundary, not an engineering convenience. For broader design context, the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework both reinforce that agent behaviour must be governed across the full runtime lifecycle.

Continuous testing will become part of the AI security baseline. Security teams that only test agents before release will miss behavioural drift, prompt injection variants, and tool misuse that appear later. The programme implication is clear: combine red-teaming, runtime validation, and connector-level governance, then measure whether those controls actually reduce unauthorized access events over time.

For practitioners

• Map AI agents to governed identity classes Inventory every agent, connector, and tool path as a non-human identity with explicit ownership, scope, and revocation requirements. Do not leave MCP servers or agent wrappers in the application inventory without access lifecycle controls.
• Put a gateway in front of all high-risk tool calls Require authenticated and authorized mediation for every request that reaches sensitive tools or data sources, and log the policy decision, not just the action. Use the gateway as the enforcement point for least privilege and auditability.
• Run adversarial tests before release and on every change Use red-teaming and evaluation tooling to probe for prompt injection, jailbreaks, leakage, and unsafe tool use before changes ship. Make those tests part of the same release workflow that governs code, model updates, and connector changes.
• Keep runtime validators active after deployment Inspect prompts, outputs, and tool-bound actions while the system is live so unsafe content can be blocked, redacted, or corrected in-session. Runtime control is what limits damage when pre-deployment review misses a new attack pattern.

Key takeaways

• AI agent security is becoming an identity problem as much as a model-safety problem.
• Open source tools matter because they make agent controls inspectable, self-hostable, and easier to integrate into existing governance.
• Practitioners should pair pre-deployment testing with runtime access control if they want meaningful risk reduction.

Key terms

• Model Context Protocol: Model Context Protocol is an open protocol that lets AI agents connect to tools and data sources. It expands what an agent can reach, so governance has to cover not only the model and its prompts, but also every system that can receive or return agent-driven data.
• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions — including calling APIs, writing code, and orchestrating other agents — with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.
• Runtime validation: A control practice that tests how an AI system behaves while it is connected to real tools and data, rather than only reviewing configuration or design documents. It matters because agentic systems can appear safe on paper and still fail when prompted, chained, or given access to connected services.
• Tool Governance: Tool governance is the control of the APIs, service accounts, connectors, and permissions an agent uses to reach other systems. It focuses on the delegated paths that convert agent intent into action, because those paths often hold the real security risk and the broadest privilege exposure.

What's in the full article

Obot's full article covers the operational detail this post intentionally leaves for the source:

• Side-by-side feature and deployment discussion for Garak, Guardrails AI, PyRIT, Promptfoo, and Obot across different AI security use cases.
• Implementation-oriented comparison of where each tool fits in pre-deployment testing, runtime validation, red teaming, and agent governance.
• Practical deployment notes for self-hosting, CI/CD integration, and workflow fit that matter once a team moves from evaluation to rollout.
• Context on how these tools can be combined into a layered AI security stack rather than used as isolated point solutions.

👉 Obot's full article covers the tool-by-tool comparison and deployment fit in more operational detail.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.