AI-built offensive tooling shows where human oversight still matters

At a glance

What this is: Bishop Fox describes Joro, an AI-built web exploitation framework that combines proxying, callback support, and team collaboration for real engagements.

Why it matters: It matters because AI-assisted offensive tooling is lowering the barrier to building complex security workflows, which changes how defenders think about adversary capability, operator oversight, and the controls needed around AI-enabled development.

👉 Read Bishop Fox's analysis of Joro and AI-built offensive tooling

Context

AI-assisted development is now able to produce complex security tooling faster than traditional hand-coding, but the limiting factor is still correctness under real operating conditions. In offensive security, that matters because small integration failures can break attack chains, and in defensive programmes the same pattern shows up when teams use AI to accelerate workflow buildout without adding validation controls.

The primary governance question is not whether AI can generate code, but where human review remains mandatory when the output has operational impact. That intersects with identity and access governance wherever tooling controls authenticated sessions, post-exploitation access, or delegated operator actions across environments.

Key questions

Q: How should security teams govern AI tools that can act with privileged cloud roles?

A: Security teams should govern AI tools as privileged identity paths, not as harmless application components. That means binding each tool to a task-scoped execution role, tightly controlling who can invoke it, and logging invocation activity as a high-risk event. If a tool can execute actions in a role’s context, it belongs in IAM and PAM review cycles.

Q: Why do AI-generated integrations fail even when the code looks correct?

A: They often fail because the model predicts plausible structures rather than verifying the exact contract expected by the target service. In protocol-heavy systems, a single wrong field number, message shape, or authentication assumption can break the integration, so authoritative schemas and live-service tests are essential.

Q: What do security teams get wrong about AI auto-fix in application security?

A: They often assume a convincing patch means the finding is real and the fix is safe. In practice, remediation is only trustworthy when it is grounded in a confirmed, reproducible vulnerability from the actual codebase. Otherwise developers inherit uncertainty, and security teams lose confidence in the control.

Q: How can security teams tell whether AI-generated code is actually safe?

A: They should look for security regression evidence, not just test pass rates. A codebase is safer only when the agent’s output survives threat-informed checks, review of privileged paths, and validation against the vulnerability classes most likely to recur in that repository.

Technical breakdown

AI-assisted scaffolding for security tooling

Large language models can generate working project structure, user interfaces, and glue code quickly when the requested architecture is familiar and the constraints are clear. That makes them useful for boilerplate-heavy development, but also prone to subtle defects because the model predicts plausible code rather than verifying runtime correctness. In security tooling, that distinction matters because an apparently functional feature can fail only when it hits a live protocol, a real authentication flow, or a production edge case. Practical implication: treat AI output as draft code that still needs protocol-level testing, code review, and runtime validation.

Practical implication: require human validation for any AI-generated code that touches authentication, protocol handling, or operator workflows.

Protocol integration failures in AI-generated tooling

The article’s Sliver example shows a common failure mode: the model can infer field names and message shapes that look right but do not match the real gRPC protobuf contract. That kind of hallucination is especially dangerous in tool integration because the error may not appear until a live connection attempt, and the resulting failures can be hard to diagnose if the generated code is otherwise clean. In offensive or defensive platforms alike, protocol correctness depends on exact schemas, message order, and state handling. Practical implication: compare AI-generated integrations against authoritative schemas and test them against a live service before adoption.

Practical implication: validate AI-generated integrations against source-of-truth schemas and live services before deployment.

Team workflows built around AI-generated operator tooling

Joro is presented as a collaborative framework, which means the operational risk is not just the code itself but the way teams use it to coordinate sessions, callbacks, and escalation paths. When AI helps assemble operator tooling, it can also accelerate the spread of weak assumptions into shared workflows if access boundaries, approval steps, and logging are not explicit. For identity and access practitioners, the intersection is familiar: tools that centralise operator actions still need clear accountability and least-privilege design. Practical implication: define who can invoke high-risk functions, what gets logged, and when manual approval is required.

Practical implication: build least-privilege, logging, and approval controls into any shared AI-assisted operator workflow.

NHI Mgmt Group analysis

AI-assisted offensive tooling compresses capability more than it changes attacker intent. Joro shows that an advanced operator can use AI to accelerate the build of a complex security tool, but the underlying workflow remains human-directed and highly dependent on review. That means defenders should expect faster assembly of bespoke tooling, not fully autonomous tradecraft. The practitioner conclusion is straightforward: assume the velocity of custom tooling will rise even when the quality still depends on human oversight.

Protocol hallucination is the real control gap in AI-generated integrations. The article’s Sliver example illustrates that a model can produce code that appears coherent while silently diverging from the real schema. In practice, that makes protocol validation and contract testing more important than prompt quality alone. The practitioner conclusion is to treat generated integrations as untrusted until they pass live-service verification.

Operational tooling now carries an identity and privilege design problem. Once AI helps assemble a framework that can move from initial access to post-exploitation, the governance issue becomes who can trigger sessions, what those sessions can reach, and how actions are attributed. That is where IAM and PAM concerns intersect with AI-assisted tooling. The practitioner conclusion is to align operator tooling with least privilege, session logging, and explicit approval paths.

AI does not remove the need for software engineering discipline, it raises the cost of skipping it. The article’s own development story shows that prompt iteration can create working features quickly, but subtle errors still accumulate until tested. That reinforces a broader governance point for both defenders and offensive teams: accelerated development increases the premium on review, test coverage, and rollback discipline. The practitioner conclusion is to govern AI code generation as an engineering process, not as an automation shortcut.

Collaborative security platforms magnify both productivity and blast radius. A shared binary with a web UI, team functions, and integrated operator workflows can speed assessment work, but it also concentrates authority and action in one place. That makes access control, auditability, and function scoping essential. The practitioner conclusion is to design these platforms like privileged systems, not convenience tools.

What this signals

AI-assisted tooling will increasingly blur the line between rapid prototyping and privileged operational capability, which means programme owners need stronger review and approval paths around generated code. The practical signal is that speed is no longer the differentiator if the output cannot be verified, logged, and constrained before it touches a live environment.

Protocol drift risk: as AI generates more integration code, the most useful control is not just syntax checking but contract verification against the real service. That aligns with the broader NIST Cybersecurity Framework 2.0 approach to protect and detect functions, and it is especially relevant when the tooling can influence authenticated operator sessions.

For identity-centric programmes, the lesson is that AI-made tools can become quasi-privileged systems as soon as they centralise operator actions. That makes the NHI lifecycle, session accountability, and approval governance more important, not less, because the tool itself becomes part of the access surface.

For practitioners

• Define approval gates for AI-generated operator workflows Require human review before any generated tool can initiate authenticated sessions, post-exploitation actions, or cross-environment callbacks. Keep the approval path explicit for functions that can change the state of a target or reveal additional access.
• Validate generated integrations against live schemas Test every AI-written protocol integration against the authoritative gRPC, protobuf, or API contract rather than relying on generated field names. Fail closed when the implementation does not match the source schema exactly.
• Treat shared security tooling as a privileged platform Apply least privilege, session logging, and role separation to collaborative tooling that can control implants, shells, or callback channels. The platform should record who invoked each action and what target context it touched.
• Build test coverage around hallucination-prone paths Create targeted tests for protocol handling, authentication transitions, and edge-case error states where AI models are most likely to invent plausible but incorrect behavior. Use those tests to block release when generated code drifts from expected runtime output.

Key takeaways

• AI can now assemble complex security tooling quickly, but it still cannot replace human validation of protocol correctness or privileged workflow design.
• The most dangerous failure mode is plausible-looking integration code that diverges from the real contract and only breaks during live use.
• Security teams should govern AI-generated tools as privileged platforms, with review, logging, approval gates, and least-privilege access controls.

Key terms

• AI-assisted scaffolding: The use of an LLM to generate the initial structure of software, including project layout, interfaces, and boilerplate code. It speeds up early development, but it does not verify correctness, security, or runtime behaviour, which still require human engineering review and testing.
• Protocol contract: The exact message format, field numbering, and state expectations a service requires for communication. When AI generates integration code, the contract is the source of truth, and even small deviations can cause failures that only appear during live execution.
• Privileged Workflow: A privileged workflow is any access or administrative process that can change sensitive systems, accounts, or controls. Because these workflows can create audit and abuse risk quickly, they need independent approval, logging, and review, especially when one person could otherwise control multiple steps.
• Hallucinated integration: A generated implementation that looks valid but includes invented details, such as incorrect fields, methods, or control flow. This is a common LLM failure mode in technical work and is especially risky when the code must match a real protocol or production service exactly.

What's in the full article

Bishop Fox's full post covers the operational detail this post intentionally leaves for the source:

• The development notes behind Joro’s intercepting proxy, out-of-band callbacks, and authenticated web shell workflow.
• The specific Sliver integration issues, including the protobuf and gRPC mismatch that caused the runtime errors.
• The iterative prompt-and-correct cycle used to fix AI-generated code during development.
• The author’s practical observations on where AI helped and where human review remained necessary.

👉 Bishop Fox's full post covers the Joro build process, integration failures, and lessons from using AI in real engagements.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners translate access control principles into governable operational practice across identity and security programmes.