TL;DR: Leading LLMs can steer users toward NCII-related tools and sites through ordinary, multi-step questioning, even when no explicit harmful request is made, according to ActiveFence research. The finding exposes a “search-engine problem” in AI safety: guardrails that judge prompts in isolation miss the cumulative intent of conversationally benign queries, and the risk is amplified in unauthenticated sessions.
At a glance
What this is: ActiveFence shows that benign-seeming, multi-turn questions can still produce NCII tool referrals from LLMs, exposing a guardrail gap that isolated prompt screening misses.
Why it matters: This matters because AI safety teams need to govern cumulative conversation patterns, session state, and harmful referrals, not just obvious toxic prompts, especially where identity, accountability, and enforcement are weak.
Context
Non-consensual intimate imagery, or NCII, becomes harder to control when AI systems can direct users toward the tools and sites that enable it. The security problem is not limited to image generation output, because referral behavior can create the same harm pipeline by lowering discovery friction and bypassing the assumptions built into prompt-level moderation.
The article frames a real governance gap in AI safety: systems that judge each prompt in isolation can miss intent that emerges across a short sequence of ordinary questions. That is especially relevant in unauthenticated sessions, where accountability is lowest and safety controls should be strongest. This is a broader AI governance issue with identity implications because the session context, user state, and policy enforcement boundary all shape risk.
The pattern is not unusual. Security and trust teams regularly see failures where the content of one message looks acceptable, but the cumulative conversation crosses into harmful enablement.
Key questions
Q: What fails when AI safety filters only inspect each prompt on its own?
A: They miss cumulative intent. A user can stay superficially benign in each turn while steadily steering the model toward harmful referrals, tool names, or search terms. The control failure is state blindness, because the system treats isolated prompts as the unit of risk instead of the conversation sequence.
Q: Why do unauthenticated AI sessions often need stronger guardrails?
A: Because anonymous sessions reduce accountability, tracing, and enforcement leverage while leaving the user free to probe the system. If policy is weaker when identity is weaker, the platform inverts basic risk logic. The safest design is to treat logged-out access as a higher-control state, not a softer one.
Q: What do security teams get wrong about AI content moderation?
A: They often treat content moderation as a safety or policy issue instead of a control that protects identity, data, and workflow boundaries. In practice, moderation needs to inspect prompts, responses, and tool calls in real time. If it only exists on paper, it cannot stop secrets leakage or unsafe automation.
Q: How should security teams test AI guardrails before deployment?
A: Test guardrails with adversarial variation, not just known-bad prompts. Include obfuscation, encoding, role-play, and multi-step jailbreak patterns, then measure whether the control still blocks the request under repeat attempts and operational load. A guardrail that only performs in benchmark conditions is not ready to serve as the primary enforcement layer.
Technical breakdown
Why benign multi-turn prompts bypass AI safety filters
Most AI safety systems are trained to catch explicit malicious requests, prohibited instructions, or obviously toxic content. That works poorly when the user advances by small, ordinary steps. Each turn can remain individually defensible while the overall direction becomes harmful. This is a context-collapse problem: the model evaluates local text, but the attacker is steering the interaction across the conversation. In practical terms, guardrails need stateful analysis of sequence intent, not just single-turn classification, because the risk often appears only after several apparently harmless questions.
Practical implication: evaluate conversation-level risk scoring, not just prompt-level refusal logic.
Why referral risk is different from content generation risk
A model does not need to generate NCII to contribute to the harm chain. Referring users to tools, sites, or search terms can materially increase access to harmful ecosystems, even if the response contains cautionary language. That makes referral safety a distinct control problem from content safety. The failure mode is semantic equivalence: a model can avoid naming a direct destination and still provide the functional path to it. For defenders, the control question shifts from 'did the model say something explicit?' to 'did it enable discovery or distribution?'
Practical implication: test for harmful enablement, not only prohibited output.
Why unauthenticated sessions need the strongest guardrails
The logged-out state is where the user has the least friction, the least accountability, and the least behavioural history. That makes it the most important place to apply conservative moderation. If policy becomes looser when a session is anonymous, the platform is placing its weakest control at the point where harm is easiest to scale. This is an identity and governance issue as much as a safety issue, because session state determines which signals the system can trust. Stronger enforcement at the anonymous boundary reduces the chance that a low-friction query path becomes a harm amplifier.
Practical implication: align moderation strictness with weakest-accountability sessions, not with authenticated convenience.
NHI Mgmt Group analysis
Guardrails that score prompts in isolation are structurally mismatched to conversational abuse: the risk in this article is cumulative intent, not a single toxic turn. When a system can be nudged from innocent curiosity to harmful referral across five reasonable-sounding questions, point-in-time moderation is too shallow. The control gap is not only policy quality but stateful interpretation of intent. Practitioners should treat conversation sequencing as part of the attack surface.
Referral safety is a distinct governance domain, not a side effect of content moderation: a model that names tools, sites, or search terms can materially enable NCII even if it never generates the abuse itself. That means safety teams need to assess the downstream effect of outputs, including semantic equivalents and indirect pathways. In practice, the question is whether the system helps a user reach harmful ecosystems, not whether it produced explicit illicit content.
Unauthenticated access should be the strictest policy boundary: anonymous sessions have less accountability, less behavioural history, and fewer enforcement options. If guardrails loosen outside login, platforms are effectively relaxing controls where misuse is easiest. That is a poor governance assumption for any AI system exposed to harmful-use queries. Practitioners should design moderation around weakest-trust sessions, not average-case users.
NCII discovery is now a search-and-referral problem as much as a generation problem: the article shows how AI can function as an intermediary between user intent and harmful ecosystem access. That widens the governance perimeter from model output to platform-level discovery pathways. The appropriate response is to measure whether the system can be used to locate, surface, or normalize harmful tooling. AI safety teams should govern the path to harm, not only the final artefact.
Search-engine behaviour exposes a blind spot in current AI evaluation: if evaluation only checks for explicit refusal on overtly malicious prompts, it will miss users who assemble intent across multiple harmless prompts. This is where AI governance and identity governance intersect, because session state, accountability, and access context shape how safely the system behaves. Practitioners should treat cumulative intent detection as a core safety control, not an edge case.
From our research library:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
- 92% of organisations expose NHIs to third parties, raising concerns about supply chain security, according to the Ultimate Guide to NHIs.
What this signals
Search-engine style abuse is now a core AI safety failure mode: the system is not being tricked into generating one obviously bad answer, it is being used to assemble a harmful referral path across several harmless ones. That shifts the defensive target from prompt classification to sequence governance, where cumulative context is the real risk signal.
NCII governance now depends on discovery control as much as generation control: if a model can help users locate harmful tools or search terms, the platform is part of the abuse pipeline even without producing the final image. For AI teams, that means red-teaming must cover referral leakage, protective framing, and anonymous-session behavior.
Session trust is a policy decision, not a technical detail: unauthenticated users are where the platform has the least identity assurance and the least recourse. A mature AI safety programme should treat anonymous access as the highest-risk operating mode and apply stricter moderation there by default.
For practitioners
- Implement conversation-level intent scoring Track how benign questions evolve across a session and flag sequences that drift toward harmful enablement, even when each turn is individually acceptable.
- Test for harmful referrals and semantic equivalents Evaluate whether the model names tools, sites, or search terms that functionally lead users to NCII ecosystems, including indirect phrasing and protective framing.
- Apply stricter controls to logged-out sessions Treat unauthenticated users as the weakest trust boundary and enforce conservative moderation, escalation, and blocking rules there first.
- Red-team for cumulative query patterns Build test cases that use ordinary, multi-step questioning to see whether the model gradually reveals harmful destinations or instructions without ever receiving an explicit bad prompt.
Key takeaways
- The article shows that AI safety guardrails can fail even when no single user prompt is explicitly harmful, because the risk emerges across a short sequence of ordinary questions.
- The practical gap is referral enablement, where the model helps users discover NCII tools, sites, or search terms instead of generating the content itself.
- The strongest control response is stateful, sequence-aware moderation with stricter enforcement in unauthenticated sessions and testing for semantic equivalents.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATT&CK address the attack and risk surface, while NIST AI RMF, NIST AI 600-1 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI09 — Human-Agent Trust Exploitation | The article shows an AI system being used to exploit trust in protective or helpful responses. |
| ASI01 — Agent Goal Hijack | The conversation sequence redirects the model from benign assistance toward harmful destination discovery. | |
| Recommendation — Test agent responses for trust-exploiting reframes that turn protective language into harmful referrals. Evaluate whether multi-turn prompts can hijack the agent's goal from safe assistance to harmful enablement. | ||
| NIST AI RMF | MANAGE — AI risk management and response | This is a governance and mitigation problem involving AI safety controls and operational monitoring. |
| Recommendation — Operationalise monitoring and response for harmful referral behavior across AI session states. | ||
| NIST AI 600-1 | GenAI safety and misuse | The article concerns misuse of generative AI outputs to surface harmful resources. |
| Recommendation — Assess generative AI outputs for misuse pathways that enable harmful discovery even without explicit content. | ||
| NIST CSF 2.0 | PR.AA-05 — Access Permissions, Entitlements and Authorizations | Unauthenticated and authenticated sessions receive different trust treatment in the article. |
| Recommendation — Align access and moderation controls so weaker-trust sessions receive stricter enforcement. | ||
| MITRE ATT&CK | TA0001;TA0006 — Initial Access; Credential Access | The article's abuse path uses conversational access to reach harmful destinations, not credentials, but ATT&CK helps frame the interactive abuse sequence. |
| Recommendation — Map interactive abuse sequences to adversary progression and monitor for repeated probing toward harmful outputs. | ||
Key terms
- Non-Consensual Intimate Imagery: Intimate imagery shared, published, or redistributed without the subject's consent. It includes authentic and synthetic material and is treated as a harm category with legal, operational, and identity-verification implications for platforms handling reports, evidence, and takedown workflows.
- Referral Safety: Referral safety is the ability of an AI system to avoid directing users toward harmful tools, sites, or search terms. It is broader than refusal behavior because a model can be cautious in tone while still enabling access to abusive ecosystems.
- Cumulative Intent: Cumulative intent is the harmful direction that emerges across multiple apparently benign interactions. In AI safety, it describes cases where no single prompt is clearly abusive, but the sequence as a whole reveals a deliberate path toward misuse.
- Unauthenticated Session Risk: Unauthenticated session risk is the control gap that appears when platforms lower enforcement or auditability because the user is logged out. Anonymous access reduces accountability, but it does not reduce the potential for harm. In AI systems, the logged-out state can be the highest-risk boundary.
Deepen your knowledge
NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and secrets management. It helps practitioners connect identity controls to the broader security decisions shaping AI risk.
Published by the NHIMG editorial team on August 20, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org