Because one filter cannot reliably balance speed, context, and nuance in a live interaction. Companions need stacked controls that screen obvious harm quickly, evaluate ambiguous behaviour more deeply, and provide safe exit paths that preserve the experience while preventing escalation.
Why This Matters for Security Teams
A single moderation filter is too brittle for AI companions because the system is not just answering a prompt, it is sustaining a relationship-like interaction over time. That creates a wider attack surface: prompt injection, grooming attempts, self-harm escalation, sexual content, coercive manipulation, and the gradual erosion of policy boundaries. Security teams should treat companion moderation as a layered control problem, not a single decision point. NIST guidance in NIST SP 800-53 Rev 5 Security and Privacy Controls is useful here because it reinforces that resilience comes from multiple safeguards working together rather than one control absorbing all responsibility.
The practical issue is that companion systems must balance user experience with safety in real time. A filter tuned too tightly will block benign emotional support, while a filter tuned too loosely will miss escalation patterns that unfold across many turns. This is where governance matters: policy, conversation memory, risk scoring, human review, and safe completion logic each cover a different failure mode. In practice, many security teams encounter the real risk only after users have already learned how to steer the companion around a single moderation gate, rather than through intentional control design.
How It Works in Practice
Effective moderation for AI companions usually works as a stack. The first layer catches obvious policy violations quickly, while deeper layers assess context, user history, and intent. That means the system may allow a neutral response, redirect a harmful request, or end the interaction safely depending on the severity and confidence level. Current guidance suggests this is closer to security engineering than content filtering: the goal is not perfect classification, but reducing harm while maintaining a usable product.
A practical design often includes:
- Pre-generation checks for high-risk prompts, including self-harm, abuse, sexual exploitation, and instruction-seeking for harmful acts.
- Post-generation validation to catch unsafe output that was not obvious at prompt time.
- Conversation-state review so repeated boundary testing is treated differently from a one-off ambiguous message.
- Escalation logic that routes high-risk cases to safer responses, crisis resources, or human moderation.
- Logging and audit trails to support model tuning, incident review, and policy enforcement.
This layered approach also helps with agentic behaviours, where an AI companion may have tool access, memory, or the ability to take actions on behalf of a user. In those cases, moderation must extend beyond text screening to include tool-use constraints, output validation, and permission boundaries. The NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 both reinforce the need to manage downstream impact, not just generated text. These controls tend to break down when the companion has long memory, plugin access, or unsupervised outbound messaging because a single filter cannot inspect every transition in the interaction chain.
Common Variations and Edge Cases
Tighter moderation often increases friction, requiring organisations to balance safety gains against user trust and conversation quality. There is no universal standard for this yet, so teams should expect different thresholds for general companionship, youth-facing products, mental health-adjacent features, and enterprise assistants. The right control set depends on the harm model, not on one generic policy layer.
Some edge cases are especially difficult. Sarcasm, coded language, and emotionally loaded but non-explicit abuse often evade simple classifiers. Cross-turn manipulation is another gap: a user may start with innocuous conversation and gradually steer the system into unsafe territory. Companion systems also need strong safe-exit behaviour, because a refusal alone can escalate frustration if it is delivered without alternatives.
For organisations operating in regulated contexts or with higher-risk audiences, the OWASP LLM Top 10 is a useful companion reference for prompt injection, data leakage, and output manipulation, while MITRE ATLAS helps frame adversarial tactics against AI systems. Best practice is evolving, but the direction is clear: moderation should be layered, state-aware, and designed for graceful containment rather than binary approval.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF, NIST CSF 2.0 and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI RMF addresses layered governance and risk treatment for companion systems. | |
| OWASP Agentic AI Top 10 | Agentic systems need controls beyond text filtering when tools or actions are involved. | |
| MITRE ATLAS | Adversarial tactics help explain how users evade a single moderation layer. | |
| NIST CSF 2.0 | PR.DS, PR.AC, DE.CM | Moderation stacks depend on access, data, and monitoring controls working together. |
| NIST AI 600-1 | GenAI guidance is relevant where companion outputs must be validated and bounded. |
Use AI RMF functions to define, measure, manage, and govern moderation risk across the product lifecycle.
Related resources from NHI Mgmt Group
- Why is single-provider AI agent governance not enough for enterprise security?
- Why do single-signal controls fail for agentic AI security?
- Why do multi agent systems create more identity risk than single AI assistants?
- Why do multi-hop AI agent workflows create more risk than single-agent automation?
