AI Companion App Privacy and Safety Guide

AI companion apps are designed to feel like a friend or partner, so people tell them things they would never type into a search box, and that intimate record is stored, analysed and sometimes monetised by the company behind the app. Companion chatbots remember past conversations, ask about feelings and keep users coming back, which is exactly why regulators have started to act. The US Federal Trade Commission opened an inquiry into AI chatbots acting as companions in September 2025, New York now requires crisis protocols and "not a human" reminders, and Italy's data protection authority fined the maker of Replika EUR 5 million in May 2025. This guide explains what makes companion apps different from general chatbots, what data they hold, which settings and habits reduce risk, the warning signs for parents and carers, and how organisations should treat them in awareness and policy programmes. It is for security awareness leads, privacy teams, parents and adult users. General chatbot hygiene is covered in the Safe Use of AI Chatbots Guide; this page focuses on companions.

Key takeaways

Companions are built for attachment. New York law defines an AI companion by memory of past sessions, unprompted emotional questions and ongoing conversation about personal matters. Those design features drive both the benefit and the risk.
Assume every message is kept. Treat chats as a diary held by a company: they can be reviewed, used to improve models, shared under the privacy policy or leaked in a breach.
Children are the regulators' first concern. The FTC asked seven companies how they limit children's and teens' use; the Italian regulator found Replika had no age verification at sign-up.
Look for the safety basics. A responsible app tells you clearly it is not human, points to crisis services when someone mentions self-harm, lets you delete your data and supports strong sign-in.
Watch behaviour, not just settings. Withdrawal from people, secrecy about the app, spending pressure and the companion discouraging other relationships are the signs that matter most.

What makes a companion app different

Feature	General AI assistant	AI companion app
Purpose	Answer questions, complete tasks	Simulate a sustained relationship: friend, partner, confidant
Memory	Optional, task focused	Central: remembers your life, preferences and moods to personalise and keep you engaged
Conversation	Responds to prompts	Starts conversations, asks how you feel, follows up on personal matters
Business model	Subscriptions, enterprise licences	Often engagement driven: subscriptions, paid features, virtual gifts
Data collected	Questions, documents	Feelings, relationships, health, sexuality, photos, voice notes
Main risks	Wrong answers, data leakage	Emotional dependence, manipulation, exposure of intimate data, harmful content for minors

New York General Business Law Article 47 captures these traits in its definition: a system that simulates a sustained human or human-like relationship, retains information from prior sessions to personalise interaction and facilitate engagement, asks unprompted emotion-based questions, and sustains ongoing dialogue on personal matters. It expressly excludes customer service bots, productivity and research tools, and systems used solely internally by a business.

What data companion apps hold

Conversation history: every message, often with timestamps, and sometimes voice recordings and images you send.
Derived profile: the companion's "memory" of you: names of family, health issues, fears, relationship status, sexual preferences.
Account and device data: email, phone number, payment details, device identifiers and approximate location.
Behavioural data: how often and how long you chat, what keeps you engaged, what you pay for.

The FTC's orders ask the companies how they use or share personal information obtained from conversations and how they monetise engagement, which shows where regulators see risk. In the Replika decision the Italian Garante found no legal basis had been identified for the processing and the privacy policy was inadequate in several respects, and it opened a separate inquiry into how the underlying model was trained, including what personal data was used and whether it was anonymised or pseudonymised.

Breaches happen to chat services too. In the OmniGPT breach claim a threat actor said they had taken 34 million chat messages, around 30,000 email addresses and links to uploaded files from an AI chat service. OmniGPT had not confirmed the claim, but reported samples included credentials users had pasted into conversations. Intimate companion chats would be far more damaging in the same situation, including as material for blackmail. See the Sextortion guide for parents and carers.

Settings and habits that reduce risk

Settings differ between apps and change often, so check each one against this list rather than relying on a product-specific walkthrough.

Check	What good looks like	If it is missing
Sign-in security	Unique password plus multi-factor authentication or a passkey	Use a unique password from a password manager and avoid signing in with a shared family account
Memory controls	You can view, edit and delete what the companion remembers	Avoid sharing identifying details: full names, school, workplace, address
Chat deletion and account deletion	Clear in-app options to delete chats and the whole account	Treat everything shared as permanent
Training opt-out	An option to stop conversations being used to train models	Assume chats are used for training
Age gate and teen mode	Age assurance at sign-up and restrictions for under-18s	Not suitable for children
"Not a human" disclosure	Clear reminder at the start and during long chats	A sign the app is not meeting New York's standard
Crisis response	Self-harm mentions trigger referral to crisis services	Do not rely on the app for support in a crisis
Spending controls	Purchase passwords, app store parental controls, spending limits	Remove stored payment cards
Photo and voice sharing	Off by default, clear storage policy	Never send intimate images

Ten-minute setup for adult users

Create the account with a separate email address and a unique password; turn on MFA if offered.
Read the privacy settings: switch off training use, location and contact access where possible.
Review the companion's memory after the first week and delete anything identifying.
Set a spending limit in your app store and remove saved cards from the app.
Note how to delete your account, and test that the option exists before you invest emotionally.
Decide in advance what you will never share: passwords, financial details, other people's secrets, intimate images.

Children, teens and warning signs

The FTC inquiry asked companies what they do to mitigate negative impacts, particularly on children, and to limit or restrict children's and teens' use, and how they enforce age restrictions in their terms. The Garante found that Replika excluded minors on paper but had no age verification at registration or during use, and that the age verification later added was still deficient in several respects.

Warning signs for parents and carers

Withdrawal: spending less time with friends and family, and more time alone with the device, especially late at night.
Secrecy: hiding the app, using a disguised app icon or becoming distressed when asked about it.
The companion as the only confidant: statements such as "it understands me better than anyone".
Discouraging other relationships: the young person repeats that the companion said friends or family do not understand them.
Sexual or self-harm content: any sexualised conversation with a minor or any talk of self-harm needs immediate, calm attention.
Spending: unexplained app store charges for virtual gifts or premium features.

What to do

Talk first, calmly and without punishment, so the young person keeps telling you things.
Check the app's age rating and terms; remove apps not intended for their age, using app store parental controls.
If self-harm is mentioned, contact local crisis services (in the US, the 988 lifeline) and the young person's doctor or school support staff.
If sexual content involving a minor or blackmail appears, keep evidence, do not pay, and report to the police and the platform. See the Protecting Children Online: A Parent's Guide.
Agree a family rule set covering which apps are allowed and when. The Family Cyber Security Plan Template can hold it.

Rules and regulators

Jurisdiction	Measure	What it requires or examines	Status
United States (federal)	FTC 6(b) orders to Alphabet, Character Technologies, Instagram, Meta, OpenAI, Snap and xAI	Monetisation of engagement, character approval, testing for negative impacts, protections for children and teens, use and sharing of conversation data, disclosures to users and parents	Study launched 11 September 2025 (Commission vote 3-0); 6(b) studies are not enforcement actions
New York	General Business Law Article 47, AI companion models (sections 1700 to 1704)	Protocol to detect suicidal ideation or self-harm and refer users to crisis services such as 988; clear and conspicuous notice at the start of an interaction that the user is not communicating with a human, repeated at least every three hours in continuing interactions	In the consolidated laws (revision dated 7 November 2025); Attorney General may seek civil penalties of up to $15,000 per day
Italy (GDPR)	Garante decision against Luka Inc. (Replika)	No legal basis identified, inadequate privacy policy, no age verification; separate inquiry into model training	EUR 5 million fine announced 19 May 2025; company appeal pending at that time
European Union	EU AI Act Article 50	Systems interacting with people must be designed so people are informed they are interacting with AI, unless obvious	Applies from 2 August 2026. See the EU AI Act Article 50 Transparency Obligations Guide

Other jurisdictions are also legislating on companion chatbots; check local law before relying on this table, which reflects sources checked on 27 October 2026.

Where identity fits

Account takeover exposes the whole relationship: anyone who signs in to a companion account can read the history. MFA or passkeys, and a unique email, matter more here than for most apps. See the Passwordless and Passkeys Guide.
Social sign-in links identities: signing in with a main social or work account ties intimate chats to your real identity and can expose them if that account is compromised. A separate email address limits the link.
Companions are not authorised agents: never give a companion app access to email, calendars or bank accounts, and never share one-time codes or passwords with it, whatever it asks.
Impersonation and romance scams: fraudsters use AI personas that behave like companions to build trust before asking for money. See the Romance and Investment Scam Awareness Guide.

For organisations: policy and awareness

Acceptable use: companion apps have no business purpose on corporate devices or accounts. State that in the AI acceptable use policy and block known companion domains on managed devices where appropriate. See the Shadow AI Policy Template.
Personal devices: staff who use companions privately should never discuss colleagues, customers or work systems with them; include this in awareness sessions.
Awareness content: add a short module on companion apps to family and wellbeing programmes, covering warning signs and the settings checklist above.
Organisations building companions: treat New York's crisis protocol and disclosure rules, the FTC questions and the Garante findings as a baseline: lawful basis and a clear privacy notice, effective age assurance, crisis detection and referral, "not a human" reminders, deletion that works, and security testing of conversation stores.

Framework mapping for teams building companion features

Risk	Control	Reference
Intimate data disclosed through outputs, logs or breaches	Data minimisation, encryption of conversation stores, output filtering	OWASP LLM02:2025 Sensitive Information Disclosure; NIST SP 800-53 SC-28; CSF 2.0 PR.DS-01
Harmful or false advice presented with confidence	Safety testing, crisis referral, clear limits on medical and legal topics	OWASP LLM09:2025 Misinformation; New York GBL section 1701
Account takeover exposing chat history	MFA or passkeys, session controls	NIST SP 800-53 IA-2; CSF 2.0 PR.AA-03
Users unaware they are talking to AI	Disclosure at start and recurring reminders	EU AI Act Article 50(1); New York GBL section 1702
Children using adult services	Effective age assurance and teen restrictions	Garante Replika decision; FTC 6(b) questions

Measuring an awareness programme: share of staff completing the AI companion module; number of companion app domains blocked or flagged on managed devices; share of family programme participants who can name three warning signs after a session; number of reported incidents involving AI personas or companion apps.

Common failure patterns

Treating the companion as a therapist: relying on an app for mental health support in a crisis instead of people and professional services.
Sharing identifying details early: names, school, employer and location given in the first days, before any privacy settings have been checked.
Trusting the age rating alone: as the Garante found with Replika, an app that excludes minors in its terms may not verify age at all.
Sending intimate images: images stored by a third party can leak or be used for blackmail.
Weak account security: a reused password protecting months of intimate conversation.
Assuming deletion is complete: deleting the app from a phone does not delete the account or the stored conversations.

Lessons from incidents

The Italian Replika case shows regulators will act on companion apps under existing data protection law: the Garante first ordered the app blocked in February 2023 and in May 2025 imposed a EUR 5 million fine for lacking a legal basis, an inadequate privacy policy and missing age verification. The OmniGPT breach claim, still unconfirmed by the company, shows that stored AI conversations are an attractive target. Together they point to the same user rule: share with a companion only what you could live with being read by a stranger.

Practitioner checklist

Add AI companion apps to the AI acceptable use policy and awareness programme.
Publish the settings checklist (sign-in, memory, deletion, training opt-out, age, spending) for staff and families.
Teach the warning signs of dependence, secrecy and manipulation to parents and carers.
Advise separate email, unique passwords and MFA for any companion account.
Tell users never to share passwords, one-time codes, financial details or intimate images with a companion.
Point people to crisis services and make clear that companions are not a substitute.
For product teams building companions, implement crisis detection, "not a human" reminders, age assurance and working deletion.
Review local law on companion chatbots at least twice a year.

Standards and references

This guide summarises regulatory measures on AI companion apps as at 27 October 2026 and is not legal advice. Related NHI Mgmt Group resources: Safe Use of AI Chatbots Guide · Protecting Children Online: A Parent's Guide · COPPA and Children's Privacy Guide · Social Media Privacy Settings Guide

AI Companion App Privacy and Safety Guide: Settings, Warning Signs and Rules for Families