Trust and Safety Platforms Buyer's Guide

Platforms buying trust and safety tooling often end up with a classifier that scores content well in a demo but cannot produce the statements of reasons, appeal records, transparency figures and evidence packages that regulators and law enforcement now expect. The market covers content moderation queues and case management, automated classifiers for text, images, video and audio, hash matching against known illegal content, fraud and abuse signals, user reporting and appeals, and transparency reporting. This vendor-neutral buyer's guide explains how to define requirements against the EU Digital Services Act (DSA) and UK Online Safety Act 2023 duties, what capabilities to require, what to ask vendors, the red flags, and how to run a proof of concept. It is for trust and safety leads, platform engineers, security architects and procurement teams. How to design the programme itself is covered in the Trust and Safety Programme Guide, and day-to-day running in Trust and Safety Operations.

Key takeaways

Buy for the decision record, not just detection. Every action needs a stored ground (illegal content or terms breach), category, whether detection and decision were automated, and the notice sent to the user. Under the DSA, those fields feed both statements of reasons and transparency reports.
Hash matching is table stakes for image services. Ofcom's illegal content codes recommend CSAM hash matching and URL detection for defined higher-risk and larger user-to-user services, and draft amendments laid before Parliament in 2026 would add hash matching for intimate image abuse (not shown as in force when checked in November 2026).
Measure classifier accuracy on your own content. The DSA transparency templates ask about the accuracy of automated moderation, so you need precision, recall and overturn rates by category and language, not a vendor headline figure.
The moderation console is a privileged system. It can remove accounts, read private messages and export user data. Treat its admin roles, API keys and integrations as high-value identities.
Plan for evidence and reporting duties. US providers must report apparent child sexual exploitation to NCMEC under 18 U.S.C. 2258A, and the platform must preserve reported content and support that workflow.

Define requirements

Start from your risk assessment and regulatory profile, then list the decisions the platform must support. Requirements differ sharply by service type and size.

Driver	What it requires of the tooling	Source
DSA notice and action and statements of reasons	Intake of notices from users and trusted flaggers, priority queues, a statement of reasons for each restriction recording the ground under Article 17(3)(d) or (e), category, and use of automated detection and decision	DSA; DSA Transparency Database documentation
DSA transparency reporting	Counts of removals, account terminations, accuracy of automated moderation and moderation team data in the harmonised templates; annual for intermediary services, twice yearly for very large platforms	Implementing Regulation (EU) 2024/2835, applying since 1 July 2025
UK Online Safety Act illegal content codes	A content moderation function, swift takedown, internal content policies, performance targets, prioritisation, trained moderators, complaints and appeals handling	Ofcom codes ICU C1 to C8 and D1 to D14
UK automated detection measures	CSAM hash matching (ICU C9) and CSAM URL detection (ICU C10) for in-scope services; the large service threshold is over 7 million monthly UK users, and some measures also apply above 700,000 at high risk	Ofcom summary of decisions
US child safety reporting	CyberTipline reporting workflow, preservation of reported content for one year after submission, secure storage	18 U.S.C. 2258A
Fraud and scams on platform	Signals on accounts, payments and adverts, links to fraud tooling, a trusted flagger route for fraud reports (Ofcom ICU D14 for some large services)	Ofcom codes

Small services with low risk can often meet their duties with a hosted moderation queue, a reporting form, a hash matching service and a well-run appeals process. Large or multi-risk services need a full platform: multi-modal classifiers, rules and workflow engines, case management for investigations, regulatory reporting and integration with fraud and identity systems. Law status changes often in this area: confirm current duties with counsel. See the Digital Services Act Security Obligations Guide.

Capability areas

Capability	What to require	Notes
Intake and reporting	User reports, trusted flagger and authority channels, API intake, deduplication, priority rules	Trusted flagger notices get priority under the DSA
Hash matching	Perceptual image and video hashing (for example PDQ and similar algorithms), cryptographic hashes for exact matches, ingestion of industry hash lists	Programmes include the NCMEC hash sharing programme, the GIFCT hash-sharing database and StopNCII
Classifiers	Text, image, video, audio and live stream models per harm category and language, with thresholds you control	Ask for per-category precision and recall on your data
Rules and workflow	Policy-as-configuration, routing by severity, queue SLAs, escalation, sampling for quality assurance	Supports Ofcom prioritisation and performance target measures
Reviewer tooling	Blurring and greyscale by default, exposure limits, wellbeing features, keyboard workflows, context panels	Moderator welfare is an operational requirement, not a nice-to-have
Enforcement and notices	Graduated actions, statements of reasons generated from structured fields, suspension after warning	DSA Article 17 content; transparency database submission for online platforms
Appeals and complaints	Internal complaint handling, reversal tracking, links to out-of-court dispute bodies where applicable	Overturn rate is a key accuracy signal
Investigations and evidence	Case files, linked accounts, legal hold, evidence export with integrity hashes, law enforcement request handling	Must support CyberTipline reporting and preservation where relevant
Reporting and analytics	Transparency report fields mapped to the DSA templates, Ofcom information requests, internal dashboards	Test with last quarter's real numbers

Meta's open source Hasher-Matcher-Actioner describes the core copy detection pattern well: content is hashed, matched against banks of known content, and an action such as logging or queuing for human review follows. Use it as a reference model when comparing products.

Evaluation criteria

Criterion	Weight (suggested)	What good looks like	How to test
Regulatory fit	High	Statement of reasons fields, transparency template mapping, appeal records, preservation and reporting workflows built in	Generate one statement of reasons and one quarter's transparency figures in the POC
Detection quality	High	Per-category, per-language precision and recall on your labelled set; thresholds tunable	Blind test on a held-out labelled sample
Hash ecosystem	High for image and video services	Supports the hash formats and lists you are eligible for; matching distance configurable	Match against test hashes and altered copies (resize, crop, re-encode)
Workflow and reviewer experience	High	Configurable queues, SLAs, QA sampling, wellbeing controls	Have your moderators run a week of real volume
Security of the platform	High	SSO with MFA, granular roles, approval for bulk actions and data export, full audit log, scoped and rotated API keys	Attempt a bulk action and an export as a junior role
Data protection	Medium to high	Residency options, retention by content type, restricted access to illegal material, no training on your data without agreement	Review DPA, sub-processors and retention settings
Integration	Medium	Event APIs, webhooks, links to identity, fraud and SIEM, bulk import of history	Wire one fraud signal and one SIEM alert
Scale and cost	Context dependent	Latency for pre-publication checks, predictable pricing per item or per call	Model peak volume including live content

Use the Buyer's Guide Methodology and Scoring Template to weight and score.

Questions to ask vendors

How does your data model record the legal ground, category, automated detection and automated decision for each action, and can you export statements of reasons in the DSA Transparency Database schema?
Which DSA transparency template fields can you produce directly, and which need our own data?
Which hash algorithms and industry hash lists do you support, and how do you handle the eligibility and terms each list imposes?
What precision and recall do your classifiers achieve per category and language, on what test set, and how will you measure it on our content?
Can we set our own thresholds and policies per market, and is every change versioned and logged?
How do you support NCMEC CyberTipline reporting and the one-year preservation of reported content under 18 U.S.C. 2258A?
Who at your company can see our content, including illegal material in queues? Where is it stored and for how long?
Do you use our content or reviewer decisions to train models used for other customers?
Which roles can perform bulk enforcement, account takeover actions or data export, and can those require a second approver?
How are API keys and service accounts for integrations scoped, rotated and revoked? Can they use short-lived tokens?
What moderator wellbeing controls are built in, and can we configure exposure limits?

Red flags

Accuracy quoted as one number: a single "99% accurate" figure across all harms hides weak categories and languages, and does not answer the DSA accuracy reporting question.
No structured decision record: free-text notes instead of fields for ground, category and automation make statements of reasons and transparency reports a manual exercise.
Fully automated removal with no review path: no human review sampling, no appeal workflow and no overturn tracking.
Weak admin controls: shared accounts, no MFA, bulk actions without approval, or exports with no audit trail. Internal support and admin tools are prime targets; in the Mailchimp breach, attackers used a hijacked internal support tool to reach customer accounts and API keys.
Your content used for training by default: including illegal material and private messages.
Illegal material copied around the vendor's estate: no clear answer on where CSAM and similar material is stored, who can access it and how it is deleted.

Where identity fits

A trust and safety platform holds some of the most powerful roles in your service: suspend any account, read reported private content, restore access and export user data. It also runs on machine identities: ingestion pipelines, classifier service accounts, webhook secrets and API keys for hash list providers. Put moderator and admin roles behind SSO with phishing-resistant MFA, use separate roles for reviewers, policy authors and administrators, require approval for bulk and high-impact actions, and keep every integration credential in a secrets manager with an owner and rotation. Abuse of admin tooling should feed your SOC; see the Fake Account Creation and Sign-up Fraud Guide for the account-level signals that should flow both ways.

Proof of concept

Agree success thresholds: minimum precision and recall per priority category, queue SLA, and the reporting outputs that must work, signed by trust and safety, legal and security.
Build a labelled test set: a representative sample of your own content across languages and harm categories, labelled by your policy team, handled under strict access rules. Never use real illegal imagery in a vendor test outside an approved hash-only workflow.
Test hash matching safely: use the provider's test hashes or benign images you own, then check matches survive resizing, cropping and re-encoding.
Run shadow mode: feed live traffic to the platform without enforcement for two to four weeks and compare its decisions with your current process.
Exercise the decision record: generate statements of reasons, an appeal, a reversal and a quarter of transparency figures from the POC data.
Test security controls: attempt bulk actions and exports from low-privilege roles, rotate an API key, and confirm every admin action appears in your SIEM.
Test moderator experience: have your reviewers work real queues and score the tooling, including wellbeing controls.
Plan rollback: keep the existing process running until thresholds are met for a full cycle, and confirm data export and deletion before signing.

Detection and measurement

Precision and recall by category and language: from QA sampling of automated decisions.
Appeal overturn rate: high rates show weak policies or classifiers.
Time to action: for priority notices, trusted flagger reports and the most severe harm categories.
Hash match volume and false positive rate: reviewed per list.
Privileged actions without second approval: should trend to zero.

Framework mapping

Framework	Reference	Relevance to the purchase
EU Digital Services Act	Notice and action, Article 17 statements of reasons, transparency reporting, trusted flaggers, internal complaints	Core data model and reporting outputs
Implementing Regulation (EU) 2024/2835	Harmonised transparency templates	Fields the platform must produce
UK Online Safety Act 2023	Ofcom illegal content codes ICU A, C and D measures	Moderation, automated detection, complaints and governance capability
18 U.S.C. 2258A	CyberTipline reporting and preservation	Evidence and reporting workflow
NIST SP 800-53 Rev. 5	AC-3, AU-2, AU-6, IR-6	Role enforcement, admin audit logging, incident reporting
NIST CSF 2.0	GV.OC-03, GV.SC-07	Legal obligations and supplier risk for the platform vendor

Standards and references

How NHI Mgmt Group can help

We provide independent advice on platform security, privileged tooling and the identities behind it. Browse vendors in our products directory or contact us.

Trust and Safety Platforms Buyer’s Guide: Choosing Moderation, Detection and Case Management Tools