A data classification tool is only worth buying if its labels are accurate enough to drive real controls, and most evaluations never measure that. Classification tools discover data, decide how sensitive it is and attach a label that encryption, DLP, access policy and AI controls can act on. Products range from labelling add-ins for office documents to discovery engines that scan databases, cloud storage and SaaS, and many now sit inside wider data security platforms. This vendor-neutral buyer's guide helps you define requirements, compare capabilities, question vendors and run a proof of concept that measures accuracy on your own data. It assumes you already have a classification scheme; if not, start with the Data Classification Guide and the Data Classification Policy Template.
Key takeaways
- Buy for enforcement, not dashboards. A label that no DLP, encryption or access rule reads is decoration; confirm which of your controls will consume it.
- Measure precision and recall on your data. Vendor accuracy claims come from vendor test sets; a proof of concept on a labelled sample of your own files is the only reliable number.
- Coverage decides value. Check every store you hold sensitive data in: file shares, collaboration suites, cloud object storage, databases, SaaS and AI tools.
- Labels must persist and travel. A label should survive copy, export and email, and be readable by other tools through metadata or an API.
- The scanner is a privileged identity. It reads everything, so its service accounts, tokens and data handling deserve the same scrutiny as a PAM tool.
Define requirements
- Scheme: how many levels you use (three or four is typical), any regulated categories (personal data, payment card data, health data, CUI, export-controlled data) and whether labels must carry sub-labels or handling caveats.
- Data estate: volumes and locations of unstructured data (shares, document libraries, mail, object storage) and structured data (databases, warehouses, data lakes), plus SaaS and generative AI tools in use.
- Labelling model: user-driven, default labels, recommended labels or fully automatic. Most programmes need a mix; see the labelling methods section of the Data Classification Guide.
- Consumers of the label: DLP, encryption and rights management, CASB, access governance, backup, retention and AI usage controls. List which ones you own today.
- Compliance drivers: ISO/IEC 27001:2022 Annex A 5.12 Classification of information and 5.13 Labelling of information, NIST SP 800-53 Rev. 5 RA-2 Security Categorization and AC-16 Security and Privacy Attributes, CIS Controls v8.1 Safeguard 3.7, and NIST CSF 2.0 ID.AM-07 (inventories of data and metadata).
- Constraints: data residency for scan results, whether content may leave your tenancy for analysis, languages and file types, and budget model (per user, per terabyte or per data source).
NIST's National Cybersecurity Center of Excellence published SP 1800-39, Data Classification Practices, as an Initial Public Draft in February 2026; it demonstrates discovering and labelling unstructured data with commercial tools and is a useful reference architecture while it is finalised. The earlier concept paper, NIST IR 8496, never left draft and is now marked withdrawn, so do not cite it as a requirement.
Capability areas
| Area | What to look for |
|---|---|
| Discovery and connectors | Native connectors for each store you named, agentless where possible, incremental scanning, scan scheduling and throttling so production systems are not overloaded |
| Detection methods | Pattern and checksum validation (card numbers, national IDs), exact data match against your own records, document fingerprinting for templates, trainable or machine learning classifiers, OCR for images and scans, context such as file location and owner |
| Structured data | Column-level classification in databases and warehouses, sampling controls, tagging that the data platform's own access policies can use |
| Labelling | Manual, default, recommended and automatic labelling; visual markings; justification when a user lowers a label; label inheritance for derived files |
| Label persistence and interoperability | Labels written into file metadata, preserved on copy and export, readable by third-party tools through an API or open metadata format (NIST SP 800-53 SC-16 covers transmitting attributes with data) |
| Policy actions | Encryption or rights management triggered by label, integration with DLP and CASB, quarantine or access removal for overexposed files |
| Generative AI | Classification of prompts, uploads and AI-generated content; labels honoured by AI assistants and retrieval pipelines. See the Generative AI Data Loss Prevention Guide |
| Reporting | Inventory of sensitive data by location, owner and exposure; label change history; export to SIEM and GRC tools |
If your main problem is finding overexposed sensitive data across cloud stores, rather than labelling it, compare against the DSPM Buyer's Guide; if it is stopping data leaving, see the DLP Buyer's Guide.
Evaluation criteria
| Criterion | Weight (example) | How to score it |
|---|---|---|
| Accuracy on your data | 25% | Precision and recall per sensitive type on a labelled sample from the proof of concept |
| Coverage of your estate | 20% | Share of in-scope stores reachable with supported, production-ready connectors |
| Enforcement integration | 15% | Number of your existing controls that read the label without custom code |
| Usability for end users | 10% | Pilot user feedback, label prompts per day, time to apply a label |
| Scale and performance | 10% | Time to complete a first full scan and daily incremental scans of the pilot data set |
| Security of the tool itself | 10% | Least-privilege connector permissions, credential handling, where content and results are stored, certifications |
| Cost and operating effort | 10% | Licence model at your volumes, tuning effort, staff hours per month to run |
Smaller organisations usually get most value from labelling built into the collaboration suite they already license, plus a short list of custom detectors. Large or regulated organisations with many data stores generally need a dedicated discovery engine and should weight coverage and interoperability higher.
Questions to ask vendors
- Accuracy: how do you measure false positives and false negatives, and can we see results per sensitive type on our sample rather than an aggregate?
- Tuning: can we build exact data match and fingerprint detectors from our own records without sending raw values to you? How are trainable classifiers trained and retrained?
- Scanning model: does content leave our environment for analysis? Where are scan results, snippets and match evidence stored, for how long, and in which region?
- Permissions: exactly which API scopes and roles does each connector need? Can it run read-only? Can credentials be held in our secrets manager and rotated?
- Persistence: where is the label stored for each file type, and what happens when a file is copied to a store you do not scan, renamed, zipped or exported to PDF?
- Interoperability: which third-party DLP, CASB, backup and AI tools read your labels today, and is the label format documented?
- Structured data: how do you sample large tables, and can column tags be pushed to the database's own access policies?
- AI data: can you classify prompts and AI outputs, and do AI assistants respect your labels when retrieving content?
- Exit: if we leave, do labels remain readable, and can we export the inventory and detector definitions?
Red flags
- Accuracy figures quoted without a method, data set or per-type breakdown.
- Connectors that require global administrator or full write access when read access would do.
- Labels that live only in the vendor's database, so they vanish when a file leaves a scanned store.
- No way to run detectors in simulation or audit mode before labels are applied or actions taken.
- Match evidence (the sensitive values themselves) stored in the vendor's cloud with no retention control.
- A proprietary label format that no other tool can read and that cannot be exported.
- Pricing per terabyte scanned with no cap, which penalises the complete discovery you need.
Proof of concept
- Build a ground truth set. Pick 2,000 to 5,000 files and a few database tables from real stores, and have data owners label them by hand against your scheme. Include hard cases: scans, spreadsheets with mixed content, other languages, and look-alike numbers that are not sensitive.
- Connect with least privilege. Grant each shortlisted tool read-only access to the same pilot stores using dedicated service accounts, and record the permissions each one demanded. See the Service Account Security Guide.
- Scan in audit mode. Run discovery and recommended labelling without applying labels or actions.
- Score accuracy. Calculate precision and recall per sensitive type against the ground truth; set a minimum (for example, recall above 90% for regulated identifiers) before the trial starts.
- Test persistence and enforcement. Apply labels to a pilot ring of 20 to 50 users, then copy, export, email and upload labelled files and confirm your DLP, encryption and AI controls act on them.
- Test rollback. Remove or downgrade labels in bulk and confirm files remain accessible and protection is reversed cleanly.
- Measure operations. Record scan duration, tuning hours, user prompts per day and help desk tickets over the trial, then score against the evaluation criteria table.
After rollout, track the share of sensitive stores scanned, the percentage of files carrying a label, label downgrade rate with justifications, and DLP incidents triggered by labels, so you can show the tool is driving protection and not just counting files.
How NHI Mgmt Group can help
We provide independent requirements, RFP and evaluation support across data security, IAM, PAM and NHI. Browse vendors in our products directory or contact us.
Related NHI Mgmt Group resources: Data Classification Guide · DSPM Buyer's Guide · DLP Buyer's Guide · Data Loss Prevention (DLP) Guide