Text-to-Cypher evaluation is the process of testing whether a model can turn natural language questions into correct Cypher queries for graph databases. It checks both language understanding and query generation, usually against a known ground truth. In multilingual settings, the same task must be validated across each target language.
Expanded Definition
Text-to-Cypher evaluation measures how reliably a system converts natural language prompts into executable Cypher for graph databases, then compares the output against an agreed reference answer. For NHI Management Group, the key issue is not only syntactic correctness, but whether the generated query preserves intent, scope, and access boundaries when the graph is used for security, identity, or knowledge retrieval workflows. In practice, evaluation often spans exact match, semantic equivalence, and execution-based checks, because two queries can look different yet return the same result, or look similar and behave differently.
The term is still applied inconsistently across the industry. Some teams use it narrowly for benchmark scoring, while others include prompt robustness, multilingual parity, and resistance to query hallucination. That makes the evaluation design as important as the score itself. A useful baseline is to anchor outcomes to the governance intent described in the NIST Cybersecurity Framework 2.0, especially where the generated query supports security operations or access decisions. The most common misapplication is treating string-match accuracy as proof of correctness, which occurs when teams ignore execution results, language variation, or schema-sensitive edge cases.
Examples and Use Cases
Implementing Text-to-Cypher evaluation rigorously often introduces more test-maintenance overhead, requiring organisations to weigh faster prototyping against stronger assurance that queries behave as intended.
- A security analyst asks for all service accounts that have not rotated secrets in 90 days, and the model must generate a Cypher query that targets the correct labels, relationships, and time filters.
- An identity team tests whether the model can answer multilingual questions about account ownership or privilege inheritance without drifting into unrelated graph nodes.
- A knowledge graph team validates that natural-language searches return the same result set as a manually authored ground-truth Cypher query, not merely a similar-looking query.
- A red team evaluates whether prompt variation causes the model to omit a required constraint, such as tenant scope or environment boundary, which could expose cross-domain data.
- An engineering group uses benchmark sets to compare models, then reviews failures where the query is valid Cypher but semantically wrong for the underlying graph schema.
Where graph-backed workflows support security operations, evaluation should also consider whether the query can safely traverse identity relationships, privileged links, or agent-generated records. That is especially relevant when the graph contains NHI metadata, ownership mappings, or delegated access paths.
Why It Matters for Security Teams
For security teams, the risk is not simply that a generated query fails. The larger concern is silent correctness failure, where a valid query returns the wrong slice of data and shapes a false operational decision. That matters in graph databases used for identity investigation, attack-path analysis, asset discovery, and NHI governance, because the query itself becomes part of the control surface. If evaluation ignores schema drift, multilingual ambiguity, or execution equivalence, teams can overestimate trust in an interface that appears reliable only in limited test cases.
Text-to-Cypher evaluation also intersects with AI security governance because it provides a measurable way to assess whether a model is fit for a bounded operational role. Guidance from the NIST Cybersecurity Framework 2.0 is relevant when the generated query supports detection, investigation, or access control decisions, while broader AI risk practices apply when the model is exposed to prompt injection or adversarial phrasing. Organisations typically encounter the real impact only after a misleading query has driven a bad investigation, at which point Text-to-Cypher evaluation becomes operationally unavoidable to address.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST CSF 2.0, NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV.OV-01 | CSF 2.0 emphasises outcome oversight for systems that affect security decisions. |
| NIST AI RMF | MEASURE | AI RMF Measure covers testing and validation of AI system behaviour and performance. |
| NIST AI 600-1 | The GenAI profile stresses evaluation of model outputs and task performance in context. | |
| OWASP Agentic AI Top 10 | Agentic AI guidance addresses output errors and tool misuse that can arise in query generation. | |
| OWASP Non-Human Identity Top 10 | NHI governance is relevant when generated graph queries touch identity and access relationships. |
Define evaluation ownership and review query-quality outcomes that affect operational decisions.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on August 24, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
