They fail because translation fidelity and task success are not the same thing. A model can preserve meaning in the translated question, yet still lose technical constraints, schema cues, or query intent when generating Cypher. That makes cross-lingual evaluation essential for catching errors where language understanding succeeds but query construction breaks down.
Why This Matters for Security Teams
Multilingual evaluation pipelines can appear healthy when the translation layer is accurate, yet still fail at the point that matters most: whether the downstream system preserves intent, structure, and constraints. For teams testing AI-assisted query generation, the risk is that a linguistically correct translation hides a functional defect in schema handling, entity mapping, or instruction following. That means evaluation can miss regressions until they surface in production behavior, not in language quality metrics.
This is especially important in security-sensitive workflows because a small semantic drift can change access scope, data filters, or evidence retrieval logic. Current guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls reinforces the need for integrity, traceability, and validation around system outputs, not just input handling. In practice, multilingual tests often overvalue human-readability and undervalue execution correctness, so teams conclude the pipeline is stable when it is only well-translated. In practice, many security teams encounter multilingual failure only after a translated prompt has already produced a structurally valid but semantically wrong query in production.
How It Works in Practice
A robust multilingual evaluation pipeline needs to test more than translation quality. It should check whether the translated prompt still carries the same task constraints, whether named entities survive intact, and whether the generated output matches the original execution intent. For Cypher generation, that means verifying labels, relationship directions, property filters, aggregation logic, and any language-specific entities that must not be altered.
The practical workflow usually has three layers:
- Translation fidelity review, to confirm the source meaning was preserved.
- Task-level equivalence testing, to compare the intent of the original and translated prompts.
- Execution validation, to run the generated Cypher against expected schema behavior and expected result sets.
Security teams should also pay attention to retrieval and prompt assembly. If the translated prompt changes token order, drops qualifiers, or normalises domain terms too aggressively, the model may still produce fluent output while losing operational meaning. This is where evaluation should include adversarial cases, such as ambiguous noun phrases, polysemous terms, and mixed-language prompts.
For AI governance, the NIST AI Risk Management Framework is useful because it frames validation as a lifecycle issue, not a one-time test. It pairs well with MITRE ATLAS for understanding how model outputs can be manipulated or misdirected through prompt and input attacks. Where multilingual systems involve agentic workflows, the same logic extends to tool use and action selection, because a correct translation does not guarantee correct execution. These controls tend to break down when the evaluation set is too small and does not include language pairs that stress domain-specific syntax, because the model then looks consistent only in the languages that are easiest to translate.
Common Variations and Edge Cases
Tighter multilingual validation often increases test cost and reviewer effort, requiring organisations to balance coverage against release speed. That tradeoff becomes sharper when the target language lacks strong annotation support or when the domain uses specialised terms that do not translate cleanly.
One common edge case is that a translation may be technically accurate but operationally unhelpful. For example, it may preserve meaning while flattening grammatical markers that the model used to infer entity scope or temporal constraints. Another issue is that some evaluation datasets measure semantic similarity rather than downstream correctness, which can reward output that sounds faithful but fails on execution.
There is also no universal standard for how much mismatch is acceptable between source and translated task structure. Current guidance suggests treating this as a risk-based decision: the more sensitive the query, the stricter the equivalence test should be. That matters in pipelines that touch access-controlled knowledge graphs, regulated records, or audit evidence, where even a small query shift can alter what is retrieved or disclosed. For that reason, teams should combine language metrics with schema-aware checks and human review for high-impact cases. The key lesson is that multilingual success is not translation success alone, but translation plus preserved operational intent.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Frames multilingual evaluation as lifecycle risk management, not just translation quality. | |
| MITRE ATLAS | Useful for adversarial prompt and input attacks that distort multilingual model behavior. | |
| NIST CSF 2.0 | PR.DS | Output integrity matters when translated prompts drive downstream data access or query generation. |
Define, measure, and monitor task-level AI risk across translation, generation, and output validation.
