What should organisations prioritise first: interpretability or robustness testing?

Why This Matters for Security Teams

For organisations deploying AI in production, the choice between interpretability and robustness testing is really a question of sequencing risk controls. Robustness testing checks whether the model can survive prompt manipulation, edge-case inputs, adversarial examples, and unstable tool behaviour. Interpretability helps teams understand why the model produced a given output, which is essential for investigation, governance, and accountability. NIST’s NIST Cybersecurity Framework 2.0 reinforces the broader principle: controls should reduce operational risk first, then improve maturity and visibility over time.

The common mistake is treating interpretability as a substitute for resilience. A model can be highly explainable and still fail under prompt injection, data drift, or unsafe tool use. It can also be robust in narrow tests while remaining opaque enough that teams cannot tell whether failures come from the model, the prompt, the retrieval layer, or downstream automation. Security teams should therefore prioritise the control that reduces the most immediate exposure in the current deployment context.

In practice, many security teams encounter model failure only after an exposed workflow has already been manipulated, rather than through intentional pre-production testing.

How It Works in Practice

Robustness testing is the first line of defence when an AI system is already making decisions, generating content, or triggering actions. The goal is to see how the system behaves under stress, especially where inputs are malformed, adversarial, ambiguous, or outside the intended operating range. For LLM-enabled systems, this includes prompt injection attempts, malicious tool instructions, retrieval contamination, unsafe chaining, and output instability across repeated runs. Guidance from the NIST Cybersecurity Framework 2.0 is useful here because it frames security as an ongoing function, not a one-time model qualification.

• Test for prompt sensitivity using benign variations and adversarial phrasing.
• Assess whether retrieval content can override system instructions.
• Check whether tool calls execute only when intended and with bounded scope.
• Compare outputs across repeated runs to identify instability and hidden brittleness.

Interpretability then becomes the governance layer that supports root cause analysis and policy decisions. It is most valuable when outputs affect customers, regulated decisions, or privileged workflows. Teams may use saliency methods, feature attribution, tracing, or structured output logs, but current guidance suggests these methods should be treated as decision-support tools rather than proof of correctness. For AI systems that rely on external context, interpretability should also include provenance of prompts, retrieved sources, and tool actions, so investigators can reconstruct what influenced the result. The NIST Cybersecurity Framework 2.0 and the OWASP Top 10 for Large Language Model Applications both support this layered view of preventive testing and diagnostic visibility.

These controls tend to break down when the model is embedded in fast-moving agentic workflows with multiple tools, weak logging, and no stable test harness because failures become non-deterministic and difficult to reproduce.

Common Variations and Edge Cases

Tighter interpretability often increases implementation overhead, requiring organisations to balance explainability against delivery speed and model complexity. In high-risk use cases such as hiring, lending, insurance, or customer-facing agentic assistants, interpretability may deserve earlier investment because governance teams need to justify decisions, investigate complaints, and demonstrate oversight. In lower-risk internal workflows, robustness testing usually has clearer immediate value because the first concern is preventing misuse, instability, or unsafe automation.

There is no universal standard for this yet, but current guidance suggests a simple rule: prioritise robustness first when the model can act, decide, or generate externally visible outcomes; prioritise interpretability earlier when the model is high-impact, regulated, or likely to be challenged. The OWASP Top 10 for Large Language Model Applications is useful for identifying concrete failure modes, while the NIST Cybersecurity Framework 2.0 helps teams map those failures to operational safeguards and response ownership.

For NHI-governed AI systems, the boundary between model risk and identity risk also matters. If an agent can use tokens, APIs, or service accounts, robustness testing must include credential abuse and privilege misuse, not just text quality. Interpretability may help explain why an agent acted, but it will not by itself stop a compromised identity from issuing harmful actions. Best practice is evolving, but organisations should avoid treating explainability as a substitute for strong access controls, logging, and adversarial testing.

Related resources from NHI Mgmt Group

• Should organisations prioritise static testing or runtime testing first?
• Should organisations prioritise external exposure or internal credential governance first?
• Should organisations prioritise secret rotation or access review first
• Should organisations prioritise discovery or access restriction first for shadow AI?