What breaks when organisations rely only on unit tests for LLM workflows?

Unit tests alone miss emergent behavior that appears when models interact with prompts, tools, retrieval layers, and external services. A response may look correct in isolation but fail in a full workflow because of bad context, unsafe outputs, or tool misuse. Teams need integration, behavioral, and regression testing to expose the failures that isolated checks cannot see.

Why This Matters for Security Teams

Unit tests are useful for proving that a function returns the expected value under a narrow set of inputs, but LLM workflows fail in ways that pure code tests are not designed to catch. The real risk is not a single bad response in isolation. It is the combination of prompt drift, retrieval quality, tool execution, and output handling across a chain of components. That is why guidance from the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 increasingly emphasises system-level assurance rather than isolated correctness checks.

Security teams often miss that an LLM workflow can pass every unit test and still fail policy, safety, or reliability requirements once it is connected to tools or live data. A model may generate a plausible answer, but that answer can still trigger an unsafe action, leak sensitive context, or mis-handle a malformed prompt. In practice, many security teams encounter these failures only after a workflow has already been integrated with production data, rather than through intentional testing of the full execution path.

How It Works in Practice

Effective testing for LLM workflows needs to reflect the actual runtime path: user input, prompt construction, retrieval-augmented generation, tool invocation, policy checks, and post-processing. Unit tests still matter, but they should be treated as the first layer, not the control plane. The missing layer is behavioural assurance, where the test asks whether the system behaves safely and consistently across a scenario, not whether one function returns a fixed string.

Practitioners usually combine several test types:

• Prompt and response tests for expected output shape, refusal behaviour, and formatting consistency.

• Integration tests for retrieval, tool calls, and service-to-service interactions.

• Adversarial tests for prompt injection, tool abuse, unsafe instructions, and context manipulation.

• Regression tests for previously observed failures so that prompt changes or model updates do not reintroduce them.

This approach aligns with the operational mindset in the NIST AI 600-1 Generative AI Profile, which pushes teams to validate behaviour across the lifecycle, not only at build time. It also fits the threat patterns tracked in the MITRE ATLAS adversarial AI threat matrix, where the issue is often not model math but how the system is manipulated in context. For agentic systems, testing should also verify tool permissions, approval gates, and whether the agent can be coerced into taking actions outside intended scope.

Where teams have retrieval layers, the test data matters as much as the model output. A clean unit test with a mocked answer will not reveal that the retriever returns stale policy text, a poisoned document, or irrelevant context that nudges the model toward a wrong decision. These controls tend to break down when prompts, retrieval, and external tools are decoupled across multiple teams because no single owner tests the full chain end to end.

Common Variations and Edge Cases

Tighter testing often increases maintenance overhead, requiring organisations to balance confidence against release speed. That tradeoff becomes sharper as workflows grow from simple chat helpers into stateful assistants with memory, tools, and privileged actions. Best practice is evolving, and there is no universal standard for how many behavioural or adversarial tests are enough.

Some teams overcorrect by using live production prompts in every test, but that can create brittle suites and unnecessary exposure of sensitive data. Others rely too heavily on synthetic inputs, which can hide real-world prompt styles, multilingual inputs, or maliciously crafted instructions. The stronger pattern is to keep deterministic unit tests for local logic, then add scenario-based tests that cover policy decisions, tool use, and failure handling.

This distinction is especially important when the workflow has identity-sensitive actions, such as accessing customer records, triggering payments, or escalating access through NHI-controlled tools. In those cases, the test plan should verify not only model output but also authorisation boundaries, logging, and rollback behaviour. The CSA MAESTRO agentic AI threat modeling framework is useful here because it treats the workflow as an attack surface, not just a prompt-response pair. The guidance breaks down most often in fast-moving environments where model versions, retrieval indexes, and tool schemas change independently and tests are not updated at the same pace.

Related resources from NHI Mgmt Group

• What breaks when organisations rely on legacy DLP for AI workflows?
• What breaks when healthcare organisations rely on annual penetration tests for PHI systems?
• What breaks when organisations rely on ChatGPT for workflows that include PHI?
• What breaks when organisations rely on data security controls that only cover storage systems and not AI workflows?