What do organisations get wrong when balancing AI performance and safety?

Why This Matters for Security Teams

When organisations compare models only on benchmark scores, they miss the operational reality that safety failures often appear as security failures, privacy failures, or compliance failures. A model that is faster, cheaper, or more accurate can still be the wrong choice if it increases prompt leakage, secrets exposure, or unsafe tool use. NHIMG research on the LLMjacking threat vector shows how quickly exposed credentials can be abused, which matters because model choice is inseparable from the identity and access surface around it.

Security teams also need to account for the fact that AI systems learn from live data paths, retrieval sources, and agent actions, not just the model card. That means performance tuning can widen the blast radius if access is too broad, logs are too permissive, or safety filters are treated as a cosmetic layer. Guidance in the NIST Cybersecurity Framework 2.0 reinforces the need to align protection with business outcomes, but in AI environments the control boundary is wider than the model itself.

In practice, many security teams discover the cost of that gap only after an internal pilot has already connected a powerful model to sensitive data and downstream tools.

How It Works in Practice

The right approach is to define minimum safety controls before model selection, then score candidate systems against both capability and risk. That includes data handling, tool permissions, output filtering, logging, human review, and rollback. If a model can outperform peers only by accepting broader context windows, weaker guardrails, or more permissive plugin access, that performance gain may be operationally unacceptable.

Practitioners should separate three layers of evaluation:

• Model capability: task quality, latency, cost, and reliability under expected workloads.

• System safety: prompt injection resistance, data leakage risk, policy enforcement, and abuse resistance.

• Governance fit: auditability, retention, access controls, and whether the deployment can satisfy internal and regulatory requirements.

That separation matters because the model is only one control point. The surrounding orchestration layer determines whether the system can safely call tools, read sensitive context, or escalate requests for approval. NHIMG’s DeepSeek breach coverage illustrates how exposure in the wider AI stack can turn a technical issue into a governance event. For control design, the model evaluation should be paired with policy-as-code, least privilege, data minimisation, and explicit boundaries for retrieval and external action.

Best practice is to require a documented safety threshold for each use case, then test candidate models against that threshold with realistic adversarial prompts and red-team scenarios. These controls tend to break down when teams move from controlled pilots to production integrations with multiple data sources and agentic tool access, because the surrounding workflows expand faster than the safety review process.

Common Variations and Edge Cases

Tighter safety controls often increase latency, reduce response richness, and add review overhead, so organisations have to balance user experience against risk tolerance. There is no universal standard for this yet, and current guidance suggests using different thresholds for low-risk productivity tasks versus customer-facing, regulated, or decision-support workflows.

One common edge case is retrieval-augmented generation: a model may be safe in isolation but unsafe once connected to sensitive documents, tickets, or customer records. Another is agentic automation, where the system is allowed to act, not just answer. In those environments, performance can degrade when controls are added, but that is often the cost of preventing overbroad access or irreversible actions. The answer is not to remove safety checks, but to place them where they do not block legitimate work.

Security leaders should also beware of procurement claims that confuse benchmark performance with operational readiness. A model may excel on synthetic tests while failing basic governance needs such as audit trails, access scoping, or prompt injection resilience. The safest choice is frequently the model that is slightly less impressive on a benchmark but materially easier to contain. The practical lesson is simple: evaluate the entire AI system, not just the model, and do not trade away control for marginal score improvements.

Related resources from NHI Mgmt Group

• What do organisations get wrong about AI safety and access control?
• What do organisations get wrong about AI agent safety at design time?
• What do organisations get wrong about AI safety and hallucinations?
• What do organisations get wrong about shadow AI governance?