Llama 3.1 405B safety testing shows scale does not ensure safety

At a glance

What this is: VirtueAI’s review argues that Llama 3.1 405B does not materially improve safety versus smaller Llama 3.1 models and still shows gaps in several risk categories.

Why it matters: This matters because AI security and governance teams need controls for model testing, use-case restriction, and release review rather than assuming that larger foundation models are inherently safer.

By the numbers:

• 72% of organisations have experienced or suspect they have experienced a breach of non-human identities, with 46% confirmed and 26% suspected.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.

👉 Read VirtueAI's safety assessment of the Llama 3.1 405B model

Context

Large model safety is not a property that appears automatically at scale. This assessment of Llama 3.1 405B shows a familiar governance problem for AI security teams: model size, benchmark coverage, and vendor claims can create a false sense of assurance if use-case testing and release controls are weak.

For AI governance programmes, the real issue is the gap between theoretical benchmark performance and the controls needed in production. That gap matters for regulated industries, privacy-sensitive deployments, and any environment where an LLM is connected to tools, data, or downstream decision systems.

The article’s starting position is typical of modern AI safety work: it combines red teaming, policy framing, and benchmark comparisons, but it also exposes how hard it is to translate model evaluation into operational assurance.

Key questions

Q: How should security teams govern LLMs that can trigger tools or workflows?

A: Treat the LLM as an untrusted decision component, not an authorizer. Give it the minimum tool scope required, enforce policy outside the model, and require logging for every action it can influence. If the model can initiate work, then privilege, approval, and revocation controls must sit around it, not inside it.

Q: Why do larger models still need red teaming?

A: Larger models still need red teaming because scale does not eliminate unsafe behaviour. In practice, bigger systems can become more capable at generating harmful, biased, or misleading content, which means security teams must test the exact failure modes that matter in production instead of assuming capability gains equal safety gains.

Q: What do security teams get wrong about AI safety testing?

A: The common mistake is treating AI safety testing as if it were just another security scan. It is not. Safety testing is about proving how a model or agent fails under pressure, while traditional security tooling is about who can access the system. Those are different governance questions and need different evidence.

Q: How should organisations respond when an LLM passes broad safety tests but fails a specific use case?

A: They should treat the use-case failure as the deciding signal, not the broad benchmark result. If a model performs poorly in a regulated or sensitive workflow, the deployment should be constrained, re-tested, or blocked until controls, prompts, and review steps are redesigned.

Technical breakdown

Why larger foundation models do not automatically become safer

Model scale increases capability, but it does not guarantee alignment, refusal quality, or resilience against harmful prompting. Safety behaviour depends on training data, reward shaping, post-training controls, and the evaluation rubric used to measure outputs. A model can improve on some dimensions, such as privacy or robustness, while regressing on others like brand safety or regulated advice. That means security teams need to treat capability and safety as separate properties, not assume one implies the other.

Practical implication: validate safety by use case and failure mode, not by parameter count or benchmark rank.

How red teaming turns policy concerns into testable risk categories

Red teaming is the process of deliberately probing a model with adversarial prompts and scenarios to expose unsafe behaviour. In practice, that means mapping abstract policy goals into testable categories such as bias, harmful content, and regulated guidance. This is especially important when models are later fine-tuned or connected to applications, because downstream use can widen the attack surface. Good red teaming is therefore not just adversarial testing, but governance evidence for release decisions.

Practical implication: build red-team tests into pre-release gates and re-run them after fine-tuning or prompt changes.

Why AI safety needs use-case controls, not just general model evaluations

General benchmark results can miss the specific context in which an enterprise deploys a model. A model that appears acceptable in broad testing may still be unsafe when used for legal, medical, financial, or identity-related tasks. That is why use-case driven risk assessment matters: it ties outputs to the actual decision environment, sensitivity of data, and potential harm. In AI governance terms, this is the difference between model assurance and deployment assurance.

Practical implication: classify high-risk use cases separately and require tighter approval, monitoring, and human review.

Threat narrative

Attacker objective: The attacker objective is to coerce the model into unsafe, biased, or policy-violating behaviour that can be operationalised in downstream systems.

1. Entry occurs when a model is exposed to adversarial prompts, unsafe fine-tuning inputs, or downstream integrations that expand the reachable attack surface.
2. Escalation happens when the system produces harmful, biased, or regulated outputs that are then trusted by users or application logic.
3. Impact is realised when those outputs influence decisions, expose sensitive information, or create compliance and safety failures in production.

NHI Mgmt Group analysis

Model scale is not a governance control. Larger models can widen capability, but they do not remove the need for release gates, scenario testing, and approval criteria. The article shows that safety claims based on scale alone are incomplete, especially when the model is intended for enterprise use. Practitioners should treat benchmark uplift as evidence to examine, not as a substitute for control design.

Use-case risk is the real security boundary. A general-purpose benchmark can miss the point if the deployment target is regulated advice, identity-related decisioning, or privacy-sensitive workflows. That creates a governance debt problem: the model passes broad testing while the business context remains unassessed. Practitioners should define risk tiers by use case before allowing integration.

AI safety testing needs to behave like change management. Once a model is fine-tuned, wrapped in tools, or connected to workflows, its risk profile changes. The article reinforces that evaluation must be repeated after material changes, not only at first release. Practitioners should build re-testing triggers into model lifecycle governance.

Regulated industries need explicit control mapping, not generic safety language. References to fairness, privacy, and harmful content are useful only when mapped to an approval process and accountable owner. The regulatory framing in the article aligns with NIST AI RMF style governance, where risk identification, measurement, and management must be operationalised. Practitioners should convert policy into named controls and named sign-off points.

Agentic AI changes the meaning of safety evidence. When an LLM is embedded in a tool-using system, output safety becomes only one layer of the problem. Identity, delegation, and runtime permissions determine whether unsafe output can actually cause harm. Practitioners should pair model testing with access governance for any AI system that can act beyond chat.

What this signals

Model evaluation is becoming part of security governance, not just AI research. As LLMs move into enterprise workflows, teams need a repeatable way to connect benchmark findings, red-team results, and approval thresholds to operational risk decisions. The practical shift is toward evidence-based release management, especially where outputs can affect regulated or sensitive processes.

Identity and access controls matter once AI systems can act. If a model or agent can call tools, access data, or trigger workflows, then its permissions become part of the safety model. That is where NHI governance, secrets control, and workload identity begin to overlap with AI assurance, and the boundary should be explicit in programme design.

AI governance debt builds when testing is one-off instead of lifecycle-based. Re-testing after fine-tuning, prompt changes, or tool expansion should be treated as standard change control, not optional validation. For teams running connected AI systems, this is where the model risk function, IAM, and operational owners need a shared approval process.

For practitioners

• Define use-case risk tiers Classify LLM deployments by sensitivity, including regulated advice, privacy exposure, and decision impact. Require a higher approval bar for any workflow that can influence customer, employee, or financial decisions.
• Make red teaming a release gate Run adversarial tests before deployment and again after fine-tuning, prompt changes, or tool integration. Block release when the model produces repeatable unsafe outputs in the target use case.
• Separate model assurance from deployment assurance Do not rely on benchmark scores alone. Validate the model in the actual application context, including data sources, human review steps, and downstream consumers.
• Map safety findings to accountable controls Assign owners for fairness, privacy, harmful content, and regulated advice, then tie each issue to a specific approval or rollback decision.

Key takeaways

• The central risk is assuming that bigger models are inherently safer when the evidence shows safety remains uneven across use cases.
• The article’s findings reinforce that AI safety must be measured in the deployment context, not only through general benchmark comparisons.
• Practitioners need lifecycle governance, red teaming, and accountable release controls before high-risk LLMs enter production.

Key terms

• Red Teaming: Red teaming is structured adversarial testing used to find how an AI system fails under realistic misuse or attack conditions. In AI security, it is a discovery method, not a proof of safety, because probabilistic behaviour and changing models prevent any lasting guarantee.
• Use-Case Risk: Use-case risk is the harm a model can cause in a specific business context, not in abstract benchmark testing. It depends on the data involved, the decision being influenced, and the consequences of a wrong or unsafe answer.
• Safety Evaluation: Safety evaluation is the structured assessment of whether a model behaves acceptably under defined conditions. It combines benchmark testing, adversarial prompts, and policy checks to show where the model remains unsafe or unreliable.

What's in the full report

VirtueAI's full article covers the evaluation detail this post intentionally leaves at the governance level:

• Regulation-based risk taxonomy scoring across 45 categories for the Llama 3.1 family
• Use-case comparison results for fairness, privacy, hallucination, over-cautiousness, robustness, and brand risk
• Examples from VirtueRed red teaming that show how unsafe outputs appear in practice
• Measurement framing for the risk assessment framework and its stated guarantees

👉 VirtueAI's full post includes the benchmark comparisons, red-team examples, and risk assessment framing behind this analysis.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and agentic AI identity. It helps security practitioners connect identity controls to broader AI and workload risk management.