Notifications
Clear all
Topic starter
18/08/2026 2:28 pm
TL;DR: The largest model in the Llama 3.1 family does not deliver meaningful safety gains over smaller variants, according to VirtueAI’s assessment, while several risk categories remain exposed, including regulated advice, violence, and harmful beliefs. The lesson is that AI safety depends on use-case controls, evaluation discipline, and governance, not model scale alone.
NHIMG editorial — based on content published by VirtueAI: Comprehensive Safety Assessment of the Llama 3.1 405B Model
By the numbers:
- 72% of organisations have experienced or suspect they have experienced a breach of non-human identities, with 46% confirmed and 26% suspected.
- When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.
Questions worth separating out
Q: How should security teams govern LLMs that can trigger tools or workflows?
A: Treat the LLM as an untrusted decision component, not an authorizer.
Q: Why do larger models still need red teaming?
A: Larger models still need red teaming because scale does not eliminate unsafe behaviour.
Q: What do security teams get wrong about AI safety testing?
A: The common mistake is treating AI safety testing as if it were just another security scan.
Practitioner guidance
- Define use-case risk tiers Classify LLM deployments by sensitivity, including regulated advice, privacy exposure, and decision impact.
- Make red teaming a release gate Run adversarial tests before deployment and again after fine-tuning, prompt changes, or tool integration.
- Separate model assurance from deployment assurance Do not rely on benchmark scores alone.
What's in the full report
VirtueAI's full article covers the evaluation detail this post intentionally leaves at the governance level:
- Regulation-based risk taxonomy scoring across 45 categories for the Llama 3.1 family
- Use-case comparison results for fairness, privacy, hallucination, over-cautiousness, robustness, and brand risk
- Examples from VirtueRed red teaming that show how unsafe outputs appear in practice
- Measurement framing for the risk assessment framework and its stated guarantees
👉 Read VirtueAI's safety assessment of the Llama 3.1 405B model →
Llama 3.1 405B safety gaps: what does this mean for AI teams?
Explore further
18/08/2026 3:16 pm
Model scale is not a governance control. Larger models can widen capability, but they do not remove the need for release gates, scenario testing, and approval criteria. The article shows that safety claims based on scale alone are incomplete, especially when the model is intended for enterprise use. Practitioners should treat benchmark uplift as evidence to examine, not as a substitute for control design.
A question worth separating out:
A: They should treat the use-case failure as the deciding signal, not the broad benchmark result. If a model performs poorly in a regulated or sensitive workflow, the deployment should be constrained, re-tested, or blocked until controls, prompts, and review steps are redesigned.
👉 Read our full editorial: Llama 3.1 405B safety testing shows scale does not ensure safety
