Red teaming open Chinese AI models exposes uneven safety

At a glance

What this is: Holistic AI’s red-team testing of open Chinese AI models found strong performance but uneven safety, with large variation in jailbreak resistance and safe-response behaviour.

Why it matters: For AI governance teams, the issue is not whether open models can perform well, but whether model onboarding, testing, and runtime controls are strong enough to manage inconsistent safety behaviour.

Context

Open source and open-weight AI models can lower deployment barriers, improve local control, and support faster experimentation, but those benefits do not remove governance risk. When safety behaviour varies widely across models, the control problem shifts from model selection to how organisations validate, constrain, and observe model behaviour before and after production release.

This article sits in AI governance rather than pure model benchmarking. The identity-adjacent question is whether organisations treat the model itself as a governed system with defined access, policy enforcement, and monitoring, or as a one-time procurement decision. For teams running agentic AI or other production AI workflows, that distinction matters because unsafe model behaviour can become an operational control failure, not just a content-quality issue.

Key questions

Q: What fails when an open AI model passes benchmarks but still jailbreaks easily?

A: The failure is that benchmark performance and safety readiness are being treated as the same thing. A model can answer well on standard tasks and still be steerable into unsafe behaviour through role-play, prompt injection, or context shifts. Organisations need separate approval criteria for capability, refusal behaviour, and runtime containment.

Q: Why do prompt injection attacks create governance risk for AI agents?

A: Prompt injection creates governance risk because the model often sits in the control path between text input and tool execution. If attackers can change what the model treats as authoritative, they can influence access decisions, data exposure, or downstream actions without compromising a traditional account. That makes prompt provenance and instruction hierarchy part of AI identity governance.

Q: How do you know if AI safety testing is actually working?

A: Look for consistent rejection of the same abuse pattern across multiple prompt variants, model versions, and tool-integrated workflows. If a system only resists one phrasing but fails under paraphrase or language shifts, the control is brittle. Strong programmes measure stability, not just pass rates on a single benchmark.

Q: When should organisations use runtime guardrails instead of relying on pre-deployment testing?

A: Use runtime guardrails whenever the system can change behavior after launch, reach tools, or affect live business operations. Pre-deployment testing remains necessary, but it cannot replace live enforcement when the risk is produced by emergent behavior. The two controls solve different parts of the problem and should be linked.

Technical breakdown

Why red teaming open models matters before production use

Red teaming is adversarial testing designed to provoke unsafe, policy-violating, or unexpected behaviour before users rely on a model. For open source and open-weight models, it matters because release transparency does not guarantee runtime safety. A model can be strong at reasoning or coding while still failing to resist jailbreaks, role-play prompts, or context shifts. The testing question is therefore not only what the model can do, but how consistently it refuses harmful requests under pressure. In governance terms, this is a model risk control, not a branding exercise.

Practical implication: Treat pre-production red teaming as a release gate, not a documentation step.

Safe-response rates and jailbreak resistance measure different failures

Safe-response rate measures how often a model stays aligned under harmful or borderline prompts, while jailbreak resistance measures how well it withstands attempts to bypass safeguards through prompt injection, role-play, or similar manipulations. Those are related but not identical. A model may refuse obvious unsafe prompts but still be brittle when attackers change the framing or context. That distinction matters for enterprise use because policy compliance in a lab does not prove resilience in production workflows where users, agents, and tool chains can shape prompts dynamically.

Practical implication: Evaluate both alignment under pressure and resistance to prompt manipulation before approving a model for use.

Continuous monitoring is the missing control when model behaviour can drift

Model safety is not fixed at deployment. Prompt patterns, usage contexts, and model updates can change how often unsafe outputs appear, especially when the system is wrapped with tools or used by agents. That is why monitoring and observability are part of governance, not optional telemetry. Organisations need a way to detect degradation, see policy-violating outputs, and trace when behaviour diverges from the approved baseline. In practice, the control failure is assuming that one-time testing can cover a living system.

Practical implication: Instrument runtime monitoring so safety regression is visible before it reaches users or downstream agents.

Threat narrative

Attacker objective: The objective is to induce unsafe model output that should have been refused, exposing where safety controls are brittle.

1. Entry occurs when a user or agent supplies a harmful, borderline, or context-shifted prompt that is intended to bypass the model’s built-in safety behaviour.
2. Credentialed access is not the issue here; the failure mode is policy bypass through jailbreak-style manipulation, including role-play and adversarial prompt framing.
3. Impact occurs when the model produces disallowed, misleading, or unsafe output that an organisation then has to catch with external governance controls.

Breaches seen in the wild

• Nx s1ngularity attack 2025: Attackers stole Nx's npm token via a GitHub Actions flaw and shipped malware that stole 2,349 secrets and abused developers' AI CLIs.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Uneven safety is a governance problem, not a benchmark problem. The article shows that model capability and safety readiness can diverge sharply in the same model family. That means procurement teams cannot use performance alone as a proxy for operational trust. The right question is whether the organisation has a repeatable control framework for testing, approving, and observing model behaviour before production use.

Open weight does not mean open risk acceptance. Open models give organisations more deployment flexibility, but they also shift responsibility for safety controls onto the adopter. That includes adversarial testing, output filtering, and ongoing observability. For AI governance programmes, the implication is clear: openness expands choice, but it also expands accountability for the control stack around the model.

Jailbreak resistance is a practical proxy for control brittleness. When a model can be steered through role-play or context changes, the failure is not just content moderation. It shows that policy enforcement is too close to the model and too far from independent governance layers. The named concept here is safety drift gap: the difference between a model that passes initial testing and one that remains aligned under real-world prompting. Practitioners should govern for that gap explicitly.

Production AI needs the same discipline as other high-risk systems. The article reinforces that AI governance has to behave like an operational control function, not a one-time review. That means testing, monitoring, auditability, and escalation paths must exist before deployment, especially where models support workflows that can affect decisions, access, or downstream automation. The practitioner conclusion is to treat model safety as a managed control surface.

Agentic and tool-using systems raise the stakes for weak safety controls. The article’s reference to agentic reasoning matters because models that can plan, call tools, or influence workflows can turn a prompt vulnerability into a broader operational issue. That is where AI governance intersects with identity and access governance: if the model can act through tools or delegated access, its safety profile affects more than text quality. Practitioners should align model controls with the privileges those systems can exercise.

From our research library:

• 13% of organisations suffered breaches of their AI models, according to IBM's Cost of a Data Breach 2025 report.

What this signals

safety drift gap: the practical issue is not whether an open model can perform, but whether its safety posture holds once prompts, users, and workflows become dynamic. Teams that separate benchmark success from containment success will make better go-live decisions and will spot brittle safeguards earlier.

Open models should be governed as living systems with release gates, runtime controls, and observability, especially when they may be wrapped into agentic workflows or tool-using applications. The programme risk is assuming a one-time evaluation can substitute for continuous control.

Where models can influence tools or downstream decisions, AI governance has to connect to access control and auditability. That intersection is where model safety becomes operational, because weak containment can turn prompt manipulation into a broader workflow or identity governance issue.

For practitioners

• Set a pre-production red-team gate Require adversarial testing for harmful prompts, role-play, and context-shift exploits before any open model is approved for production use.
• Separate model quality from safety approval Score performance, refusal behaviour, and jailbreak resilience independently so a strong benchmark result cannot override weak containment.
• Add runtime safety guardrails Use policy-based filtering and reinforcement layers in the application path so unsafe outputs are intercepted after deployment as well as before it.
• Monitor for safety drift over time Track prompt and response patterns, unusual output shapes, and policy-violating events so degradation is visible as models and usage contexts change.

Key takeaways

• Open Chinese AI models can deliver strong performance while still showing inconsistent safety behaviour under adversarial prompting.
• The article’s testing results show that jailbreak resistance and safe-response rates can vary sharply across models, so benchmark strength alone is not enough for approval.
• Enterprises need independent red teaming, runtime guardrails, and continuous observability before treating an open model as production-ready.

Key terms

• Red Teaming: Red teaming is structured adversarial testing used to find how an AI system fails under realistic misuse or attack conditions. In AI security, it is a discovery method, not a proof of safety, because probabilistic behaviour and changing models prevent any lasting guarantee.
• Jailbreak resilience: The ability of a model to resist attempts to override its safeguards through prompt injection, role-play, or contextual manipulation. It measures whether the model can keep its policy boundaries intact when an attacker tries to persuade it to ignore them.
• Safe-response rate: The proportion of model outputs that remain aligned with policy when tested against harmful, borderline, or sensitive prompts. It is a governance metric, not just a benchmark score, because it shows how often the model preserves intended safety behaviour under pressure.
• Safety Drift: Safety drift is the gradual loss of intended safety behaviour after a model has been tested or deployed. It can result from changed prompts, changing workflows, tool use, or model updates, which is why monitoring must continue after launch.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps security practitioners connect identity controls to AI risk in the programmes they run.