Notifications
Clear all
Topic starter
18/08/2026 2:42 pm
TL;DR: Refusal behavior in Qwen3-4B-Instruct is not isolated to a single vector, but emerges as geometric separation in activation space that collapses after abliteration, which also helps explain spillover across domains, according to CRACKEN. The finding matters because orthogonal-looking safety directions can still share the same underlying mechanism, making multi-dimensional safety more fragile than simple vector similarity suggests.
NHIMG editorial — based on content published by CRACKEN: LLMs refusal is geometric, not categorical, after abliteration
By the numbers:
- The study used 250 prompts across 6 domains to map refusal behavior in activation space.
- PCA explained 93.2% of the variance with just 2 components, which is why the geometry was interpretable.
Questions worth separating out
A: The most common failure is spillover.
Q: Why do orthogonal refusal vectors not guarantee safe isolation in an LLM?
A: Because vectors describe where a behaviour points in activation space, not whether the network implements that behaviour with separate machinery.
Q: How can security teams tell whether AI safety controls are actually independent?
A: They should look for evidence of structural separation, such as distinct pathways, non-overlapping control layers, and post-change regression testing across adjacent behaviours.
Practitioner guidance
- Test safety interventions by mechanism, not only by prompt category Run ablation, probing, and representation analysis to see whether changes made for one policy area affect unrelated behaviours in the same layer or projection pathway.
- Map shared-layer dependencies before tuning refusal controls Identify which model layers and matrices carry multiple safety behaviours so governance teams can document where one intervention may create a broader blast radius.
- Validate behavioural independence after any safety update Re-test adjacent policy domains after each model change, especially where the same residual-stream components are used to enforce multiple refusals.
What's in the full article
CRACKEN's full blog post covers the mechanistic detail this post intentionally leaves at the analysis layer:
- PCA plots and clustering metrics for before-and-after activation-space comparisons across harmful and harmless prompts
- The layer-by-layer discussion of why layer 18 was chosen and how middle-layer behaviour influenced the result
- The ablation method used on o_proj and down_proj, including how the shared mechanism was modified
- The author’s references to sparse autoencoders, Gemma Scope, and transformer circuit research that support the geometry argument
👉 Read CRACKEN's analysis of LLM refusal geometry and abliteration →
LLM activation space after abliteration: what changed in refusal?
Explore further
18/08/2026 3:26 pm
Shared mechanism, not vector similarity, is the governing risk. The article shows why orthogonal-looking safety directions can still fail together when they are implemented through the same weight matrices. That is a familiar control problem in identity security too: logical separation is not the same as operational isolation. For AI governance, the lesson is that representation-level independence must be proven, not assumed.
A question worth separating out:
Q: What should teams do after a model ablation or safety fine-tuning exercise?
A: Re-run cross-domain evaluations immediately and compare refusal behaviour before and after the change across all adjacent policy areas. The goal is to detect collateral degradation early, document the mechanism affected, and decide whether the tuning introduced unacceptable blast radius before the model returns to production.
👉 Read our full editorial: LLM refusal is geometric, not categorical, after abliteration
