Look for the disappearance of clear separation between harmful and harmless prompts in activation space, along with spillover into unrelated refusal domains. A small change in vector metrics is not enough to rule out collapse. The stronger signal is when formerly distinct prompt regions begin overlapping after intervention.
What collapse looks like in activation space
Model refusal collapse is not the same as a simple boundary shift. A shift moves the refusal decision surface while preserving separation between harmful and harmless prompts; collapse erodes that separation itself. The clearest sign is that prompts that used to land in distinct regions now overlap after the intervention, so the model no longer cleanly distinguishes refusal-relevant inputs from ordinary ones.
That distinction matters because collapse changes the geometry of the representation, not just the threshold. If the activation clusters blur together, the model can start refusing unrelated prompts, or it can lose the stable signal that previously supported refusal in the right cases. A single metric drift, by itself, does not prove collapse.
How to tell spillover from a legitimate policy change
Practitioners should look for spillover into unrelated refusal domains. If an intervention aimed at one harmful category starts affecting prompts that were never part of that category, the model is likely losing a general refusal structure rather than refining it. That is especially important when the behavior changes across semantically different prompt sets, not just within the original target set.
The practical test is whether the intervention preserves coarse semantic grouping. In a healthy shift, nearby prompts still cluster by meaning even if the refusal boundary moves. In collapse, formerly separate prompt regions begin to interpenetrate, so the model’s internal representation no longer supports clean discrimination. That usually shows up as broader refusal instability, not a tidy re-labeling of one class.
Do not overread small vector changes. A modest movement in activation distance can happen without any meaningful collapse, especially when the model is still keeping harmful and harmless prompts separable. The stronger evidence is structural: loss of cluster separation, cross-domain spillover, and refusal behavior that becomes less tied to the intended prompt category.
What practitioners should validate before calling it collapse
Validation should compare multiple prompt families, not a single benchmark. You want to see whether the intervention preserves distinct regions for harmful and harmless prompts across several refusal domains, and whether that separation remains stable under small prompt variations. If the same intervention makes many unrelated prompt families converge, that is much more consistent with collapse than with a narrow policy adjustment.
What to verify: Check whether the intervention preserves separability in activation space across both targeted and untargeted prompt sets, and whether the refusal change is localized or broad. If only the target set moves while neighboring regions stay distinct, it is more likely a shift. If the boundary disappears and unrelated prompts start behaving like the target class, treat it as collapse.
Common mistake: Teams often rely on one vector metric or one held-out prompt set and conclude the model is safe. For this kind of behavior, the error is assuming that any movement is a failure, or that one stable score means the geometry is still intact. The right question is whether separability survives under evaluation of the full prompt manifold.
Practitioner takeaway: Treat collapse as a structural loss of separation, not a numeric drift event. If the intervention preserves prompt clustering and only moves the refusal boundary, that is a shift; if distinct regions merge and refusal spills into unrelated domains, the model has likely collapsed.
Related resources from NHI Mgmt Group
- What are the signs that a cloud transformation program has stalled at migration rather than operating-model change?
- What are the signs that a truth probe is genuinely capturing model knowledge rather than a superficial pattern?
- What are the signs that a model is moving into ungrokking or semi-grokking rather than stable generalization?
- What are the signs that a model's feature importance is shifting after retraining an ensemble?