Ungrokking is the reverse of grokking, where a model regresses from strong test performance back to poor generalization after continued training. It suggests that optimization can push the model away from an efficient solution and back toward a circuit that fits the training set without generalizing well.
How ungrokking happens
Ungrokking describes a late-training reversal in generalisation: a model that once performed well on held-out data starts fitting the training distribution more narrowly and loses its earlier test performance. The key idea is not just overfitting in the usual sense, but a regression after additional optimisation changes the learned circuit.
This makes ungrokking especially interesting as a training-dynamics phenomenon. The model may first discover an efficient representation that generalises, then move into a different solution basin that keeps lowering training loss while degrading out-of-sample performance. In practice, this means that “more training” is not always monotonic progress.
Ungrokking is usually discussed in the context of mechanistic interpretability and toy tasks, where researchers can observe the transition more clearly. The concept matters because it shows that a model’s internal algorithm can shift even when surface metrics like training loss continue to improve.
Why it matters for model training
The term is useful because it highlights a failure mode in optimisation, not just a static model property. A system can appear to have learned a useful rule, only to lose that behaviour later as gradient updates push it toward a less general solution.
That matters when practitioners rely on a test peak as proof of stable competence. If a model is checkpointed too late, trained too long, or fine-tuned without careful validation, the best generalising state may already have passed. Ungrokking therefore reinforces the need to watch validation trajectories over time, not only final training outcomes.
For security and reliability work, the broader lesson is that hidden behavioural regressions can appear after apparently successful training phases. A model that looked dependable at one checkpoint may become less robust, less interpretable, or more brittle later in the optimisation process.
How ungrokking differs from ordinary overfitting
Ungrokking is related to overfitting, but it is not just a synonym for “the model memorised the data.” In many ungrokking cases, the model first finds a compact, generalising circuit and only later shifts toward a different representation that fits the training set more directly.
That distinction matters because the model may temporarily generalise well before degrading, which can mislead evaluation if only one training epoch or checkpoint is inspected. Ordinary overfitting often implies a steady divergence; ungrokking emphasises a reversal after a seemingly successful phase.
It also differs from simple noise or random training instability. The phenomenon is usually framed as an optimisation path issue, where the continued search for lower training loss changes the learned computation itself.
What practitioners should watch for
When a model shows strong early generalisation, treat that checkpoint as a candidate state worth preserving and comparing against later ones. Late-stage training can quietly undo earlier gains, especially when the task is small, the data is narrow, or the optimisation path keeps finding sharper solutions.
Common misunderstanding: a lower training loss does not guarantee that the model is becoming more useful. For ungrokking-prone runs, practitioners should evaluate held-out performance, robustness, and behavioural consistency across multiple checkpoints rather than assuming the final model is the best model.
Practitioner takeaway: ungrokking is a reminder to manage training as a search over multiple candidate behaviours, not as a one-way march toward improvement.
