Catastrophic Forgetting

Expanded Definition

Catastrophic forgetting is the loss of previously learned capability when a model is retrained too aggressively on new data. In NHI and agentic AI settings, it matters because an agent that loses prior task competence can begin failing established workflows, policy checks, or tool-use patterns after a fine-tuning cycle.

The concept is related to model drift, but it is not the same thing. Drift usually describes changes in input data, user behavior, or model outputs over time. Catastrophic forgetting describes a specific training failure: new learning overwrites older knowledge instead of layering on top of it. Definitions vary across vendors when this shows up in continual learning, adapter tuning, or retrieval-augmented pipelines, so NHI Management Group treats it as a governance issue as much as a modeling issue. A useful external reference point is the NIST Cybersecurity Framework 2.0, which emphasizes controlled change and risk management across system lifecycles. The most common misapplication is assuming a successful retraining run preserves all prior capability, which occurs when teams validate only the new task and skip regression testing on legacy behaviors.

Examples and Use Cases

Implementing continual learning rigorously often introduces testing and change-control overhead, requiring organisations to weigh faster adaptation against the cost of regressions in older tasks.

• An internal support agent is fine-tuned on a new product line and then starts misrouting tickets for legacy products because older classification patterns were overwritten.
• A code-generation agent learns a new framework update but begins producing insecure legacy authentication flows, which should have remained suppressed by prior safety tuning.
• A document-processing model adapts to a new contract template and loses accuracy on established clause extraction workflows used by finance and legal teams.
• A tool-using AI agent receives domain-specific reinforcement and then forgets approved escalation paths, creating brittle behavior in production automation.

These failures are easier to detect when organisations preserve checkpoints, run pre/post-training regression suites, and compare outputs against a stable baseline. For NHI programs, the Ultimate Guide to NHIs is useful because it frames model and identity governance together: a retrained agent is only safe if its permissions, secrets, and action boundaries remain intact. The same lifecycle discipline recommended by NIST Cybersecurity Framework 2.0 applies here, especially where training changes affect operational trust. In practice, teams use this term when deciding whether a new fine-tune should be merged, rolled back, or isolated as a separate model version.

Why It Matters in NHI Security

Catastrophic forgetting turns model maintenance into an identity risk because a degraded model may still hold the same tool access, secrets reach, and automation privileges as before. In other words, the business impact is not limited to lower accuracy. It can become a control failure when an agent that used to honor constraints no longer remembers them after retraining. That is why governance must include versioned checkpoints, regression validation, rollback paths, and clear approval gates for any model that can act on behalf of an organisation.

This matters especially in environments already struggling with NHI sprawl. NHI Mgmt Group reports that only 5.7% of organisations have full visibility into their service accounts, which means a retrained agent can become harder to monitor precisely when its behavior changes. When model forgetting affects authentication, orchestration, or access decisions, the issue crosses from ML quality into operational security. Organisations typically encounter the consequences only after a broken release, failed automation, or policy bypass, at which point catastrophic forgetting becomes operationally unavoidable to address.