When a tool, MCP server, agent framework package or model provider your agents depend on is compromised, the attacker inherits whatever your agents can reach: their credentials, their data and the hosts they run on. Agent supply chain incidents differ from a classic dependency compromise in three ways: the component often holds live API keys or OAuth tokens on your behalf, it may run on developer laptops as a local MCP server rather than in a managed pipeline, and it can instruct the agent as well as steal from it. This guide gives incident responders, platform and IAM teams a response sequence for the four common cases, with the triage questions, evidence, credential rotation scope and detections for each. For prevention see the AI Supply Chain and AI-BOM Guide; for a fill-in template see the MCP Incident Response Playbook.
Key takeaways
- Scope by credential, not by package. The blast radius is every secret the compromised component or its host could read, so the rotation list is usually far longer than the list of affected installs.
- Removing it upstream does not remove it from you. postmark-mcp kept forwarding email from installed copies after npm removed it, and deleting an attacker's exfiltration repository does not revoke what was in it.
- Local MCP servers are endpoints. They run with the user's privileges, so a malicious one is a malware incident on a developer machine plus a credential incident in every system that user can reach.
- Hosted platforms need their own path. When an MCP hosting service or model provider is breached, you cannot inspect their hosts: rotate what you gave them and watch for use of it.
- Your inventory decides your speed. Teams that already know which agents use which tools, versions and credentials contain in hours; teams that do not spend days finding installs.
The four incident types
| Type | Example | What the attacker gains | First containment move |
|---|---|---|---|
| Malicious or hijacked MCP server or agent tool | postmark-mcp Malicious MCP Server 2025: version 1.0.16 silently BCC'd every email to an attacker | Data passing through the tool, the credential configured for it, and the ability to inject instructions into tool results | Block or remove the server in every client configuration and gateway; pin to the last known-good version or the publisher's official server |
| Compromised agent framework or AI library package | LiteLLM PyPI Package Breach 2026: versions 1.82.7 and 1.82.8 stole cloud, Kubernetes and LLM API keys | Code execution on every host that installed it, and everything those hosts can read | Isolate affected hosts and CI runners; block the versions in your package proxy |
| Supply chain malware that abuses local AI tools | Nx s1ngularity Attack 2025: a post-install script prompted AI CLIs with permission-bypass flags to search for secrets | Developer secrets found with the help of the user's own AI agent | Treat developer machines as compromised; revoke tokens before analysis completes |
| Breach of a hosting platform or provider | Smithery.ai MCP Hosting Breach 2025: one fly.io token controlled an organisation of 3,243 apps, mostly hosted MCP servers | Keys and traffic you sent to the hosted servers or provider | Rotate every key passed to the platform; move to scoped, short-lived credentials |
These map to OWASP ASI04 Agentic Supply Chain Vulnerabilities and LLM03:2025 Supply Chain, and to MITRE ATLAS AML.T0010 AI Supply Chain Compromise (sub-techniques AML.T0010.001 AI Software, AML.T0010.003 Model, AML.T0010.005 AI Agent Tool) and ATT&CK T1195.001 Compromise Software Dependencies and Development Tools.
Triage: the first hour
- Confirm the advisory: read the maintainer's or provider's own statement and any GitHub Security Advisory or OSV entry. The OpenSSF malicious-packages repository publishes malicious package reports in OSV format. Record the affected versions, dates and indicators exactly.
- Find every install: search lockfiles, container images, CI caches, package proxy download logs, MCP client configuration files on developer machines (for example the JSON files where desktop and IDE clients list MCP servers) and any MCP gateway or registry you run.
- Fix the exposure window: from the first malicious version's publish time to the time you blocked it, per host. Use installation timestamps, not today's date.
- List the credentials in reach: for each affected host or tool, every environment variable, config file, keychain entry, cloud role and OAuth grant it could read. For a tool, include the credential configured for it and any secrets that passed through it.
- Decide containment: block versions, remove servers, isolate hosts and start rotation in parallel. Do not wait for full forensics before revoking high-value keys.
- Open supplier contact: contact the maintainer or provider and ask for indicators, timeline and whether the credentials you gave them were accessed.
Credential rotation scope
Most of the effort in these incidents is rotation. Use the table to set scope by incident type.
| Credential | Rotate when | Notes |
|---|---|---|
| API key or token configured for the compromised tool or MCP server | Always | For postmark-mcp this meant Postmark API and SMTP credentials |
| Secrets that passed through the tool | When the tool saw message bodies, files or records | Emails, tickets and documents often contain keys; scan and rotate those too |
| Cloud credentials on affected hosts | Package or local server ran code on the host | AWS, Azure and Google Cloud keys, plus any role the host can assume |
| Kubernetes, database and SSH credentials | Same | LiteLLM's malware collected kubeconfig, SSH keys and database passwords |
| LLM provider API keys | Same, or when a provider or gateway is breached | Check provider usage for spend or model calls you did not make |
| Source control and package publishing tokens | Developer machine or CI runner affected | GitHub, npm and PyPI tokens; move publishing to OIDC trusted publishing. See the npm Trusted Publishing and Provenance Guide |
| OAuth refresh tokens and user grants held by the agent | Agent runtime affected or tool could read tokens | Revoke grants and refresh tokens; access tokens may stay valid until expiry |
| Keys passed to a hosted MCP platform | Hosting platform breached | Everything you configured there before the fix date |
Run rotation from a list with owners and a verification step: confirm the old credential fails, not just that a new one exists. The Secrets Rotation Runbook Template gives a structure.
Evidence to collect
- Package evidence: lockfiles, package proxy logs, the malicious artefact itself (hash it and store it offline), and the advisory text with its retrieval time.
- Host evidence: process and network telemetry from EDR for affected hosts, new persistence (LiteLLM's malware used a systemd backdoor under
~/.config/sysmon/and alitellm_init.pthlauncher), and outbound connections to the indicator domains. - Agent evidence: tool call logs for the compromised tool, including arguments and results, so you can see what data it received and whether its responses carried instructions.
- Identity evidence: sign-in and API logs for every credential on the rotation list, from the start of the exposure window, to find use from unfamiliar IPs or user agents.
- Configuration evidence: MCP client configuration history and gateway allow lists, to establish when the server was added and by whom.
Detections during and after the incident
| Signal | Source | Why it matters |
|---|---|---|
| Install or download of a blocked package version | Package proxy, EDR, CI logs | Re-introduction through a cache or transitive dependency |
| AI CLI started with permission-bypass flags from a script, not a terminal | EDR process command lines | The Nx malware used --dangerously-skip-permissions, --yolo and --trust-all-tools |
| New MCP server entry in a client configuration file | EDR file monitoring or configuration management | Malicious startup commands are a documented local MCP risk in the MCP security best practices |
| Use of a rotated credential | Cloud, SaaS and provider logs | Shows the attacker held it and tells you where they tried it |
| Tool output that contains instructions or unexpected recipients | Agent tool call logs or gateway inspection | A compromised tool can steer the agent, not only steal from it |
| Unusual LLM provider spend or model calls | Provider usage and billing data | Stolen AI keys are commonly resold or used for free compute |
See the Detection Engineering for Non-Human Identities Guide for building credential-use detections.
Eradication and recovery
- Rebuild affected CI runners and, where persistence was found, developer machines from known-good images.
- Reinstall the component at a verified clean version or switch to the publisher's official build; pin by version and hash.
- Re-issue credentials with narrower scopes and shorter lifetimes than before. The MCP specification forbids token passthrough and recommends progressive, least-privilege scopes, so give each server its own credential issued for it.
- Re-enable agents one at a time, with tool call logging on and an alert on the indicators.
- Clean agent memory or context stores if the tool could write to them.
- Keep blocking indicators for at least 90 days and watch for use of rotated credentials.
Decision guidance
- Rotate everything or rotate selectively: if malicious code ran on a host, rotate everything that host could read. Selective rotation is defensible only when the component was a remote tool with no code on your hosts and logs show what it received.
- Block the tool or keep the agent running: blocking one MCP server usually leaves the agent useful. Stopping the whole agent is right when the compromised component is the framework or runtime itself.
- Small organisations: keep a single list of approved MCP servers and AI packages with versions, and the credential each uses; it turns a multi-day search into a lookup.
- Large organisations: route MCP traffic through a gateway that can block a server centrally, and mirror packages through a proxy with a block list. See the MCP Gateway and MCP Server Governance Guide.
Measuring readiness and response
- Time to locate all installs of a named component across hosts, images and MCP client configurations.
- Time from advisory to last credential rotated and verified.
- Share of agent tools with a recorded owner, version and credential in the inventory.
- Share of MCP servers using per-server, scoped credentials rather than shared or static keys.
- Use of rotated credentials detected after rotation (should fall to zero; any event is a lead).
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV.SC-08, RS.AN-03, RS.CO-03 | Suppliers included in response; root cause analysis; information sharing |
| NIST SP 800-53 Rev. 5 | IR-4, IR-6(3), SR-3, SR-8, SI-7, CM-7(5) | Incident handling, supply chain coordination, notification agreements, integrity, allow-listed software |
| NIST SP 800-161 Rev. 1 | C-SCRM practices | Supplier risk management context for agent components |
| OWASP | ASI04; LLM03:2025 Supply Chain | Risk definitions for agentic and LLM supply chains |
| MITRE ATLAS | AML.T0010, AML.T0010.005, AML.T0083, AML.T0098 | AI supply chain compromise; credentials from agent configuration; tool credential harvesting |
| MITRE ATT&CK | T1195.001, T1195.002, T1552.001 | Dependency and software supply chain compromise; credentials in files |
Common failure patterns
- Waiting for the registry: assuming removal from npm or PyPI ends the incident. Installed copies keep running, as postmark-mcp showed.
- Rotating only the obvious key: rotating the tool's own API key but not the secrets that passed through it or sat on the same host.
- Missing developer machines: local MCP servers and AI CLIs sit outside CI and server inventories, which is where Nx-style malware looks for secrets.
- Trusting the exfiltration takedown: GitHub removing exfiltration repositories did not revoke the 2,349 secrets GitGuardian counted in the Nx case.
- Static, shared tool credentials: one key reused by several MCP servers or agents means one compromise forces many rotations and outages.
- Ignoring transitive dependencies: agent frameworks and MCP plugins pull in AI libraries such as LiteLLM without a direct entry in your manifest.
Practitioner checklist
- Keep an inventory of agent tools, MCP servers and AI packages with version, host, owner and the credential each uses.
- Subscribe to GitHub Security Advisories and OSV data for the AI packages you run, including malicious package reports.
- Pre-write the rotation list template by incident type and assign owners.
- Be able to block a package version in your proxy and an MCP server in your gateway within one hour.
- Collect MCP client configuration files and AI CLI command lines in EDR on developer machines.
- Give every MCP server its own scoped, short-lived credential; no shared or passed-through tokens.
- Verify each rotation by confirming the old credential fails, and alert on any later use of it.
- Rehearse one agent supply chain scenario a year with the platform, IAM and developer tooling teams.
Standards and references
- Model Context Protocol: Security Best Practices
- OWASP: Top 10 for LLM Applications 2025 (LLM03 Supply Chain)
- OWASP GenAI Security Project: Top 10 for Agentic Applications
- MITRE ATLAS: AI Supply Chain Compromise
- MITRE ATT&CK: Compromise Software Dependencies and Development Tools
- NIST: SP 800-161 Rev. 1 Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations
- NIST: SP 800-53 Rev. 5
- OpenSSF: Malicious Packages repository
Related NHI Mgmt Group resources: MCP Incident Response Playbook · Local MCP Server Security Guide · Third-Party Incident Response Playbook · Model Supply Chain Guide · Revoking Delegated Access During an Active Agent Incident