Lies-in-the-loop attacks expose trust gaps in AI coding assistants

At a glance

What this is: This is Checkmarx Zero’s analysis of Lies-in-the-Loop attacks, where malicious code is shaped to deceive AI coding assistants and their human reviewers.

Why it matters: It matters because AI-assisted development now sits inside software supply chain governance, and identity, provenance, and trust controls must account for manipulated feedback loops as well as code quality.

👉 Read Checkmarx’s analysis of Lies-in-the-Loop attacks in AI-assisted development

Context

Lies-in-the-Loop attacks exploit a basic governance weakness in AI-assisted development: systems that learn from feedback can be steered by misleading signals. In practice, that means the assistant may approve behaviour that looks safe in one context but becomes malicious in another, while developers are nudged toward trusting the AI’s explanation. The primary issue is not hallucination alone. It is the collapse of trust validation in a workflow that assumes human review is a reliable backstop.

For identity and access programmes, the relevance is indirect but real. AI coding assistants, internal agents, and developer workflows increasingly act like software identities that influence code acceptance, remediation, and policy decisions. That makes provenance, traceability, and privilege boundaries part of the same governance problem. In other words, if an assistant can shape what gets approved, it has become part of the decision path and must be treated accordingly.

Key questions

Q: How should security teams use AI-assisted code review safely?

A: Use it as a triage layer that accelerates first-pass detection, then require a separate validation step for findings that affect access control, authentication, secrets, or release gating. The safest pattern is hybrid review, where deterministic analysis and human judgement backstop the model’s reasoning.

Q: Why do human reviewers still miss malicious code when AI assistants are involved?

A: Because the assistant can provide a plausible explanation that makes unsafe code look normal, and reviewers often trust that explanation under delivery pressure. The reviewer is no longer examining code in isolation. They are reacting to a shaped narrative, which creates a trust exploit rather than a simple scanning failure.

Q: What breaks when AI assistants learn from misleading developer feedback?

A: The feedback loop starts to reinforce the attacker’s framing, so the assistant becomes more likely to approve the same pattern again. Over time, the organisation trains its own tooling to normalise risky behaviour. That weakens both review quality and the ability to detect repeated deception in later builds.

Q: What should teams do when AI-assisted code review and supply chain risk overlap?

A: They should tighten provenance checks, restrict dependency reach, and require security review for packages or changes that could alter build behaviour. The important question is not whether the code looks safe once. It is whether the artefact can later behave differently and expand its blast radius after approval.

Technical breakdown

How Lies-in-the-Loop attacks manipulate context and feedback

A Lies-in-the-Loop attack uses code, dependencies, or runtime cues that appear benign during review but change behaviour when specific conditions are met. The attacker is not only hiding malicious logic. They are also shaping the signals an AI assistant uses to judge safety, especially if the assistant incorporates prior feedback, contextual memory, or explanation-based reinforcement. That creates a circular trust problem: the assistant learns from the very interaction the attacker has already influenced. Static scans may miss the context switch, and human reviewers may defer to the assistant’s confidence rather than the code’s actual behaviour.

Practical implication: security teams need context-aware validation that does not rely on AI feedback as evidence of safety.

Why human-in-the-loop does not equal secure-by-design

Human-in-the-loop controls help only when the human has independent grounds to challenge the system. In these attacks, the AI’s explanation can be persuasive enough to override doubt, so the human becomes part of the deception chain rather than the control. That matters because review quality depends on time, expertise, and friction, and attackers exploit all three. The result is a governance failure, not just a tooling failure. The workflow rewards fast acceptance of plausible output, even when the underlying artefact behaves differently under hidden conditions.

Practical implication: teams should separate explanation from approval and require independent verification for AI-recommended code changes.

Package blast radius and provenance checks for AI-assisted development

The safest response is to narrow the impact of any single dependency, model suggestion, or agent recommendation. Package blast radius analysis looks at what else a component can reach, modify, or influence if it turns out to be malicious. Provenance checks add traceability so the team can see where the artefact came from, how it changed, and who approved it. Together, these controls reduce the chance that a deceptive package survives review and then propagates into downstream builds or production workflows. This is especially relevant where AI assistants are embedded directly into developer tooling.

Practical implication: restrict dependency reach, verify source lineage, and gate high-risk changes on independent security checks.

Threat narrative

Attacker objective: The attacker wants malicious code to be accepted by both the AI assistant and the human reviewer so it can move into production unnoticed.

1. Entry occurs when the attacker introduces a seemingly benign package or code path that behaves differently under hidden runtime conditions.
2. Credential access is not the main objective here; instead, the attacker gains trust access by persuading the AI assistant and developer that unsafe behaviour is acceptable.
3. Impact occurs when the malicious code is approved, committed, and allowed to progress into the software supply chain and downstream environments.

NHI Mgmt Group analysis

Lies-in-the-Loop is a software trust failure, not just an AI safety issue. The attack works because review systems assume that an assistant’s confidence, explanation, or feedback loop reflects actual safety. That assumption breaks when the artefact is designed to behave differently under specific conditions. For practitioners, the lesson is to treat AI-assisted approval as a risk signal, not as assurance.

Human-in-the-loop controls can become a false sense of control when the human is fed manipulated evidence. The article shows that developers can be nudged into rubber-stamping unsafe code if the assistant’s explanation is convincing enough. That turns review into part of the attack surface. The control gap is not human presence. It is independent verification under conditions the attacker cannot predict.

Package blast radius is the right named concept for this threat class. A malicious dependency is dangerous not only because of what it contains, but because of what it can influence if accepted into a build or agent workflow. Limiting the spread of untrusted components is more useful than assuming a review step will catch every deception. Practitioners should map where AI recommendations can alter downstream privilege, code paths, or release decisions.

Agentic development workflows are creating a new governance layer above traditional code scanning. Static and dynamic analysis remain necessary, but they are no longer sufficient when the decision-maker can be manipulated at the interaction layer. That means organisations need provenance, policy, and approval controls that sit alongside code inspection. The field is moving from vulnerability detection to decision integrity, and that shift is now unavoidable.

What this signals

Package blast radius is becoming a governance metric, not just a supply chain concern. When AI assistants participate in review and remediation, the question is no longer whether a package is malicious in isolation. The question is how far a deceptive artefact can travel before its behaviour is challenged. That pushes teams toward provenance, dependency scoping, and approval separation as core programme controls, not optional hardening.

A practical response is to connect secure development workflows to identity and access boundaries. If an assistant can influence what gets merged or promoted, then the model context, prompt sources, and build permissions all need auditability. The standard to borrow here is least privilege for decision paths, not only for runtime systems. See OWASP Agentic AI Top 10 and the broader AI governance direction in the NIST AI Risk Management Framework.

The deeper shift is from code scanning to decision integrity. Teams will need to prove not only that a dependency was scanned, but that the path from evidence to approval was resistant to manipulation. That is a different operating model, and the organisations that treat it as a tooling tweak will keep absorbing the same class of failure.

For practitioners

• Implement independent approval gates for AI-assisted code changes Require high-risk packages, dependency updates, and security-sensitive refactors to pass a separate human review that does not rely on the assistant’s own explanation or confidence score. This reduces the chance that manipulated feedback becomes the deciding evidence.
• Add package blast radius analysis to dependency review Map what a new package can reach, modify, or influence before it is merged. Focus on build pipelines, secrets exposure paths, and release permissions so a deceptive dependency cannot quietly widen its access.
• Treat AI output as untrusted input Validate any assistant recommendation against source code, provenance records, and runtime behaviour. Use security tests that are designed to catch context-dependent payloads, not just obvious static indicators.
• Track provenance for models, prompts, and dependencies Record which model, prompt context, dependency version, and review path influenced each code decision. That audit trail is essential when you need to explain why a risky change was accepted and whether the same pattern is recurring.

Key takeaways

• Lies-in-the-Loop attacks exploit the trust relationship between AI assistants and human reviewers, which means review workflows themselves are part of the attack surface.
• The core failure is not hallucination alone, but manipulated approval logic that can allow deceptive dependencies to move through the software supply chain.
• Practitioners should pair AI-assisted development with independent verification, provenance tracking, and blast-radius controls before risky code is accepted.

Key terms

• Lies-in-the-Loop attack: A Lies-in-the-Loop attack is a deception pattern that uses misleading code, dependencies, or feedback signals to persuade an AI assistant and human reviewer that unsafe behaviour is acceptable. The goal is to corrupt the decision process, not just hide malicious logic in source code.
• Package blast radius: Package blast radius is the amount of downstream code, build logic, credentials, or production behaviour a dependency can influence if it is accepted into a workflow. Limiting blast radius reduces the damage caused by a malicious or compromised package and makes review failures less consequential.
• Decision Integrity: Decision integrity is the degree to which an organisation’s outputs remain consistent, explainable, and aligned to approved meaning. In data and AI programmes, it depends on controlled definitions, trusted sources, and the ability to prevent different tools from inventing conflicting interpretations.
• Human-in-the-Loop (HITL): A governance pattern requiring human approval before an AI agent takes high-impact, irreversible, or out-of-scope actions. HITL is a critical control for agentic AI identity governance.

What's in the full report

Checkmarx's full article covers the operational detail this post intentionally leaves for the source:

• The Claude Code experiment design, including how the malicious package behaved differently across runtime contexts.
• The specific feedback cues that caused the AI assistant to misclassify unsafe behaviour as acceptable.
• The developer interaction pattern that allowed the assistant’s explanation to override human concern.
• The vendor’s proposed workflow safeguards for in-IDE scanning, remediation guidance, and blast-radius analysis.

👉 Checkmarx’s full post covers the Claude Code experiment, the deception pattern, and the workflow controls it tested.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and identity controls that underpin secure access decisions. It is designed for practitioners who need to connect identity governance to modern development and runtime risk.