Attack and Defense Strategies

Attack and defense strategies are the testing methods used to probe model weaknesses and measure whether protections hold. In LLM privacy work, attacks try to extract sensitive information, while defenses aim to reduce that exposure. Evaluating both sides gives a more realistic view of privacy risk than relying on one perspective alone.

Expanded Definition

Attack and defense strategies are the paired methods used to test whether a model or system can be exploited, and whether protections still hold under pressure. In LLM privacy work, the attack side focuses on whether sensitive data can be elicited, reconstructed, or inferred; the defense side focuses on reducing that exposure through controls that change the model’s behavior, output filtering, training, or surrounding governance.

The term is broader than a single red-team exercise. It covers prompt-based extraction, membership inference, memorisation testing, jailbreak-style probing, and defensive techniques such as data minimisation, access controls, redaction, output moderation, and privacy-preserving training. The useful boundary is practical: if a method is meant to reveal weakness, it belongs on the attack side; if it is meant to reduce or contain that weakness, it belongs on the defense side.

Industry usage is still evolving, especially where privacy, model safety, and security overlap. A common misunderstanding is to treat one successful defense test as proof of safety. In practice, the value of this term comes from evaluating both sides together, because a defense that looks strong in isolation may fail under a more realistic attack path.

Examples and Use Cases

Teams use attack and defense strategies in several concrete ways:

• Red teams prompt an LLM with carefully structured queries to see whether it reproduces training data or other sensitive context.
• Privacy engineers compare baseline model output against a hardened version to measure whether safety layers reduce leakage without breaking usefulness.
• Security reviewers test whether retrieval, memory, or tool access creates a path for private content to appear in responses.
• Product teams evaluate whether output filters, policy prompts, or post-processing controls fail open when the model is stressed.
• Researchers use paired tests to compare exposure before and after mitigation, which helps separate genuine improvement from cosmetic change.

In practice, the tradeoff is usually between tighter protection and lower utility. A model that refuses too much can become less usable, while a model that answers too freely can leak more than intended. The point of the strategy pair is to quantify that tension instead of guessing at it.

For background on the attack patterns that can shape this testing, LLMjacking: How Attackers Hijack AI Using Compromised NHIs shows how credential abuse can become part of an AI attack path.

Security Implications

When attack and defense strategies are treated as separate activities, organisations can mistake a narrow test for a real assurance result. That creates false confidence: the system may withstand one class of prompt attack but still leak through memory, retrieval, logs, or downstream integrations.

It also creates blind spots in measurement. If the attack strategy is weak, the defense looks stronger than it is. If the defense is tested only against the obvious attack, it may miss more subtle extraction paths such as indirect prompting, repeated probing, or cross-session leakage. A useful practitioner observation is that privacy weakness often appears as an interaction problem, not a single broken control.

The strongest evidence comes from testing both paths in the same evaluation cycle, then comparing what changed in the model, the guardrails, and the surrounding system. That is the difference between a control that is theoretically present and one that is actually effective under realistic pressure.

For a broader view of real-world identity and credential abuse patterns that can amplify AI exposure, The 52 NHI breaches Report provides case-based context on how compromise paths often compound.

Security, Operational and Governance Implications

This term matters because it describes how privacy assurance is actually earned. A team cannot claim a model is safe simply because it has a policy layer or a moderation tool; the relevant question is whether the defense still holds when an attacker adapts. That makes the term useful for governance, not just testing.

Operationally, the strategy pair should influence how releases are approved, how regressions are detected, and how ownership is assigned across model, application, and data teams. The same test that proves one mitigation works can also show where a different safeguard is needed, such as better retrieval filtering, tighter data handling, or stronger logging boundaries.

For governance, the important point is repeatability. Attack and defense strategies should be revisited as the model, prompts, datasets, and surrounding controls change. If the test conditions are stale, the security conclusion is stale too.

Where privacy testing intersects with adversarial AI technique tracking, MITRE ATLAS adversarial AI threat matrix is a useful reference for mapping attack behaviour to control gaps.

Related resources from NHI Mgmt Group

• Who should own fraud and AI attack defense when bot activity touches identity, application, and security teams?
• What are the signs that a cyber defense program is failing to stop common attack paths?
• What breaks when organisations rely on perimeter defense against autonomous attack chains?
• Why does Agentic AI make NHI attack surface expand so significantly?