LLM platform abuse is exposing gaps in AI assistant security

At a glance

What this is: This is an analysis of LLM platform abuse and how malicious proxy ecosystems are being used to bypass controls around AI assistants, scrape prompts and enable phishing and impersonation.

Why it matters: It matters because identity, access and abuse controls now have to account for AI-enabled fraud paths that sit outside traditional human IAM, NHI and bot defence assumptions.

👉 Read Arkose Labs' analysis of LLM platform abuse and AI assistant proxies

Context

LLM platform abuse is the misuse of generative AI assistant infrastructure to bypass controls, hide attacker activity and scale fraud. For identity teams, the issue is not just content abuse. It is the way malicious operators exploit access paths, proxy layers and service boundaries that were never designed to verify intent at runtime.

The practical problem is broader than one product category. If attackers can stand up illegal reverse proxy services, scrape prompts and reuse AI assistant capabilities for phishing or deepfake creation, then conventional abuse detection is only seeing the surface of the threat. That pushes the issue into IAM, NHI governance, bot management and fraud operations at the same time.

Key questions

Q: How should security teams detect LLM platform abuse across proxy networks?

A: Teams should look for correlated behavioural signals rather than single indicators. Repeated challenge failures, unusual origin changes, country hopping, bursty request patterns and inconsistent device behaviour are all signs of proxy-mediated abuse. The goal is to detect the abuse service, not just the individual session.

Q: Why do AI assistant platforms create new fraud risks for identity teams?

A: AI assistant platforms can be abused as infrastructure for phishing, impersonation and prompt harvesting. That matters because the access path itself becomes the attack surface, so identity and abuse controls have to evaluate intent, session behaviour and downstream misuse together.

Q: What do security teams get wrong about prompt scraping?

A: They often treat prompt scraping as a model issue only. In practice, it is also a data exposure and service-abuse problem because attackers can extract operational context, reuse outputs and turn those assets into illegal tooling or social engineering content.

Q: Who should own response when LLM abuse becomes a phishing channel?

A: Ownership should sit across fraud, IAM, security operations and bot management, with clear escalation paths. If AI assistant abuse can produce phishing, impersonation or deepfake material, then it is no longer a niche model risk. It is an enterprise abuse and identity governance issue.

Technical breakdown

How malicious reverse proxies hide LLM platform abuse

The article describes a proxy ecosystem that sits between users and AI assistants, masking origin, distributing requests across multiple countries and helping operators evade geo-restrictions and tracking. In practice, this is not just traffic shaping. It is infrastructure for concealment, allowing attackers to industrialise access to LLM services while making attribution and takedown more difficult. The abuse path also creates a secondary market where one group builds illegal access services and another consumes them for phishing, scraping or deepfake generation. The result is a layered fraud supply chain rather than a single attacker session.

Practical implication: Security teams need controls that inspect proxy behaviour, request patterns and abuse clusters, not just account-level access events.

Why prompt scraping turns AI assistants into attack infrastructure

Prompt scraping is the extraction of conversation content, operational instructions or model outputs at scale so it can be reused or monetised elsewhere. In this case, scraped prompts help build a commercial service that imitates LLM access while bypassing provider constraints. That changes the threat model from isolated misuse to systematic theft of interaction data and service capacity. Once prompts and outputs are harvested, attackers can package them into illegal tooling, improve evasion, or use the data to create more convincing impersonation and phishing material. The abuse is therefore both exfiltration and enablement.

Practical implication: Teams should treat prompt activity as sensitive operational data and monitor for scraping patterns that indicate service abuse.

How AI-resistant challenges and workflow anomaly detection disrupt abuse

Arkose Labs describes a defence stack combining bot detection, workflow anomaly detection, API instrumentation and AI-resistant challenges. The important mechanism is not the branding of the control, but the fact that LLM platform abuse spans multiple signals: device behaviour, network anomalies, challenge response quality and unusual request sequencing. When those signals are correlated, defenders can distinguish legitimate assistant usage from automated abuse services and proxy relays. This matters because generic rate limiting often misses distributed abuse, while static challenges are increasingly solvable by machine vision and AI solvers. Defences therefore need to adapt to both automation and model-assisted evasion.

Practical implication: Instrument assistant access for behavioural and API-level anomalies, then tune response controls to break proxy-driven abuse chains.

NHI Mgmt Group analysis

LLM platform abuse is becoming a fraud supply chain, not a one-off misuse problem. The article shows two attacker classes: those who build proxy infrastructure to bypass restrictions and those who consume that infrastructure for phishing and other downstream abuse. That means defenders are no longer dealing with a single malicious session, but with a reusable ecosystem that can be sold, repackaged and redirected. The practitioner implication is that abuse controls must target the service layer and the distribution layer, not only the user interaction layer.

Generic bot controls are not enough when attackers operate across countries and proxy layers. The article’s proxy stack uses decentralised hosting and traffic distribution to hide behaviour from normal perimeter and abuse analytics. That reveals a category-level problem for identity security teams: visibility breaks down when the abusive actor is not tied to one account, one geography or one fixed infrastructure endpoint. The practitioner implication is to correlate request behaviour, origin variance and challenge outcomes across the full assistant access path.

AI assistant misuse is forcing IAM, fraud and bot management into the same control plane. The harmful actions described here include impersonation, spear phishing and prompt scraping, which means the security boundary is no longer just authentication or bot mitigation. The field now has to treat AI assistant access as a governed identity channel with fraud consequences. The practitioner implication is to align abuse response, identity governance and content-risk controls around the same telemetry.

Prompt scraping is the named concept that best explains this threat pattern. The article describes attackers extracting prompts to build illegal services and to fuel downstream abuse. That turns assistant interactions into a harvestable asset, not a disposable transaction. The practitioner implication is to recognise prompt scraping as a governance and data-handling problem, not only a model-security concern.

From our research:

• 85% of organisations lack full visibility into third-party vendors connected via OAuth apps, according to The State of Non-Human Identity Security.
• Lack of credential rotation is cited as the top cause of NHI-related attacks by 45% of organisations, according to the same research.
• For a broader governance baseline, Top 10 NHI Issues helps teams prioritise the controls most likely to fail first.

What this signals

Prompt scraping is the governance concept teams should start using. Once attackers can extract prompts, the assistant becomes a reusable source of operational and social-engineering material, not just a user interface. That means programme owners should begin tracking prompt visibility, abuse clustering and downstream misuse as part of the same control story.

The visibility problem is already familiar in adjacent identity domains. 85% of organisations lack full visibility into third-party vendors connected via OAuth apps, a pattern that should worry teams extending AI assistants into production workflows, because blind spots scale faster than controls do.

Practitioners should expect AI assistant abuse to be measured in business misuse, not only security telemetry. As that happens, bot defence, IAM and fraud teams will need shared escalation criteria and a common view of abuse lifecycle, especially where proxy infrastructure obscures attribution.

For practitioners

• Instrument AI assistant traffic for proxy behaviour Correlate IP rotation, country hopping, request bursts and repeated challenge failures so you can separate legitimate usage from distributed abuse services.
• Treat prompts as sensitive operational data Log and review unusual prompt extraction patterns, especially where repeated queries suggest scraping, abuse automation or model-output harvesting.
• Link bot management to fraud response Route AI assistant abuse signals into fraud, IAM and threat operations workflows so phishing, impersonation and deepfake preparation are handled as one problem.
• Test challenge resilience against machine solvers Validate that visual and workflow challenges still resist computer vision and AI-assisted bypass attempts, not just basic automation.

Key takeaways

• LLM platform abuse is an identity and fraud problem as much as it is a model-abuse problem, because proxies and scraping turn AI assistants into reusable attacker infrastructure.
• The article shows that attackers can monetise AI assistant access through decentralised services, which makes visibility, attribution and takedown materially harder for defenders.
• Teams need to govern assistant access as a behavioural channel, with telemetry that connects proxy use, prompt extraction and downstream fraud activity.

Key terms

• LLM Platform Abuse: The misuse of large language model services or assistant platforms to support fraud, phishing, scraping or other malicious activity. It usually involves abusing legitimate access paths, proxy infrastructure or automation so the attacker can hide origin, scale operations and reuse assistant capabilities for downstream harm.
• Prompt Scraping: The large-scale extraction of prompts, responses or interaction data from AI systems for reuse, resale or abuse. In practice, it turns assistant conversations into harvestable assets and can expose operational context, sensitive instructions or model outputs that attackers use to improve evasion and social engineering.
• Reverse Proxy Abuse: The use of proxy services to mask the source of traffic while routing requests through distributed infrastructure. For identity and fraud teams, this matters because origin-based controls, attribution and takedown become less effective when attackers can rotate locations and hide behind intermediary services.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.

This post draws on content published by Arkose Labs: A New Threat Vector Emerges. Read the original.