Spotify-style API scraping shows where legitimate access turns unsafe

At a glance

What this is: This is a research blog on Spotify-style large-scale API scraping and why legitimate API behaviour can become an extraction channel at scale.

Why it matters: It matters to IAM and security teams because authenticated access, client trust, and rate-limited APIs can still enable bulk data extraction when identity and usage governance do not account for aggregation.

By the numbers:

• The Anna’s Archive group scraped metadata for 256 million tracks and 186 million unique ISRCs, covering an estimated 99.9% of Spotify tracks at the time of collection.
• The group archived approximately 86 million music files, representing around 99.6% of total listens.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.

👉 Read Equixly's analysis of Spotify-style API scraping and abuse-aware testing

Context

API scraping becomes a security problem when legitimate, authenticated requests are used to reconstruct datasets at a scale the business never intended. The core issue is not a broken endpoint in isolation, but a governance gap between allowed per-request behaviour and unsafe aggregate extraction, especially where identity, client trust, and automation intersect.

In IAM terms, this sits adjacent to non-human identity governance because automated accounts, connected applications, and scripted clients can all exploit the same permissions model that was designed for ordinary users. The article is about Spotify, but the operating pattern is typical across large platforms with enumerable APIs, permissive client assumptions, and limited controls on cumulative data access.

Key questions

Q: Where does API security fail in practice when requests are individually valid?

A: It fails when the security model stops at the request boundary and ignores what can be reconstructed across many valid calls. A single authorised request may be harmless, but millions of them can reveal a complete dataset. The control gap is aggregation risk, which requires testing traversal, coverage, and cumulative extractability, not just authentication and input validation.

Q: Why do rate limits and bot controls often miss large-scale scraping?

A: Because many controls are built to detect intensity, not coverage. Attackers can spread requests across accounts, IPs, devices, and time windows while staying inside expected workflows. If the defence only watches for spikes or malformed clients, it will often detect the problem after meaningful data has already been extracted.

Q: How can security teams tell whether an API is enabling large-scale scraping?

A: Look for sequential request patterns, repeated access across related endpoints, high-frequency lookups from the same source, and unusually broad traversal of records. If those behaviours can assemble a meaningful identity dataset faster than expected, the endpoint is enabling extraction. Rate limits alone are not enough if the returned metadata remains highly joinable.

Q: What should teams do when legitimate automation becomes an extraction channel?

A: Tighten the access scope of connected apps, service accounts, and scripted clients, then add controls that constrain lifetime extraction and dataset reconstruction. If the business must support automation, separate ordinary usability from bulk-access paths and monitor the total value moving through them. That is how you reduce silent abuse without breaking normal operations.

Technical breakdown

Why individually valid API calls become unsafe in aggregate

A single API request can be authenticated, authorised, and correctly formed, yet still participate in abuse when repeated millions of times. The failure mode is aggregation, where small, permitted responses combine into a dataset that was never meant to be reconstructable. This is why conventional checks such as authentication, schema validation, and per-request authorisation do not solve the problem on their own. The security question shifts from “is each call allowed?” to “what can be extracted over time across accounts, IPs, and workflows?”

Practical implication: Test extractability across workflows, not just endpoint validity.

How scraping operations evade intensity-based detection

Many bot and anti-scraping controls watch for spikes, malformed clients, or high request rates from a single source. Large-scale scrapers avoid those triggers by spreading traffic across accounts, devices, regions, and time windows, while staying inside documented workflows. They may use residential proxies, fingerprint variation, headless browsers, and low-and-slow traversal to keep each request looking ordinary. That makes the system appear healthy until the data has already been lifted in volume.

Practical implication: Build controls that score coverage and lifetime extraction, not only request bursts.

What abuse-aware API security testing actually measures

Abuse-aware API security testing looks at system-level properties that traditional API tests miss. It evaluates whether identifiers are enumerable, whether endpoints can be traversed systematically, whether multiple responses can be aggregated into a complete picture, and whether rate limits only slow attackers rather than bound what they can take. In other words, it treats the API as an economic channel, not just a technical interface. That is the right model when the threat is silent extraction rather than a classic exploit.

Practical implication: Prioritise testing for traversal, aggregation risk, and total extractable value.

Threat narrative

Attacker objective: The attacker’s objective is silent bulk extraction of valuable data while remaining within the appearance of legitimate API use.

1. Entry begins with legitimate access to high-performing APIs or connected applications that return data the caller is authorised to see.
2. Escalation occurs through automation, distributed identities, fingerprint variation, and traversal that turns permitted lookups into large-scale collection.
3. Impact is the reconstruction and export of a dataset that was never intended to be available in bulk, often without triggering obvious security alerts.

NHI Mgmt Group analysis

API abuse without exploits is now a governance problem, not just a bot problem. The article shows that systems can behave correctly at the request level while failing catastrophically at the dataset level. That means IAM and security teams need to think about cumulative access, not only authentication success. The practitioner conclusion is simple: if a caller can lawfully traverse enough surface area, the control failure is upstream of the endpoint.

Coverage-aware risk management is the right concept for large platforms. Traditional controls often optimise for intensity, but the actual loss mechanism here is coverage over time. This is a distinct failure mode because the attacker does not need to break a control, only to stay beneath its alert threshold while continuing to harvest value. The practitioner conclusion is to measure lifetime extractability as a first-class security metric.

Non-human identity governance matters here because automation is the extractor. When bots, scripted accounts, and connected apps can collect data at scale, the access model for non-human identities becomes part of the scraping defence. That does not mean every API abuse case is an NHI problem, but it does mean service accounts, app credentials, and delegated access should be reviewed for cumulative data movement. The practitioner conclusion is to align NHI governance with volume-aware access constraints.

Abuse-aware testing should become part of pre-production assurance. Equixly’s central argument is that testing can expose when correct behaviour becomes unsafe in aggregate. That aligns with a broader security pattern: controls that only observe production alerts are too late for silent extraction. The practitioner conclusion is to validate traversal, aggregation, and detection thresholds before release.

Aggregation risk is the named concept teams should adopt. Aggregation risk is the point at which individually allowed API responses become an exfiltration surface when combined at scale. It is the hidden failure mode behind many scraping incidents because no single call looks suspicious. The practitioner conclusion is to design controls that constrain what can be reconstructed, not just what can be requested.

What this signals

API teams should expect scraping pressure to keep shifting from obvious bot traffic toward distributed, low-and-slow extraction that looks like normal usage until it is too late. The practical response is to instrument coverage, lifetime value, and reconstruction risk alongside rate limits and authentication checks.

Aggregation risk: this is the failure mode that makes valid requests dangerous in aggregate, and it should become part of API design reviews and threat modelling. The strongest control is not just blocking abuse, but reducing what any one identity, app, or workflow can reconstruct over time.

For practitioners

• Test for aggregation risk across API workflows Validate whether individually acceptable responses can be combined into a reconstructable dataset across endpoints, accounts, and sessions. Include traversal tests, pagination abuse, identifier enumeration, and long-run extraction scenarios, not only input validation and auth checks.
• Measure extraction by lifetime coverage Track how much data a single account or a coordinated cohort can recover over time, then set alerting and enforcement thresholds around coverage, not only request rate. This helps expose slow, distributed scraping that evades per-minute limits.
• Review delegated and automated access paths Map connected applications, service accounts, and scripted clients to the data they can repeatedly retrieve, then reduce over-broad access where repeated lookups create reconstruction risk. Pay particular attention to APIs that trust clients to behave like humans.
• Add abuse-oriented API testing before release Extend pre-production testing to simulate distributed accounts, rotating fingerprints, and low-and-slow automation so you can see what a determined scraper can extract without tripping ordinary bot rules. Use the findings to adjust controls before the interface goes live.

Key takeaways

• The core risk is not a broken API call, but a system that becomes unsafe when valid requests are aggregated at scale.
• The evidence points to a repeatable abuse pattern in which automated clients, distributed identities, and permissive endpoints enable silent extraction.
• Teams should test for reconstructability, lifetime coverage, and delegated-access abuse before production traffic does it for them.

Key terms

• Aggregation Risk: Aggregation risk is the chance that one vulnerability, supplier dependency or platform weakness creates losses across many organisations at once. In cyber insurance, it matters because correlated failures can produce claims at a scale that historical loss models struggle to predict.
• Coverage-Based Detection: Coverage-based detection looks at how much of a system, catalogue, or dataset a caller can traverse, not just how fast it is going. It is useful where attackers stay below rate thresholds but still extract value steadily. This approach complements intensity-based controls by measuring lifetime reach and cumulative loss.
• Abuse-Aware API Testing: Abuse-aware API testing evaluates whether a system remains safe when ordinary features are used at scale, across many identities, and in ways that reconstruct hidden value. It focuses on traversal, aggregation, enumerability, and economic extractability. The goal is to identify unsafe behaviour before attackers do.
• Non-Human Identity (NHI): A digital identity assigned to a non-human entity such as a software application, service account, API key, bot, machine, or AI agent that enables it to authenticate and interact with systems without direct human involvement. NHIs now outnumber human identities in most enterprises by 25 to 50 times.

What's in the full report

Equixly's full blog covers the operational detail this post intentionally leaves for the source:

• Step-by-step analysis of the Spotify scraping pattern, including how API responses were reconstructed at scale
• Specific abuse signals that indicate an API is becoming extractable even when each request is valid
• Detailed testing questions for identifying aggregation risk, traversal feasibility, and client trust assumptions
• Practical examples of how coverage-aware protections differ from simple rate limiting

👉 Equixly's full post covers the scraping mechanics, coverage failure mode, and testing questions in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle controls. It is designed for practitioners building governance across service accounts, tokens, and delegated access paths.