MITRE ATT&CK T1594 Search Victim-Owned Websites: Reducing and Detecting Reconnaissance

Your own websites are the one reconnaissance source you both publish and log, which makes them the best place to cut what an adversary learns and to spot them learning it. MITRE ATT&CK T1594 Search Victim-Owned Websites covers adversaries searching websites owned by the victim for information that can be used during targeting: department names, locations, key staff with their roles and contact details, and business relationships. ATT&CK v20 also includes looking for hidden directories or files, through automated wordlist scanning and by reading sitemap.xml and robots.txt. The technique is in the Reconnaissance tactic (TA0043), on the PRE platform, with no sub-techniques. Unlike most reconnaissance, it touches infrastructure you run, so ATT&CK's detection strategy points at your application logs. This guide is for web, application security, detection engineering and threat intelligence teams. Bot defences and rate limiting are covered in the Web Scraping Defence Guide; this page covers the ATT&CK technique view: what adversaries look for, recorded cases, what to log and how to reduce exposure.

Key takeaways

• Your site is a targeting brief. Staff directories, partner pages, press releases and contact forms tell an adversary who to impersonate, who to target and what pretext will sound right.
• Contact forms are an attack channel. ATT&CK records EXOTIC LILY using victim contact forms to generate phishing emails and TA578 filling in contact forms to direct victims to adversary-controlled URLs.
• robots.txt is a map, not a lock. RFC 9309 states that robots.txt rules are not a form of access authorisation and that listing paths makes them discoverable. Protect sensitive paths with authentication, not disallow lines.
• This is the reconnaissance you can see. ATT&CK's analytic for T1594 uses application logs: rapid request sequences, high volume from one source and suspicious referer or user-agent values.
• Internal sites count too. ATT&CK records actors in the Cutting Edge campaign reconnoitring victims' internal websites through proxied connections, so intranet and application access logs belong in scope after a perimeter compromise.

What T1594 covers

ATT&CK describes the information on victim-owned sites as including names of departments and divisions, physical locations, and data about key employees such as names, roles and contact information (linked to T1589.002 Email Addresses), plus details of business operations and relationships. Results lead to other reconnaissance (T1598 Phishing for Information, T1596 Search Open Technical Databases), resource development (T1585 Establish Accounts, T1586 Compromise Accounts) and initial access (T1199 Trusted Relationship, T1566 Phishing). Hidden content discovery overlaps with T1595.003 Wordlist Scanning, which ATT&CK lists as one way of finding concealed directories and files.

How adversaries use it: recorded cases

The procedure examples below are from ATT&CK v20 and the reports it cites. Attribution is as stated by those sources.

The contact form cases matter because the message arrives through a channel your staff trust, often bypassing email filtering that would inspect an external sender. For the people side, see the Social Engineering Guide and MITRE ATT&CK T1598 Phishing for Information.

What to detect and where

ATT&CK v20 detection strategy DET0810 for T1594 has one analytic using the Application Log data component. It recommends monitoring for rapid successions of requests indicative of web crawling, large quantities of requests from a single source (especially one known to be associated with an adversary), and analysing web metadata such as referer and user-agent fields. Legitimate crawlers, uptime monitors and partners generate the same patterns, so baselining is essential.

One rule of thumb: a source that reads robots.txt and then requests disallowed paths is worth a look, because well-behaved crawlers do the opposite. Treat these as hunting leads and enrichment for later alerts rather than paging alerts on their own. Retain web logs long enough to look back when a phishing campaign or intrusion is later linked to a source.

Mitigations: reducing what your sites give away

ATT&CK v20 maps T1594 only to M1056 Pre-compromise: the technique cannot be easily mitigated with preventive controls, and effort should focus on minimising the amount and sensitivity of data available to external parties. The steps below are NHI Mgmt Group recommendations.

1. Review staff information: publish role-based addresses (for example a press or partnerships mailbox) instead of individual names and direct lines for IT, finance and help desk staff. Keep leadership pages but avoid publishing assistants' details.
2. Authenticate, do not hide: move admin consoles, staging sites, internal documentation and partner portals behind authentication and remove them from robots.txt, since RFC 9309 notes listed paths become discoverable.
3. Remove forgotten content: find and delete backup files, directory listings and exposed version control folders. See the Directory Listing and Exposed Backup Files Guide and Exposed .git Directory Response.
4. Strip technology clues: suppress version banners and verbose error pages, and strip author and software metadata from published documents.
5. Harden contact forms: route submissions to a ticketing queue rather than an individual mailbox, render links as plain text, and apply rate limiting and bot challenges. Tell receiving staff that form submissions are untrusted input.
6. Rate limit and challenge crawlers: apply the bot management controls in the Bot Management Guide to the sections adversaries value most.

Where identity fits

Two identity angles are worth checking. First, public pages that name administrators and help desk staff make help desk social engineering easier; verification for password and MFA resets should not rely on anything a website discloses. Second, the Cutting Edge and Leviathan cases show internal site reconnaissance after access is gained, often through an appliance or service account. Internal web applications should log the authenticated identity, and service accounts used by edge devices or integrations should not have interactive access to intranet content they do not need.

Implementation: a phased approach

1. Weeks 1 to 2, inventory: list every public web property, including marketing microsites, careers sites and supplier-hosted pages, with an owner for each.
2. Weeks 2 to 4, content review: walk each site as an adversary would. Record named staff, email formats, partner names, technology clues and paths in robots.txt and sitemap.xml.
3. Weeks 3 to 6, logging: make sure access logs from origin servers and the CDN reach the SIEM with client IP, ASN, user-agent, referer and status code. Build a baseline of known good crawlers.
4. Weeks 4 to 8, reduce: apply the mitigations above, starting with authentication for anything sensitive and role-based contact details for targeted teams. Test changes on staging; keep the previous robots.txt and page versions so changes can be reverted.
5. Ongoing, hunt and review: run the analytics as weekly hunts, review sites after each redesign, and repeat the adversary walk-through twice a year.

Measuring it

• Named high-risk staff: count of IT, finance and help desk individuals named with direct contact details on public sites, trending down.
• Sensitive paths unauthenticated: number of admin, staging or internal paths reachable without authentication.
• Log coverage: percentage of public web properties whose access logs reach the SIEM.
• Contact form lures: malicious submissions caught per month and time to block the sender or URL.
• Hunt yield: crawl or path-guessing sources investigated per month and the share later linked to phishing or scanning.

Common failure patterns

• Disallow as protection: sensitive paths listed in robots.txt and left unauthenticated, which advertises them.
• Logs at the CDN only: the origin logs requests from the CDN, while the CDN logs that show the real client are not retained or forwarded.
• Forms to personal inboxes: contact form submissions land directly in an individual's mailbox with clickable links and no filtering.
• Microsites forgotten: agency-built campaign sites and old careers portals stay online without an owner or logging.
• Alerting on every crawler: a crawl rule pages the SOC dozens of times a day and is switched off within a week, instead of being run as a hunt.

Framework mapping

The AI view of this technique is in AML.T0003 Search Victim-Owned Websites. Related techniques: T1593 Search Open Websites/Domains (see MITRE ATT&CK T1593 Search Open Websites/Domains) covers third-party sites such as social media and code hosting, T1591 Gather Victim Org Information (see T1591 Gather Victim Org Information Defensive Guide) covers what the information is used for, and T1595 Active Scanning (see T1595 Active Scanning Defensive Guide). For internet scan data and certificate records, see MITRE ATT&CK T1596 Search Open Technical Databases.

Practitioner checklist

• Every public web property has an owner and its access logs, including CDN logs, reach the SIEM.
• No admin, staging or internal path relies on robots.txt or obscurity for protection.
• IT, finance and help desk staff are reached through role-based contacts, not named direct lines.
• Contact form submissions go to a ticket queue, are treated as untrusted and are rate limited.
• Version banners, verbose errors and document metadata are removed from public content.
• Crawl, path-guessing and robots-then-disallowed-path hunts run weekly against a crawler baseline.
• Internal web applications log authenticated identity, and edge device service accounts cannot browse intranet content.
• An adversary walk-through of public sites is repeated at least twice a year and after redesigns.

Standards and references

• MITRE ATT&CK: T1594 Search Victim-Owned Websites
• MITRE ATT&CK: M1056 Pre-compromise
• IETF: RFC 9309 Robots Exclusion Protocol
• CISA: AA24-038A PRC State-Sponsored Actors Compromise and Maintain Persistent Access to U.S. Critical Infrastructure
• Google Cloud (Mandiant): APT41 Has Arisen From the DUST
• MITRE ATT&CK: EXOTIC LILY (G1011)
• MITRE ATT&CK: Cutting Edge (C0029)

Related NHI Mgmt Group resources: Web Scraping Defence Guide, OSINT and Reconnaissance Guide, ATT&CK Reconnaissance Tactic (TA0043) Guide, Vishing and Help Desk Social Engineering Awareness Guide.