Web server access logs record almost every exploitation of a public-facing application, yet most organisations keep them only for troubleshooting and never write a detection against them. When an internet-facing application is exploited, the request that did it, the web shell that followed and the data that left are usually all in the access log of Apache, NGINX or IIS, often days before endpoint tooling notices anything. This guide is for detection engineers, SOC analysts and web platform owners. It covers the log formats and fields that matter, how to configure logging so the evidence is there, detection logic for scanning, exploitation, web shells, credential attacks and data theft, and how to measure the result. Response to a confirmed web shell is covered in the Web Shell Response Playbook; this guide is about finding it.
Key takeaways
- The default formats are not enough. Common and Combined formats omit response time, bytes received, the virtual host and the real client IP behind a proxy; add them before you need them.
- Rarity beats signatures. NSA and ASD guidance finds web shells by looking for URIs requested by very few user agents or client IPs, which works even when the shell has a convincing name.
- Successful responses matter most. A 404 storm is noise; a 200 on a path that has never returned 200 before, or a POST to a static directory, is the signal.
- Proxy headers can lie. X-Forwarded-For is trivially forged unless the server only trusts it from known proxies, so log both the connection peer and the forwarded address.
- Logs are evidence and a target. Ship them off the host quickly, protect the log directory and treat a gap in logging as a detection in its own right.
Log formats and the fields that matter
Apache HTTP Server defines the Common Log Format as "%h %l %u %t \"%r\" %>s %b" and the Combined format as the same string plus \"%{Referer}i\" \"%{User-agent}i\". NGINX's predefined combined format is equivalent: $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent", and it is used whenever access_log names no format. IIS writes W3C extended logs by default to %SystemDrive%\inetpub\logs\LogFiles, with time in UTC.
| Evidence | Apache (LogFormat) | NGINX (variable) | IIS W3C field | Default? |
|---|---|---|---|---|
| Client IP (as seen by the server) | %h or %a | $remote_addr | ClientIP (c-ip) | Yes |
| Connection peer behind a proxy | %{c}a with mod_remoteip | $realip_remote_addr with the realip module | Not native; add a custom field | No |
| Forwarded client chain | %{X-Forwarded-For}i | $http_x_forwarded_for | Custom field (IIS 8.5+) | No |
| Time | %t (local time with offset) | $time_local or $time_iso8601 | Date, Time (UTC) | Yes |
| Method, path, query | %r (request line) | $request | Method, UriStem, UriQuery | Yes |
| Final status | %>s | $status | HttpStatus, HttpSubStatus, Win32Status | Yes |
| Bytes sent | %b (body) or %O (with headers, mod_logio) | $body_bytes_sent or $bytes_sent | BytesSent | Apache and NGINX yes; IIS no |
| Bytes received | %I (mod_logio) | $request_length | BytesRecv | No |
| Duration | %D (microseconds) or %T (seconds) | $request_time (millisecond resolution) | TimeTaken | IIS yes; others no |
| Virtual host | %v or %{Host}i | $host or $server_name | Host, SiteName | No |
| Authenticated user | %u | $remote_user | UserName | Yes |
| Referer and user agent | %{Referer}i, %{User-agent}i | $http_referer, $http_user_agent | Referer, UserAgent | Yes in Combined and W3C |
A good security format adds bytes in and out, duration, virtual host, the connection peer and the forwarded chain, and a request ID shared with the application log. NGINX supports escape=json (from 1.11.8) so the same fields can be written as JSON, which removes most parsing ambiguity. Do not log cookies, Authorization headers or full query strings that carry tokens: the OWASP Logging Cheat Sheet lists session IDs, access tokens, passwords and connection strings as data to exclude or mask.
Getting the real client IP right
Behind a load balancer, CDN or reverse proxy, the access log's client IP is the proxy unless you configure otherwise. Apache's mod_remoteip processes X-Forwarded-For from right to left and stops at the first address that is not trusted. Its documentation warns that if neither RemoteIPTrustedProxy nor RemoteIPInternalProxy is set, it trusts all hosts presenting the header, and that it is trivial for a client to impersonate another address. NGINX's realip module has the same model with set_real_ip_from.
- Trust only known proxies: list your load balancer and CDN ranges explicitly; never trust the header from the internet.
- Log both addresses: in Apache,
%arecords the overridden client address and%{c}athe connection peer. A mismatch between the two on a request that did not pass through your proxy is itself suspicious. - Block direct origin access: if clients can reach the origin directly, forwarded headers are attacker controlled. Restrict origin ingress to proxy ranges.
- Normalise in the SIEM: map the true client IP into a single field so that rate rules and threat intelligence matching work across tiers.
What attacks look like in access logs
| Activity | ATT&CK | Access log pattern | Noise level |
|---|---|---|---|
| Vulnerability scanning | T1595.002 | Bursts of 404 and 400 from one IP across many unrelated paths, scanner user agents, requests for admin consoles and known vulnerable files | Very high: use for enrichment, not alerts |
| Wordlist and content discovery | T1595.003 | Hundreds of sequential paths with near-identical response sizes, many 404 with occasional 200 or 403 | High |
| Path traversal and sensitive file probing | T1190 | ../ or encoded variants in the path or query, requests for .env, .git/config, backup and configuration files; a 200 on any of these is critical | Medium; 200 responses are rare |
| Injection attempts | T1190 | SQL or template syntax in query strings, very long parameters, 500 responses clustered on one endpoint, response time spikes | Medium |
| Exploitation of a known flaw | T1190 | Requests to the specific vulnerable endpoint from advisories, unusual method or Referer for that path, followed by new files being requested | Low once tuned to your stack |
| Web shell use | T1505.003 | A new URI requested by very few IPs and user agents, often POST, returning 200, with no Referer on a non-root path | Low with rarity analytics |
| Credential stuffing and spraying | T1110.004, T1110.003 | High rate of POST to the login endpoint, mostly 401 or 200 with failure page size, many distinct usernames per IP | Medium |
| Data theft over HTTP | T1567, T1213 | Unusually large bytes sent to one client, bulk export endpoint use, sequential object IDs | Low with byte baselines |
| Application layer flood | T1499.002 | Request rate far above baseline on expensive endpoints, rising duration and 503 responses | Low |
The Apache security tips page gives the simplest example of all: counting requests containing ../ in the access log and listing recent "client denied" entries in the error log. If a request for a sensitive file such as .env appears with status 200 instead of being denied, the configuration needs fixing immediately. See the Path Traversal Guide and the SQL Injection Prevention Guide.
Finding web shells with rarity analytics
The NSA and ASD guidance on detecting and preventing web shell malware builds its access log analytics on a simple observation: attackers rarely know which user agents normally use a server, and a web shell is visited by its operator and nobody else. Its recommended queries, which it provides for Apache and IIS, filter successful responses (status 200 to 299) and then rank URIs by how few distinct user agents, client IPs and referers have requested them.
- Restrict to success: keep requests with status 2xx; failed probes are not shells.
- Group by URI stem: strip query strings so one script is one row.
- Count distinct client IPs and user agents per URI: the guidance flags URIs with five or fewer unique client IPs, and its scripts list the bottom 5 percent by default.
- Check referers: missing referers on non-root URIs and referers that are unusual for the site are additional weak signals.
- Review, do not page: the guidance stresses that no single analytic is sufficient and that applications generating unique URIs per request reduce effectiveness, so tune per application.
Add two signals that work well in practice. First, any URI in a static or upload directory that answers a POST. Second, a script file whose first ever 200 response appeared in the last seven days on a server that has had no deployment. Pair access log findings with file integrity monitoring on the web root, because the file write is often what confirms the shell. The NSA and ASD guidance also lists unusually large responses, recurring off-peak access and geographically disparate requests as indicators, but describes them as subjective and likely to flag benign URIs.
Lessons from incidents
- Exploit plus shell in one log: in the ToolShell SharePoint exploitation, attackers sent a POST to
/_layouts/15/ToolPane.aspxwith a Referer of/_layouts/SignOut.aspx, then requestedspinstall0.aspx, which returned the server's ASP.NET machine keys via GET. A rule for POST to that endpoint with that Referer, and a rarity rule on new .aspx files under LAYOUTS, would each have caught it in the IIS log. Microsoft's guidance also includes hunting the csReferer and csHost fields for the campaign's command and control domain. - Look-alike file names: the CISA and FBI advisory on MOVEit Transfer exploitation describes the LEMURLOOT web shell named
human2.aspx, chosen to resemble the legitimatehuman.aspx, which acted only on requests carrying anX-siLock-Commentheader with a 36 character password. A name check would not have caught it; rarity of requesting clients and a first-seen file would. - Stolen keys outlive the patch: where a shell exfiltrates secrets such as machine keys, API keys or database connection strings, patching does not end the incident. Rotate the secrets, then watch the access logs for requests that use the old material. See the Web Shell Response Playbook.
Where identity fits
Web servers hold machine credentials: database passwords in configuration files, cloud keys in environment variables, signing keys such as ASP.NET machine keys, and API tokens for back-end services. Access logs show the requests that steal them (a 200 on .env, web.config or .git/config, or a shell returning key material), and the identity provider and cloud logs show the stolen credential being used later from a new network. Correlate both, and treat any confirmed exposure of a secret file as a credential rotation event. See the API Key Management Guide.
Configuring logging for detection
- Define one security format per server type: add duration, bytes in and out, virtual host, connection peer, forwarded chain and a request ID. On IIS, enable BytesSent, BytesRecv and Host, which are off by default, and use custom fields (IIS 8.5 and later) for X-Forwarded-For.
- Prefer structured output: JSON escaping in NGINX, or a strict delimiter scheme in Apache, so attacker-supplied characters in the path or user agent cannot break parsing. NGINX's default escaping writes quotes, backslashes and characters outside 32 to 126 as
\xXX; avoidescape=none. - Ship off the host quickly: forward with a log agent or syslog over TLS. If NGINX buffering is used, set
flushso that buffered lines are written within a short interval. - Protect the local copy: Apache's security tips say the logs directory should be owned by root and not writable by others, because a writable directory lets a user replace a log with a symlink to another file.
- Keep error logs too: application errors, denied requests and module warnings often show an exploit attempt that the access log records only as a 500.
- Pilot and compare: roll the new format to one server, compare parsed fields in the SIEM against raw lines, then roll to the rest. Roll back by reverting the format directive, which needs only a reload.
See the Apache HTTP Server Hardening Guide, the NGINX Security Hardening Guide and the IIS Hardening Guide for the wider configuration.
Decision guidance
- Origin logs, edge logs or both: CDN and WAF logs see attacks the origin never receives, but only origin logs prove what the application actually served. Collect both for internet-facing applications that hold sensitive data; origin only is acceptable for low-risk static sites.
- Full collection or sampling: never sample security logs for applications that accept uploads, logins or administrative actions. Sampling is tolerable only for high-volume static content.
- Signature or behaviour: signature rules for current advisories are quick wins but age fast; rarity and baseline rules take longer to tune and keep working. Run both.
- Small teams: start with three rules: 200 on sensitive files, POST to static or upload paths, and the weekly rarity report. Leave scanner detection to the WAF.
- Large estates: standardise one format across all web tiers, tag logs with application owner and environment, and build per-application baselines.
Detection and measurement
- Coverage: percentage of internet-facing web servers sending access and error logs to the SIEM in the security format (target 100).
- Latency: median minutes from request to searchable event (target under 5).
- Parse quality: percentage of lines parsed with every security field populated.
- Rarity review cadence: rarity reports reviewed per application per week, with findings closed.
- Time to detect test shells: minutes from deploying a benign test script in a controlled environment to alert.
- Retention: days of access logs searchable (target at least 90 hot and 12 months archived).
Common failure patterns
- Logging the load balancer's IP: every request appears to come from the proxy, so rate limits, blocklists and investigation all fail.
- Trusting X-Forwarded-For from anywhere: attackers set the header to an internal or allow-listed address and the log records a lie.
- Alerting on 404 volume: internet background scanning produces constant noise that teaches analysts to ignore the queue, while the one 200 that mattered is lost.
- Logging secrets: tokens in query strings, Authorization headers or cookies end up in the SIEM and widen the blast radius. Mask them at the source.
- Logs only on the compromised host: an attacker with a shell can delete or alter local logs. Without off-host copies, the investigation depends on what they left.
- Ignoring the error log: the error log often contains the stack trace or module error that identifies the vulnerable component.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.PS-04, DE.CM-01, DE.AE-02, DE.AE-03 | Log generation, network monitoring, analysis and correlation |
| NIST SP 800-53 Rev. 5 | AU-2, AU-3, AU-9, AU-12, SI-4, SI-10 | Event logging, record content, log protection, generation, monitoring, input validation |
| CIS Controls v8.1 | 8.2, 8.5, 8.9, 8.11 | Collect, detail, centralise and review audit logs |
| ISO/IEC 27001:2022 | Annex A 8.15, 8.16 | Logging; monitoring activities |
| MITRE ATT&CK v20 | T1190, T1505.003, T1595.002, T1595.003, T1110.004, T1567 | Techniques visible in access logs |
| MITRE D3FEND | D3-WSAA Web Session Activity Analysis, D3-NTA Network Traffic Analysis | Defensive techniques the rules implement |
Practitioner checklist
- Adopt a security log format with duration, bytes in and out, virtual host, connection peer and forwarded chain on every web tier.
- Trust forwarded headers only from listed proxy ranges and block direct origin access.
- Mask cookies, Authorization headers and token parameters before logs leave the server.
- Forward access and error logs off the host within minutes, using structured output.
- Alert on 200 responses for sensitive files and on POST to static or upload directories.
- Run a weekly rarity report of 2xx URIs with five or fewer client IPs or user agents.
- Write a rule for each current advisory affecting your web stack and retire it when patched and verified.
- Correlate access log findings with file integrity monitoring and credential use in identity logs.
- Rotate any secret a confirmed shell or file exposure could have read.
Standards and references
- Apache HTTP Server: mod_log_config
- NGINX: Module ngx_http_log_module
- Microsoft Learn: IIS Log Files configuration
- NSA and ASD: Detect and Prevent Web Shell Malware
- OWASP: Logging Cheat Sheet
- Apache HTTP Server: mod_remoteip
- CISA: CL0P Ransomware Gang Exploits MOVEit Vulnerability (AA23-158A)
- Microsoft Security: Disrupting active exploitation of on-premises SharePoint vulnerabilities
Related NHI Mgmt Group resources: Web Shell Response Playbook · Web Application Firewall Guide · Detection Engineering Guide · SIEM Guide · VPN and Firewall Log Detection Guide