Join our Newsletter — 33% off our NHI Course
Home› Guides› Web Server Access Log Detection Guide: Formats, Fields…
Guide SOC & Detection Engineering

Web Server Access Log Detection Guide: Formats, Fields and Rules for Finding Attacks

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 27 October 2026 14 min read
On this page

Web server access logs record almost every exploitation of a public-facing application, yet most organisations keep them only for troubleshooting and never write a detection against them. When an internet-facing application is exploited, the request that did it, the web shell that followed and the data that left are usually all in the access log of Apache, NGINX or IIS, often days before endpoint tooling notices anything. This guide is for detection engineers, SOC analysts and web platform owners. It covers the log formats and fields that matter, how to configure logging so the evidence is there, detection logic for scanning, exploitation, web shells, credential attacks and data theft, and how to measure the result. Response to a confirmed web shell is covered in the Web Shell Response Playbook; this guide is about finding it.

Key takeaways

  • The default formats are not enough. Common and Combined formats omit response time, bytes received, the virtual host and the real client IP behind a proxy; add them before you need them.
  • Rarity beats signatures. NSA and ASD guidance finds web shells by looking for URIs requested by very few user agents or client IPs, which works even when the shell has a convincing name.
  • Successful responses matter most. A 404 storm is noise; a 200 on a path that has never returned 200 before, or a POST to a static directory, is the signal.
  • Proxy headers can lie. X-Forwarded-For is trivially forged unless the server only trusts it from known proxies, so log both the connection peer and the forwarded address.
  • Logs are evidence and a target. Ship them off the host quickly, protect the log directory and treat a gap in logging as a detection in its own right.

Log formats and the fields that matter

Apache HTTP Server defines the Common Log Format as "%h %l %u %t \"%r\" %>s %b" and the Combined format as the same string plus \"%{Referer}i\" \"%{User-agent}i\". NGINX's predefined combined format is equivalent: $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent", and it is used whenever access_log names no format. IIS writes W3C extended logs by default to %SystemDrive%\inetpub\logs\LogFiles, with time in UTC.

EvidenceApache (LogFormat)NGINX (variable)IIS W3C fieldDefault?
Client IP (as seen by the server)%h or %a$remote_addrClientIP (c-ip)Yes
Connection peer behind a proxy%{c}a with mod_remoteip$realip_remote_addr with the realip moduleNot native; add a custom fieldNo
Forwarded client chain%{X-Forwarded-For}i$http_x_forwarded_forCustom field (IIS 8.5+)No
Time%t (local time with offset)$time_local or $time_iso8601Date, Time (UTC)Yes
Method, path, query%r (request line)$requestMethod, UriStem, UriQueryYes
Final status%>s$statusHttpStatus, HttpSubStatus, Win32StatusYes
Bytes sent%b (body) or %O (with headers, mod_logio)$body_bytes_sent or $bytes_sentBytesSentApache and NGINX yes; IIS no
Bytes received%I (mod_logio)$request_lengthBytesRecvNo
Duration%D (microseconds) or %T (seconds)$request_time (millisecond resolution)TimeTakenIIS yes; others no
Virtual host%v or %{Host}i$host or $server_nameHost, SiteNameNo
Authenticated user%u$remote_userUserNameYes
Referer and user agent%{Referer}i, %{User-agent}i$http_referer, $http_user_agentReferer, UserAgentYes in Combined and W3C

A good security format adds bytes in and out, duration, virtual host, the connection peer and the forwarded chain, and a request ID shared with the application log. NGINX supports escape=json (from 1.11.8) so the same fields can be written as JSON, which removes most parsing ambiguity. Do not log cookies, Authorization headers or full query strings that carry tokens: the OWASP Logging Cheat Sheet lists session IDs, access tokens, passwords and connection strings as data to exclude or mask.

Getting the real client IP right

Behind a load balancer, CDN or reverse proxy, the access log's client IP is the proxy unless you configure otherwise. Apache's mod_remoteip processes X-Forwarded-For from right to left and stops at the first address that is not trusted. Its documentation warns that if neither RemoteIPTrustedProxy nor RemoteIPInternalProxy is set, it trusts all hosts presenting the header, and that it is trivial for a client to impersonate another address. NGINX's realip module has the same model with set_real_ip_from.

  • Trust only known proxies: list your load balancer and CDN ranges explicitly; never trust the header from the internet.
  • Log both addresses: in Apache, %a records the overridden client address and %{c}a the connection peer. A mismatch between the two on a request that did not pass through your proxy is itself suspicious.
  • Block direct origin access: if clients can reach the origin directly, forwarded headers are attacker controlled. Restrict origin ingress to proxy ranges.
  • Normalise in the SIEM: map the true client IP into a single field so that rate rules and threat intelligence matching work across tiers.

What attacks look like in access logs

ActivityATT&CKAccess log patternNoise level
Vulnerability scanningT1595.002Bursts of 404 and 400 from one IP across many unrelated paths, scanner user agents, requests for admin consoles and known vulnerable filesVery high: use for enrichment, not alerts
Wordlist and content discoveryT1595.003Hundreds of sequential paths with near-identical response sizes, many 404 with occasional 200 or 403High
Path traversal and sensitive file probingT1190../ or encoded variants in the path or query, requests for .env, .git/config, backup and configuration files; a 200 on any of these is criticalMedium; 200 responses are rare
Injection attemptsT1190SQL or template syntax in query strings, very long parameters, 500 responses clustered on one endpoint, response time spikesMedium
Exploitation of a known flawT1190Requests to the specific vulnerable endpoint from advisories, unusual method or Referer for that path, followed by new files being requestedLow once tuned to your stack
Web shell useT1505.003A new URI requested by very few IPs and user agents, often POST, returning 200, with no Referer on a non-root pathLow with rarity analytics
Credential stuffing and sprayingT1110.004, T1110.003High rate of POST to the login endpoint, mostly 401 or 200 with failure page size, many distinct usernames per IPMedium
Data theft over HTTPT1567, T1213Unusually large bytes sent to one client, bulk export endpoint use, sequential object IDsLow with byte baselines
Application layer floodT1499.002Request rate far above baseline on expensive endpoints, rising duration and 503 responsesLow

The Apache security tips page gives the simplest example of all: counting requests containing ../ in the access log and listing recent "client denied" entries in the error log. If a request for a sensitive file such as .env appears with status 200 instead of being denied, the configuration needs fixing immediately. See the Path Traversal Guide and the SQL Injection Prevention Guide.

Finding web shells with rarity analytics

The NSA and ASD guidance on detecting and preventing web shell malware builds its access log analytics on a simple observation: attackers rarely know which user agents normally use a server, and a web shell is visited by its operator and nobody else. Its recommended queries, which it provides for Apache and IIS, filter successful responses (status 200 to 299) and then rank URIs by how few distinct user agents, client IPs and referers have requested them.

  1. Restrict to success: keep requests with status 2xx; failed probes are not shells.
  2. Group by URI stem: strip query strings so one script is one row.
  3. Count distinct client IPs and user agents per URI: the guidance flags URIs with five or fewer unique client IPs, and its scripts list the bottom 5 percent by default.
  4. Check referers: missing referers on non-root URIs and referers that are unusual for the site are additional weak signals.
  5. Review, do not page: the guidance stresses that no single analytic is sufficient and that applications generating unique URIs per request reduce effectiveness, so tune per application.

Add two signals that work well in practice. First, any URI in a static or upload directory that answers a POST. Second, a script file whose first ever 200 response appeared in the last seven days on a server that has had no deployment. Pair access log findings with file integrity monitoring on the web root, because the file write is often what confirms the shell. The NSA and ASD guidance also lists unusually large responses, recurring off-peak access and geographically disparate requests as indicators, but describes them as subjective and likely to flag benign URIs.

Lessons from incidents

  • Exploit plus shell in one log: in the ToolShell SharePoint exploitation, attackers sent a POST to /_layouts/15/ToolPane.aspx with a Referer of /_layouts/SignOut.aspx, then requested spinstall0.aspx, which returned the server's ASP.NET machine keys via GET. A rule for POST to that endpoint with that Referer, and a rarity rule on new .aspx files under LAYOUTS, would each have caught it in the IIS log. Microsoft's guidance also includes hunting the csReferer and csHost fields for the campaign's command and control domain.
  • Look-alike file names: the CISA and FBI advisory on MOVEit Transfer exploitation describes the LEMURLOOT web shell named human2.aspx, chosen to resemble the legitimate human.aspx, which acted only on requests carrying an X-siLock-Comment header with a 36 character password. A name check would not have caught it; rarity of requesting clients and a first-seen file would.
  • Stolen keys outlive the patch: where a shell exfiltrates secrets such as machine keys, API keys or database connection strings, patching does not end the incident. Rotate the secrets, then watch the access logs for requests that use the old material. See the Web Shell Response Playbook.

Where identity fits

Web servers hold machine credentials: database passwords in configuration files, cloud keys in environment variables, signing keys such as ASP.NET machine keys, and API tokens for back-end services. Access logs show the requests that steal them (a 200 on .env, web.config or .git/config, or a shell returning key material), and the identity provider and cloud logs show the stolen credential being used later from a new network. Correlate both, and treat any confirmed exposure of a secret file as a credential rotation event. See the API Key Management Guide.

Configuring logging for detection

  1. Define one security format per server type: add duration, bytes in and out, virtual host, connection peer, forwarded chain and a request ID. On IIS, enable BytesSent, BytesRecv and Host, which are off by default, and use custom fields (IIS 8.5 and later) for X-Forwarded-For.
  2. Prefer structured output: JSON escaping in NGINX, or a strict delimiter scheme in Apache, so attacker-supplied characters in the path or user agent cannot break parsing. NGINX's default escaping writes quotes, backslashes and characters outside 32 to 126 as \xXX; avoid escape=none.
  3. Ship off the host quickly: forward with a log agent or syslog over TLS. If NGINX buffering is used, set flush so that buffered lines are written within a short interval.
  4. Protect the local copy: Apache's security tips say the logs directory should be owned by root and not writable by others, because a writable directory lets a user replace a log with a symlink to another file.
  5. Keep error logs too: application errors, denied requests and module warnings often show an exploit attempt that the access log records only as a 500.
  6. Pilot and compare: roll the new format to one server, compare parsed fields in the SIEM against raw lines, then roll to the rest. Roll back by reverting the format directive, which needs only a reload.

See the Apache HTTP Server Hardening Guide, the NGINX Security Hardening Guide and the IIS Hardening Guide for the wider configuration.

Decision guidance

  • Origin logs, edge logs or both: CDN and WAF logs see attacks the origin never receives, but only origin logs prove what the application actually served. Collect both for internet-facing applications that hold sensitive data; origin only is acceptable for low-risk static sites.
  • Full collection or sampling: never sample security logs for applications that accept uploads, logins or administrative actions. Sampling is tolerable only for high-volume static content.
  • Signature or behaviour: signature rules for current advisories are quick wins but age fast; rarity and baseline rules take longer to tune and keep working. Run both.
  • Small teams: start with three rules: 200 on sensitive files, POST to static or upload paths, and the weekly rarity report. Leave scanner detection to the WAF.
  • Large estates: standardise one format across all web tiers, tag logs with application owner and environment, and build per-application baselines.

Detection and measurement

  • Coverage: percentage of internet-facing web servers sending access and error logs to the SIEM in the security format (target 100).
  • Latency: median minutes from request to searchable event (target under 5).
  • Parse quality: percentage of lines parsed with every security field populated.
  • Rarity review cadence: rarity reports reviewed per application per week, with findings closed.
  • Time to detect test shells: minutes from deploying a benign test script in a controlled environment to alert.
  • Retention: days of access logs searchable (target at least 90 hot and 12 months archived).

Common failure patterns

  • Logging the load balancer's IP: every request appears to come from the proxy, so rate limits, blocklists and investigation all fail.
  • Trusting X-Forwarded-For from anywhere: attackers set the header to an internal or allow-listed address and the log records a lie.
  • Alerting on 404 volume: internet background scanning produces constant noise that teaches analysts to ignore the queue, while the one 200 that mattered is lost.
  • Logging secrets: tokens in query strings, Authorization headers or cookies end up in the SIEM and widen the blast radius. Mask them at the source.
  • Logs only on the compromised host: an attacker with a shell can delete or alter local logs. Without off-host copies, the investigation depends on what they left.
  • Ignoring the error log: the error log often contains the stack trace or module error that identifies the vulnerable component.

Framework mapping

FrameworkReferenceRelevance
NIST CSF 2.0PR.PS-04, DE.CM-01, DE.AE-02, DE.AE-03Log generation, network monitoring, analysis and correlation
NIST SP 800-53 Rev. 5AU-2, AU-3, AU-9, AU-12, SI-4, SI-10Event logging, record content, log protection, generation, monitoring, input validation
CIS Controls v8.18.2, 8.5, 8.9, 8.11Collect, detail, centralise and review audit logs
ISO/IEC 27001:2022Annex A 8.15, 8.16Logging; monitoring activities
MITRE ATT&CK v20T1190, T1505.003, T1595.002, T1595.003, T1110.004, T1567Techniques visible in access logs
MITRE D3FENDD3-WSAA Web Session Activity Analysis, D3-NTA Network Traffic AnalysisDefensive techniques the rules implement

Practitioner checklist

  • Adopt a security log format with duration, bytes in and out, virtual host, connection peer and forwarded chain on every web tier.
  • Trust forwarded headers only from listed proxy ranges and block direct origin access.
  • Mask cookies, Authorization headers and token parameters before logs leave the server.
  • Forward access and error logs off the host within minutes, using structured output.
  • Alert on 200 responses for sensitive files and on POST to static or upload directories.
  • Run a weekly rarity report of 2xx URIs with five or fewer client IPs or user agents.
  • Write a rule for each current advisory affecting your web stack and retire it when patched and verified.
  • Correlate access log findings with file integrity monitoring and credential use in identity logs.
  • Rotate any secret a confirmed shell or file exposure could have read.

Standards and references

Related NHI Mgmt Group resources: Web Shell Response Playbook · Web Application Firewall Guide · Detection Engineering Guide · SIEM Guide · VPN and Firewall Log Detection Guide

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 27 October 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.