Proxy and Web Gateway Log Detection Guide

Almost every intrusion talks to the internet over HTTP or HTTPS, and the web proxy or secure web gateway is often the one place that records who made each request, to which host, with which client and how much data moved. Those logs are usually collected for web filtering and then ignored by the SOC, or kept at too little detail to answer an investigation. This guide is for detection engineers, threat hunters and SOC analysts. It covers which fields proxy and gateway logs carry and how to normalise them, the visibility gaps that TLS and bypass routes create, detection logic for command and control (C2), tool downloads, exfiltration and risky browsing, how to correlate with DNS, endpoint and identity data, and how to measure coverage. Hardening and policy for the proxy itself sit in the Proxies and Forward vs Reverse Proxy Security Guide and the Secure Web Gateway and Web Filtering Guide; this guide is about the evidence.

Key takeaways

The proxy log is an identity log as well as a network log. When users and service accounts authenticate to the proxy, every request carries a username, which turns an IP address into a person or workload.
Allowed requests are where intrusions hide. Blocked categories show policy working; the rare domain, the odd user agent and the large upload in the allowed traffic show what an intruder did.
TLS decides how much you can see. Without inspection, an HTTPS request is a CONNECT to a hostname and port with byte counts; with inspection you also get the full URL, method, user agent and response type.
Normalise before you write rules. Map native fields to one schema, such as the W3C Extended Log File Format names used by Sigma's proxy category or the OCSF HTTP Activity class, so detections survive a product change.
Bypass breaks every detection. If servers or users can reach the internet without the proxy, the absence of an alert means nothing, so watch the firewall for direct web egress.

What proxy and gateway logs record

The joint guidance Best Practices for Event Logging and Threat Detection, led by the Australian Signals Directorate's ACSC with CISA, the FBI, the NSA, the UK NCSC and other partners, lists web proxies used by organisational users and service accounts among the priority log sources for enterprise networks, and web proxies used by organisational users among the priority sources for enterprise mobility. The fields that make those logs useful are below, with the W3C Extended Log File Format name (the convention Sigma uses for its category: proxy log source) and the matching OCSF area.

Field	W3C / Sigma name	OCSF HTTP Activity area	Why detection needs it
Timestamp (UTC, milliseconds)	`date`, `time`	`time`	Joins proxy events to DNS, endpoint and identity events
Client IP	`src_ip` (c-ip)	`src_endpoint`	Host attribution where there is no username
Authenticated user	username (cs-username)	`actor`	Turns traffic into a person or service account
Method	`cs-method`	`http_request.http_method`, `activity_id`	CONNECT versus GET versus POST changes the meaning of the volume
Host and full URL	`cs-host`, `c-uri`, `c-uri-stem`, `c-uri-query`	`http_request.url`	Domain rarity, path patterns, file extensions
User agent	`c-useragent`	`http_request.user_agent`	Spots scripted clients, tools and impersonated browsers
Referrer	`cs-referrer`	`http_request.referrer`	Requests with no referrer to rare hosts suggest non-browser traffic
Status code	`sc-status`	`http_response.code`	Repeated failures, redirects to downloads
Bytes each way	`cs-bytes` (client to server), `sc-bytes` (server to client)	`traffic`	Upload versus download ratio for exfiltration
Content type and file details	Response content type	`http_response`, `file`	Executables, scripts and archives from rare hosts
Destination IP	`dst_ip`	`dst_endpoint`	Direct-to-IP requests, hosting provider ranges
Policy action and category	Product specific	Product specific (extension or `unmapped`)	Blocked, warned and allowed decisions; uncategorised destinations
TLS details	Product specific	`tls`, `ja4_fingerprint_list`	SNI, certificate issuer and client fingerprints

In the W3C format, field prefixes say which party the data passed between: c client, s server, r remote, cs client to server, sc server to client, and sr and rs for the proxy's own leg to the remote server. Read each product's byte fields carefully against this convention before building ratio rules, because one swapped field turns every upload detection into a download detection.

Log formats and normalisation

Native formats you will meet

Squid native access.log: time (Unix seconds with millisecond resolution, logged when the transaction ends), elapsed milliseconds, client address, result code and HTTP status (for example TCP_MISS/200), bytes delivered to the client, method, URL, user, hierarchy code with the peer or origin host, and content type. The user field is filled from HTTP authentication, an external ACL helper, TLS authentication or IDENT, and shows - when none is available.
Squid result codes worth parsing: TCP_DENIED means access controls denied the request, TCP_TUNNEL marks a binary tunnel (the usual outcome of an uninspected HTTPS CONNECT), and NONE covers unusual or empty responses and is also seen with CONNECT. HIER_DIRECT means the object came from the origin server.
W3C extended logs: a #Fields directive at the top of the file declares which fields each line holds, so parsers must read the header rather than assume a column order.
Cloud gateway exports: cloud secure web gateways deliver logs through streaming connectors, syslog over TLS or object storage exports, usually as JSON with product-specific names for category, action and risk score. Map them field by field to the same schema as on-premises proxies.

Choosing a target schema

Two open schemas cover proxy data well. Sigma's taxonomy (version 2.1.0) defines category: proxy with W3C field names such as c-uri, c-useragent, cs-host, cs-method, sc-status, src_ip and dst_ip, which makes community rules portable. The OCSF schema models web transactions as the HTTP Activity class in the Network Activity category (class 4002), with activity IDs per HTTP method (1 Connect, 3 Get, 6 Post) and network attributes for proxy, tls, traffic and ja4_fingerprint_list. Pick one, document the mapping per product, and test it with known requests before any rule depends on it. See the Sigma Rules Guide and the SIEM Log Source Onboarding Checklist.

Visibility gaps to close first

Gap	What you lose	How to close or compensate
HTTPS without inspection	Path, query, method inside the tunnel, user agent, content type; only host, port and bytes remain	Inspect high-risk categories and uncategorised sites; keep SNI and certificate data; use JA4 client fingerprints where the product supports them. See the TLS Inspection Guide.
Unauthenticated traffic	User attribution; only client IP remains	Require authentication for users and named service accounts; keep DHCP and VPN assignment logs to map IPs
Bypass routes	Everything that does not cross the proxy	Block direct TCP 80 and 443 egress at the firewall except from the proxy, and alert on allowed direct web connections
Exempt destinations	Traffic to trusted cloud and collaboration services, often the busiest exfiltration path	Log exempted traffic even when it is not inspected or filtered
Short retention	The ability to scope an intrusion found months later	The joint guidance notes incidents can take up to 18 months to discover; keep searchable proxy logs at least 12 months where privacy rules allow
Clock drift	Reliable joins with other sources	UTC in ISO 8601 with milliseconds, as the joint guidance recommends

Proxy logs record browsing by named people, so retention, access and use need a documented basis under the privacy rules that apply to you; the Squid project itself advises keeping logs unavailable unless anonymised when sharing results. Restrict who can search raw URLs and query strings, which can carry personal data and sometimes credentials.

Detections for command and control and tool transfer

Behaviour	ATT&CK	Detection logic	Fidelity
Periodic callbacks to one host	T1071.001	Same user or host to the same domain at regular intervals over hours, small and similar request sizes; score interval regularity. See the Beaconing and C2 Detection Guide.	Medium
Rare domain, non-browser client	T1071.001	Domain contacted by one or two hosts in 30 days, with no referrer and a user agent that is rare across the estate	Medium
Direct-to-IP web requests	T1071.001	Host field is a bare IP address rather than a name, outside known service ranges	Medium
CONNECT to unusual ports	T1572	CONNECT method to any port other than 443 (or the approved list), allowed or denied	High
Tunnelling and reverse proxy services	T1572, T1090.002	Requests to tunnelling, reverse-proxy and anonymiser service domains from hosts with no approved need	High
C2 over legitimate web services	T1102	Scripted user agents or server hosts calling collaboration, paste or code-hosting APIs they never used before. See the Cloud Services Abuse for C2 Guide.	Low to medium
Domain fronting	T1090.004	With inspection: TLS SNI host differs from the HTTP Host header in the same transaction	High where visible
Algorithmically generated domains	T1568.002	Bursts of requests or denials to long, high-entropy, never-seen domains from one host; pair with DNS logs. See the DNS Logging and Detection Guide.	Medium
Tool download	T1105	Executable, script, DLL or archive content types from a rare or newly seen domain, especially to servers or by a non-browser user agent	Medium to high
Remote access tool sessions	T1219	First connection from a host to a remote access or RMM vendor's relay domains where that tool is not approved	High

Hunting by stack counting

Least frequent user agents: count distinct hosts per user agent over 30 days and review those seen on one to three hosts; scripting libraries, outdated browser builds and malformed strings stand out.
Least frequent domains by server: servers browse very little, so list every domain a server segment reached that no other server reached.
First seen per organisation: domains contacted for the first time today, joined to category (uncategorised or newly registered) and to endpoint process data.
Long-lived CONNECT sessions: sort tunnels by duration and by low steady byte rate; interactive C2 tends to hold sessions open.
JA4 rarity: where TLS client fingerprints are logged, list fingerprints seen on few hosts and compare them with the claimed user agent. JA4 TLS client fingerprinting is BSD 3-Clause licensed; the wider JA4+ methods use the FoxIO License 1.1, which permits internal business use, so check licensing before building them into a product.

Detections for exfiltration and risky browsing

Behaviour	ATT&CK	Detection logic	Fidelity
Large upload to cloud storage	T1567.002	Client-to-server bytes per user per storage or file-sharing domain above the user's 30-day baseline, especially to personal or unsanctioned tenants	Medium
Upload-heavy session to a rare host	T1567, T1041	Upload to download ratio far above normal for the host, sustained over minutes	Medium
Paste and code-sharing uploads	T1567	POST requests to paste sites or new code repositories from servers or privileged users	Medium
Unencrypted exfiltration	T1048.003	Large POST or PUT bodies over plain HTTP to external hosts	High
Phishing click to download	T1566.002, T1204.001	Request with an email or webmail referrer followed within minutes by an archive or script download from a new domain	Medium
Drive-by and malvertising chains	T1189	Redirect chains through several new domains ending in a file download, or a block on a known exploit or malware category	Medium
Repeated policy blocks	None (policy signal)	One user or host hitting malware, phishing or command and control categories repeatedly; a single block is noise, a pattern is a lead	Medium
Unsanctioned AI and SaaS use	None (policy signal)	Uploads to AI services or SaaS tenants outside the approved list. See the Shadow AI Discovery Guide.	Medium

For the volume patterns and tooling behind these rows, see the Data Exfiltration Guide, the T1567 Exfiltration Over Web Service page and Detecting Rclone and File-Sync Exfiltration Tooling.

Correlating with DNS, endpoint and identity evidence

DNS: a domain resolved but never requested through the proxy suggests a bypass path or a non-HTTP protocol; a proxy request for a domain with no internal DNS query suggests the client resolved it elsewhere.
Endpoint: join on host and time to the process that made the request. A browser fetching an archive is ordinary; a scripting engine or office application doing so is not. See the Zeek Network Monitoring Guide for network-side metadata that complements the proxy.
Identity: join the proxy username to sign-in logs. Browsing as a user who is on leave, or from a host the user has never signed in to, points to stolen credentials or session theft.
Firewall: allowed direct web egress from a host that also uses the proxy shows the bypass in use. See the VPN and Firewall Log Detection Guide.

Where identity fits

Servers, build agents, scripts and AI agents use the proxy too, and their traffic is where the clearest signals sit, because machine behaviour is narrow and repetitive. Give each workload its own proxy credential or certificate rather than a shared service account, so a detection can name the workload; then baseline the destinations each one needs and alert on anything new. Watch query strings for API keys and tokens passed in URLs, which end up in proxy logs and must be redacted or access-controlled, and rotate any credential found there. Infostealer infections often show first as an archive download followed by uploads to a new host; see the Infostealer Threat Guide for the credential side.

Rolling out detections

Confirm coverage: list every egress path, confirm each one reaches the proxy or gateway, and confirm the firewall blocks direct web egress elsewhere.
Fix the fields: enable authentication, full URL logging for inspected traffic, user agent, referrer, both byte directions, content type and policy category; forward over an encrypted channel to the SIEM.
Normalise: map to Sigma or OCSF names, then send test requests (a known user, a CONNECT to a test port, a small upload) and check every field lands correctly.
Baseline for 30 days: domains per server segment, user agents per host group, upload volumes per user, and normal categories.
Enable high-fidelity rules first: CONNECT to unusual ports, tunnelling services, unapproved remote access tools, unencrypted large uploads and direct web egress.
Run behavioural rules as hunts: rarity, beaconing and upload ratio rules go to a hunting queue until false positives are understood, with allowlists that expire.
Roll back safely: version parsers and rules so a product update that changes the log format means reverting one parser, not disabling detections.

Decision guidance

Inspect or not: decrypting everything gives the best detection data but raises privacy, pinning and performance problems; most organisations inspect uncategorised, newly registered and high-risk categories, exempt health and banking, and log metadata for the rest.
Full logs or summaries: summarised logs cut cost but lose per-request timing, which beaconing and tool-download rules need; keep per-request logs for servers and privileged users at minimum.
Small organisations: start with five rules: CONNECT to unusual ports, tunnelling or anonymiser services, unapproved remote access tools, executable downloads by servers and large uploads to unsanctioned storage.
Large estates: normalise every proxy and gateway into one schema, baseline per segment and per workload, and use the proxy username as the join key into identity analytics.
Proxy or endpoint telemetry: endpoint data shows the process; proxy data shows the full destination and works for unmanaged devices on the network. Mature programmes use both.

Measuring whether it works

Egress coverage: share of outbound web connections, from firewall data, that crossed the proxy or gateway (target close to 100 for server segments).
Attribution rate: share of proxy events with a username or named workload.
Parse completeness: share of events with host, user agent, both byte counts and action populated.
Inspection share: share of HTTPS bytes to uncategorised and newly registered destinations that were inspected.
Time to detect: minutes from a scheduled test (CONNECT to a test port, upload to a test storage tenant) to alert.
Retention: months of proxy logs searchable (target at least 12).

Common failure patterns

Logging only blocks: some deployments keep denied requests and drop allowed ones to save cost, which removes the evidence that matters.
Anonymous proxy access: without authentication, every alert ends at an IP address that DHCP has since reassigned.
Exempting trusted cloud services from logging: collaboration and storage services are exempted from inspection for performance and then also from logging, which hides the most common exfiltration route.
Servers allowed straight out: server segments reach the internet directly "temporarily" and the proxy never sees the C2 traffic from the hosts that matter most.
Swapped byte fields: a parser maps client-to-server and server-to-client bytes the wrong way round, so upload rules never fire.
Unprotected raw URLs: query strings with tokens and personal data are searchable by everyone with SIEM access.

Framework mapping

Framework	Reference	Relevance
NIST CSF 2.0	PR.PS-04, DE.CM-01, DE.AE-02, DE.AE-03	Log generation, network monitoring, analysis and correlation
NIST SP 800-53 Rev. 5	SC-7(8), SC-7, AC-4, AU-2, AU-6, AU-9, AU-11, SI-4	Traffic through authenticated proxies, boundary protection, flow enforcement, logging, review, log protection, retention, monitoring
ISO/IEC 27001:2022 Annex A	8.15, 8.16, 8.23	Logging, monitoring activities, web filtering
MITRE ATT&CK v20	T1071.001, T1572, T1090.002, T1090.004, T1102, T1105, T1219, T1567.002, T1041, T1048.003	Techniques visible in proxy and gateway logs
MITRE D3FEND	D3-UA URL Analysis, D3-NTA Network Traffic Analysis, D3-PHDURA Per Host Download-Upload Ratio Analysis, D3-PMAD Protocol Metadata Anomaly Detection, D3-OTF Outbound Traffic Filtering	Defensive analytics and controls the rules implement

Practitioner checklist

Block direct web egress except from the proxy and alert on any allowed bypass.
Require authentication for users and give each workload its own proxy identity.
Log allowed and denied requests with host, URL, method, user agent, referrer, both byte directions, content type and category.
Forward logs off the proxy over an encrypted channel in UTC with millisecond timestamps.
Map every product to one schema (Sigma proxy fields or OCSF HTTP Activity) and test the mapping.
Enable rules for CONNECT to unusual ports, tunnelling services, unapproved remote access tools and large uploads first.
Hunt monthly on rare user agents, rare server destinations and long-lived tunnels.
Restrict and redact query strings that can carry tokens or personal data.
Keep proxy logs searchable for at least 12 months where privacy rules allow.

Proxy and Web Gateway Log Detection Guide: Hunting C2, Exfiltration and Risky Browsing