Secure Web Gateway and Web Filtering Guide (SWG)

Web filtering only works if every web request actually passes through it, and most organisations have more ways around their gateway than they realise. A secure web gateway (SWG) sits between users and the internet, categorising and scanning traffic, enforcing acceptable use and stopping users from reaching malicious or risky sites. Encrypted DNS, QUIC, personal hotspots, anonymising proxies and unmanaged devices all create paths that skip it. This guide covers SWG functions, deployment choices, category policy and exceptions, protective DNS, privacy limits, bypass routes and measurement. It is for network and security engineers, architects and SOC teams running or replacing a web gateway.

Key takeaways

Coverage beats features. A gateway with excellent scanning that sees most traffic loses to a simpler one that sees all of it. Measure how much traffic reaches the policy point before tuning anything else.
Layer DNS filtering under the proxy. Protective DNS is cheap, covers devices and protocols the proxy never sees, and eligible UK and US public sector organisations can use free services from the NCSC and CISA. It does not replace URL-level inspection.
Without TLS inspection the gateway sees only the host name. Category and reputation checks still work on the server name, but malware scanning, DLP and inline cloud app controls need decryption. See the TLS Inspection Guide.
Close the bypass routes deliberately. Manage browser DoH and QUIC settings, block UDP 443 to the internet where the proxy cannot inspect it, block anonymising and residential proxy services, and decide what happens off-network.
Treat it as monitoring of people. Web logs reveal health, religion, union and personal matters. Run a DPIA, tell staff, minimise what you log and set a retention schedule before switching on full logging.

What a secure web gateway does

An SWG is a forward proxy with a policy engine. Products are delivered as on-premises appliances, virtual machines or cloud services, and the same functions are often bundled into security service edge (SSE) and secure access service edge (SASE) offerings, which NIST SP 800-215 places alongside cloud access security brokers (CASB) and ZTNA in the modern enterprise network. The table lists the functions to expect and what each depends on.

Function	What it does	Depends on
URL categorisation and reputation	Maps each host or URL to categories (gambling, malware, phishing, newly registered domains, anonymisers) and a risk score; drives block, warn or allow	Server name only for HTTPS without decryption; full URL path with decryption
Real-time content classification	Classifies pages the database has not seen by analysing content, scripts and page structure at request time	Decryption; added latency on first visit
Download scanning	Signature, heuristic and file-type checks on downloads; blocks executables, scripts, macros and archives by policy	Decryption; archive and password-protected file handling
Sandboxing	Detonates unknown files in an instrumented environment before or after delivery	Hold-until-verdict or deliver-and-retract policy; size limits
TLS inspection	Terminates and re-encrypts HTTPS so other engines can see content; validates upstream certificates	Trusted interception CA on endpoints; exemption list
Data loss prevention	Inspects uploads and form posts for sensitive data and blocks, warns or logs. See the Data Loss Prevention (DLP) Guide	Decryption; classification labels and detectors
Inline CASB and application control	Identifies cloud apps and actions (upload, share, post, login to a personal tenant) and applies controls per action	Decryption; app signatures kept current
Remote browser isolation (RBI)	Runs the page in a remote container and sends only a safe rendering, often for uncategorised or risky categories	Cost per user; compatibility with complex web apps
Bandwidth and safe browsing controls	Throttles streaming and large downloads, enforces search engine safe search and video restrictions	Decryption for safe search enforcement on some engines

URL reputation lists versus behaviour-based filtering

Reputation databases are fast and predictable but always behind: phishing often runs on fresh domains or compromised legitimate sites that still carry a good category. Behaviour-based or real-time analysis inspects the page itself (login forms that imitate a brand, obfuscated scripts, redirect chains) and catches unseen sites, at the cost of latency, more false positives and a hard dependency on decryption. In practice use both: block known-bad categories and newly registered domains outright, send uncategorised and low-reputation sites to real-time analysis or isolation, and let well-categorised business sites through with scanning.

Deployment models

Model	How traffic arrives	Strengths	Weaknesses
Explicit proxy	Browser or OS configured with a proxy address (commonly TCP 8080 or 3128) by policy, PAC file or WPAD	Clear user authentication; HTTP CONNECT gives the host name even without decryption	Applications that ignore system proxy settings go direct unless the firewall blocks them
Transparent proxy	Firewall or router redirects TCP 80 and 443 to the proxy (policy-based routing, WCCP or inline)	No client configuration; catches proxy-unaware apps	Authentication is harder; non-web ports and UDP are not covered
Endpoint agent or tunnel	Agent on the device steers traffic to a cloud or on-premises gateway on and off network	Follows the user; can enforce fail-closed behaviour; carries device posture	Managed devices only; agent tampering and outages need monitoring
DNS filtering	Devices or resolvers use a filtering resolver; blocked names resolve to a block page or fail	Covers every protocol and IoT; very low cost and latency	No URL path, no content scanning; bypassed by hard-coded or encrypted DNS and direct IPs
Cloud SWG	Agent, PAC file or IPsec/GRE tunnel from the site to the provider's points of presence	Consistent policy for remote users and branches; no appliances to size	Latency and data location depend on the provider; outbound IP addresses change what destination sites see
On-premises SWG	Appliance or VM in the data centre or branch	Logs and keys stay in-house; suits regulated and air-gapped environments	Remote users hairpin through VPN; capacity planning and patching are yours

Choosing: small organisations with mostly remote staff usually start with protective DNS plus a cloud SWG agent on managed laptops. Large organisations combine an agent for laptops, a transparent or tunnel path for sites and servers, and DNS filtering underneath, keeping on-premises gateways where regulation, latency or sovereignty require them. If remote access is moving to a cloud service, see the ZTNA Guide for how private application access and internet access share the same agent.

PAC files and WPAD

A proxy auto-configuration file is a JavaScript function, FindProxyForURL(url, host), that returns DIRECT, PROXY host:port or SOCKS host:port, with semicolon-separated fallbacks. Serve it with the MIME type application/x-ns-proxy-autoconfig. Keep it short and string-based: isInNet() and dnsResolve() trigger DNS lookups for every request and slow browsing. Browsers strip the path and query from HTTPS URLs before passing them to the function, so a PAC file can only route HTTPS by host name. A PROXY gw:8080; DIRECT fallback quietly sends traffic straight out when the gateway is down; use it only if fail-open is a decision you have made. Web Proxy Auto-Discovery (WPAD), which finds the PAC file through DHCP option 252 or a DNS lookup for a host named wpad, can be hijacked by anyone able to answer those lookups on the local network. Prefer a PAC URL pushed by device management, disable automatic proxy detection where it is not used, and register or block the wpad name in internal DNS.

Protective DNS

Protective DNS (PDNS) is a recursive resolver that refuses to resolve known malicious domains. It is the cheapest web control to deploy and the one that reaches devices a proxy never sees: printers, IoT, servers and guest networks. CIS Controls v8.1 safeguard 9.2 asks for DNS filtering on all enterprise assets.

UK: the NCSC PDNS service is a "recursive DNS resolver which prevents access to domains known to be malicious". It is mandated for central government departments by the Cabinet Office, free to eligible organisations, holds its blocking rules in a Response Policy Zone (RPZ), shows a block page, can feed block data to a SIEM, and offers PDNS Roaming, which sends DNS from remote Windows, macOS and iOS devices to the service over DNS over HTTPS.
US: the CISA Protective DNS Resolver, introduced in 2022, filters queries upstream of federal civilian agency networks against threat intelligence and blocks, redirects or sinkholes matches. It is mandated for federal civilian executive branch agencies and open to critical infrastructure entities on a limited basis through a pilot, at no cost.
Everyone else: commercial PDNS and the DNS layer of most SWG services work the same way. Point internal resolvers at the protective service as forwarders, block outbound TCP and UDP 53 and DNS over TLS (TCP 853) from everything except those resolvers, and log queries centrally. See the DNS Security Guide for RPZ, DoH policy and DNS logging.

Designing category policy and exceptions

Most organisations need three tiers, short enough to explain to a new starter.

Always block (security): malware, phishing, command and control, cryptomining, newly registered or newly observed domains, parked domains, anonymisers and proxy avoidance, peer-to-peer, dynamic DNS, and categories that are illegal in your jurisdiction.
Block or warn (acceptable use and data risk): gambling, adult, hacking tools, personal webmail and file sharing, unsanctioned generative AI tools, online code formatters and paste sites. A warn page with a reason field ("caution" or "coach") changes behaviour without creating tickets. The Samsung ChatGPT source code leak and the JSONFormatter and CodeBeautify leak show why paste-style sites deserve their own rule. See the Shadow AI Discovery Guide.
Allow with scanning: everything else, with download scanning, file-type controls and DLP. Treat uncategorised sites as a risk tier of their own: isolate them, or allow read-only browsing and block downloads and form submissions.

File types: block executables, scripts, disk images (ISO, IMG, VHD), HTML application and shortcut files from uncategorised and low-reputation sites, in line with CIS safeguard 9.6. Safe browsing: enforce safe search on search engines and video sites where your policy requires it, especially in education. Bandwidth: throttle streaming categories at sites with limited links rather than blocking them.

Exceptions: run them through a ticket with a named requester, business reason, scope (user, group or device rather than everyone), approver and an expiry date, normally 30 to 90 days. Recategorisation requests go to the provider; local overrides should be the exception. Review exceptions monthly and remove expired ones automatically. Exempting a domain from TLS inspection is a separate decision from allowing it, so record both.

Bypass routes and how to close them

Bypass	How it works	Control
Browser DNS over HTTPS	Browser resolves names over HTTPS (RFC 8484) to a public resolver, skipping protective DNS. ATT&CK T1572 Protocol Tunneling notes DoH can also hide C2 lookups	Set browser DoH policy to off or to your resolver; block known public DoH endpoints at the proxy; see the DNS Security Guide
QUIC and HTTP/3	Browsers prefer QUIC over UDP 443, which many proxies and firewalls cannot inspect	Disable with browser policy (Chrome `QuicAllowed` set to Disabled) and block outbound UDP 443 so clients fall back to TCP
Proxy-unaware applications	Apps ignore system proxy settings and connect directly	Default-deny outbound web traffic, SOCKS and non-standard ports at the firewall except from the gateway; use an agent or transparent mode
Anonymising and residential proxies, VPN apps, Tor	Traffic tunnels to an external relay (ATT&CK T1090.002 External Proxy, T1090.003 Multi-hop Proxy). The FBI warns residential proxy networks are "a standard tool criminals use to look like ordinary users online"	Block proxy avoidance and anonymiser categories, known residential proxy IP ranges and Tor; application control for VPN clients; alert on long-lived connections to unknown hosts
Domain fronting	TLS server name shows an allowed host while the HTTP Host header targets another (T1090.004)	TLS inspection that compares the SNI with the Host header (ATT&CK mitigation M1020)
Personal hotspots and home networks	Laptop leaves the corporate network and the on-premises gateway	Always-on agent with fail-closed policy; block tethering where your device management allows it; DNS roaming
Unmanaged and BYOD devices	No agent, no certificate, no proxy settings	Separate network with DNS filtering only; reach corporate SaaS through a managed or enterprise browser. See the Enterprise Browser and Browser Security Guide

Privacy and employee monitoring

Web logs are personal data about workers and often include special category data (health sites, union pages, religious content). This section is not legal advice; work with your data protection officer. In the UK, the ICO's guidance on monitoring workers (under review following the Data (Use and Access) Act) says "You must tell workers about monitoring in a way that is accessible and easy to understand", that you "must be clear about your purpose and select the least intrusive means to achieve it", and that you "must ensure you have a retention schedule". It expects a DPIA before high-risk monitoring.

Minimise: log blocked and security events in full; consider logging allowed traffic at domain level only, or pseudonymising user names in the reporting tier.
Exempt sensitive categories from decryption: health, banking and personal finance are the usual list.
Separate roles: analysts should not browse a named person's history without a recorded case.

See the DPIA Guide: Running a Data Protection Impact Assessment and the Insider Threat Programme Guide.

Where identity fits

Policy is only as good as the identity attached to each request. Authenticate users through your identity provider (SAML or the agent's device certificate and user token), not by source IP, which breaks with shared and NAT addresses. Servers, CI runners, scripts and AI agents also call the web, and often get a blanket bypass or a shared proxy account whose password sits in configuration files. Give each workload its own allowlist of destinations, authenticate it with a per-workload credential or certificate, and log it under that identity so an unexpected destination stands out. NIST SP 800-53 SC-7(8) asks for outbound traffic to be routed through authenticated proxy servers. The gateway's own admin console and API keys can change policy for the whole organisation: enforce phishing-resistant MFA and scope API tokens. See the Service Account Security Guide.

Rolling out a web gateway

Baseline: from a week of firewall, DNS and proxy logs, identify top destinations, uncategorised volume, direct egress, QUIC and DoH usage.
Deploy DNS filtering first: forward internal resolvers to protective DNS in logging mode for a week, review blocks for false positives, then enforce and restrict outbound DNS.
Pilot ring: deploy the agent or PAC file to IT and security staff with security categories enforced, acceptable use categories in monitor mode and TLS inspection on a small category set.
Tune: build the decryption exemption list (pinned apps, health and finance), fix broken business apps and document every exception with an owner and expiry.
Expand in waves: 10%, 50%, then 100% of managed devices, with a published support route and a warn page that explains how to request access.
Close bypasses: push browser DoH and QUIC policies, block UDP 443 and direct TCP 80 and 443 egress, block outbound DNS other than resolvers, and switch the agent to fail-closed once stability is proven.
Roll back safely: keep the previous PAC file and agent policy versioned; a rollback switches steering to the previous gateway or to DNS filtering only, never to unfiltered direct access without an approved, time-limited decision.

Detection and measurement

Send gateway logs to the SIEM with timestamp, user, device, source IP, destination host and URL (or host only, per your privacy decision), category, risk score, action, policy rule, bytes up and down, file hash and verdict for downloads, and whether the session was decrypted. Alert on the following:

Agent disabled, uninstalled or reporting fail-open; devices that stop sending traffic while still active in device management.
Large uploads to personal storage, paste sites or unsanctioned AI tools.
Repeated blocks to command and control or newly registered domains from one host, which usually indicates infection rather than browsing.
Connections to anonymiser, residential proxy or Tor infrastructure, and beacon-like regular requests to one host.

Metrics that show it is working:

Coverage: percentage of managed devices with a healthy agent or proxy configuration, and percentage of outbound web traffic seen by the gateway versus firewall egress logs (set a target close to 100% and investigate the remainder).
Inspection rate: percentage of HTTPS traffic decrypted and scanned, excluding approved exemptions.
Bypass volume: direct UDP 443, outbound DNS and public DoH attempts blocked per week, trending down.
Exception hygiene: open exceptions, percentage past expiry (target zero) and median age.
Threat outcomes: malicious downloads and phishing pages blocked, and time from first block to investigation of the host.

Framework mapping

Framework	Reference	What it covers
CIS Controls v8.1	9.2, 9.3, 9.4, 9.6	DNS filtering services, network-based URL filters, restricting browser extensions, blocking unnecessary file types
ISO/IEC 27001:2022 Annex A	A.8.23, A.8.16, A.8.12	Web filtering, monitoring activities, data leakage prevention
NIST SP 800-53 Rev. 5	SC-7, SC-7(8), SI-3, SI-4, AU-12	Boundary protection, routing traffic to authenticated proxy servers, malicious code protection, system monitoring, audit record generation
NIST CSF 2.0	PR.IR-01, DE.CM-01	Networks protected from unauthorised logical access and usage; networks and network services monitored
MITRE ATT&CK	T1090 (.002, .003, .004), T1572	External and multi-hop proxies, domain fronting, protocol tunnelling including DoH
ATT&CK mitigations	M1037, M1031, M1020	Filter network traffic, network intrusion prevention, SSL/TLS inspection

Common failure patterns

Fail-open by default: a PAC file with a DIRECT fallback or an agent that disconnects on error means every outage, and every attacker who kills the agent, gets unfiltered access.
Proxy without egress control: the firewall still allows direct TCP 443 and UDP 443 from user networks, so malware and proxy-unaware tools ignore the gateway entirely.
Server and automation bypass lists that only grow: whole subnets of servers and build agents exempted years ago, now with unrestricted internet access and no logging.
Uncategorised treated as allowed: new phishing and malware domains are uncategorised by definition; allowing them with no isolation or download block removes most of the protection.
Logging everything forever: full URL logs for all staff with no DPIA, no retention schedule and broad analyst access create a privacy liability and a sensitive dataset attackers would value.

Practitioner checklist

Measure gateway coverage against firewall egress logs and fix gaps before tuning policy.
Deploy protective DNS for all assets and block outbound DNS, DoT and public DoH except from your resolvers.
Push browser policies for DoH and QUIC, and block outbound UDP 443 where the gateway cannot inspect it.
Default-deny direct outbound web traffic at the firewall except from the gateway and approved workload paths.
Publish a three-tier category policy with warn pages, and isolate or restrict uncategorised sites.
Run exceptions through tickets with owners and expiry, and review them monthly.
Give servers, CI runners and AI agents per-workload destination allowlists and credentials instead of blanket bypass.
Complete a DPIA, inform staff, exempt sensitive categories from decryption and set a log retention schedule.
Alert on agent tampering, fail-open states, anonymiser use and large uploads to unsanctioned services.
Version PAC files and policies, and test rollback to a filtered fallback, never to direct access.

Secure Web Gateway and Web Filtering Guide