Every Linux server runs code as root from whichever repositories its package manager trusts, so a badly scoped or stale signing key decides whether a supply chain attack lands on one host or on the whole fleet. APT and RPM both verify signatures, but they verify different things, they trust keys in different ways, and a single trusted=yes, gpgcheck=0 or globally trusted vendor key quietly removes the protection. This guide explains the chain of trust in APT and DNF, how to onboard third-party repositories with keys scoped to one source, how to run internal mirrors and re-sign safely, how to rotate keys, and what signing cannot protect against. It is for Linux platform teams, build and release engineers and security architects responsible for software supply chain controls. For signing your own artefacts more broadly, see the Code Signing Security Guide.
Key takeaways
- APT signs the repository, RPM signs the package. apt-secure verifies a signed Release file that hashes Packages files that hash each .deb; DNF checks a signature on each RPM and only checks metadata when
repo_gpgcheckis on. - Scope every third-party key to its repository. A key dropped into
trusted.gpg.dcan sign for any APT source; useSigned-Bywith a keyring in/etc/apt/keyrings. In RPM land, imported keys are global, so keep the keyring small. - The insecure switches are the real risk.
trusted=yes,allow-insecure,gpgcheck=0,sslverify=0and--nogpgcheckin scripts are what attackers rely on, and they are easy to find with a fleet-wide grep. - Internal mirrors move the trust root to you. If you re-sign, your signing key now protects every server; hold it in an HSM or isolated signer and treat the mirror service account as privileged.
- Signatures prove origin, not intent. The xz-utils backdoor came through a legitimate maintainer and a legitimate release; staged rollouts, provenance and reproducible builds cover what signing cannot.
How repository trust works in APT and RPM
| Aspect | APT (Debian, Ubuntu) | DNF / RPM (RHEL, Fedora, SUSE zypper is similar) |
|---|---|---|
| What is signed | The Release file (InRelease clear-signed, or Release plus detached Release.gpg) | Each RPM header and payload; optionally repository metadata (repomd.xml) |
| How a package is trusted | Its hash is listed in a Packages file whose hash is in the signed Release file | Its own signature verifies against an imported key |
| Individual package signature | Not checked by apt-secure (debsig-verify exists but is rarely used) | Checked when gpgcheck=1; localpkg_gpgcheck for files installed from disk |
| Where keys live | Keyring files: /usr/share/keyrings (package managed), /etc/apt/keyrings (operator managed), legacy trusted.gpg and trusted.gpg.d | The RPM database keyring (imported with rpmkeys --import, listed with rpm -qa gpg-pubkey*), key files often in /etc/pki/rpm-gpg |
| Key scope | Per source with Signed-By; global without it | Global for package signatures; repo_gpgcheck keys are stored per repository |
| Replay and freeze protection | Valid-Until in Release, enforced by Check-Valid-Until (default yes) | Metadata expiry (metadata_expire, default 48 hours) controls refresh, not signed freshness |
| Unsigned source behaviour | Refused by default; downgrade from signed to unsigned raises an error | Installs if gpgcheck=0; upstream DNF default is False, distributions set it to 1 |
The Debian apt-secure manual is explicit about the limits: Release signing protects against network man-in-the-middle attacks and compromised mirrors, but not against compromise of the master server that signs, or theft of the Release signing key. RPM signatures have the same property: they prove a key holder signed, nothing more.
Distribution signing keys
- Debian: stable releases are signed by the ftp-master automatic archive key and a per-release stable key; other suites and the security archive by the ftp-master key only. Debian now uses one ftp-master key per release and announces replacements on debian-devel-announce in advance.
- Red Hat: RHEL packages are signed with release key 2 (fingerprint
567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51), shipped in/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release. Red Hat added release key 3 in 2024 for sigstore container signing and release key 4, a post-quantum ML-DSA-87 plus Ed448 key, for products released after October 2025 (RHEL 9.7 and 10.1). Auxiliary keys exist for disaster recovery only. - Practical point: keep a record of expected fingerprints for every distribution and vendor key you trust, taken from the publisher's page, so drift can be detected automatically.
Onboarding a third-party repository
Vendor install instructions frequently pipe a key into a global keyring or use apt-key add, which is deprecated and was only kept until Debian 11 and Ubuntu 22.04. Use a standard procedure instead.
- Get the key over a channel you trust: HTTPS from the vendor's own domain, and compare the fingerprint with a second source such as the vendor's documentation page or support portal.
gpg --show-keysdisplays the fingerprint without importing. - APT: store it scoped. Put the dearmored keyring in
/etc/apt/keyrings/vendor.gpg(readable by the_aptuser) and reference it from a deb822 file, for example avendor.sourcesstanza withTypes: deb,URIs: https://packages.vendor.example/apt,Suites: stable,Components: mainandSigned-By: /etc/apt/keyrings/vendor.gpg. Signed-By can also hold a fingerprint, and a trailing!on the fingerprint pins the exact key rather than accepting its subkeys. - APT: pin what the repository may supply. Use apt preferences (
/etc/apt/preferences.d) so a vendor repository can only provide its own packages and cannot replaceopensslorlibc6with a higher version. - DNF: write the .repo file explicitly with
gpgcheck=1,repo_gpgcheck=1where the vendor signs metadata,gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-vendor,sslverify=1andincludepkgs=listing the packages you expect. Note thatdnf config-manager --add-repoenables the new repository immediately, so review the generated file. - DNF: control precedence.
priority(default 99, lower wins even with an older version) lets distribution repositories win over third-party ones for overlapping packages. - Record the decision: owner, purpose, key fingerprint, expiry and review date in the configuration management repository that deploys the source.
Choosing a trust model
| Model | How it works | Good for | Trade-off |
|---|---|---|---|
| Direct vendor repositories | Each server talks to distribution and vendor repositories with their keys | Small estates, cloud images with internet egress | Many keys to manage; every server needs egress; no staging |
| Caching proxy or pass-through mirror | Internal mirror keeps upstream signatures unchanged | Most organisations: one egress point, upstream keys still verified on the client | No control over when upstream changes reach clients unless snapshots are used |
| Snapshotted mirror | Mirror freezes dated snapshots (for example with Pulp, Red Hat Satellite content views, aptly) that move through test and production | Regulated and large estates that need staged rollout | For APT, a re-published snapshot needs a new Release signature from your key |
| Re-signed internal repository | Your own key signs repository metadata or packages after review | Air-gapped networks, internally built packages | Your signing key becomes the most valuable credential in the Linux estate |
For small organisations, a pass-through mirror with upstream signatures and a short list of approved vendor repositories is usually enough. Large estates benefit from snapshots because they turn "a malicious version appeared upstream" into "it reached the test ring first".
Internal mirrors, re-signing and the signing key
- Isolate the signer: keep the repository signing key in an HSM, a cloud KMS that supports OpenPGP signing through a signing service, or an offline signer. The mirror sync job should request signatures, never hold the private key on disk.
- Treat the pipeline identity as privileged: the service account or CI token that can publish to the internal repository can push root-level code to every server. Give it a dedicated identity, no interactive login, short-lived credentials and alerting on publication outside the schedule. See the Service Account Security Guide.
- Verify before re-signing: the sync must validate upstream signatures first; re-signing unverified content launders it.
- Keep Valid-Until meaningful: when you sign APT snapshots, set an expiry so a stale or frozen mirror is detected by clients rather than silently serving old, vulnerable packages.
- Protect the transport too: HTTPS with
sslverify=1and, where supported, client certificates (sslclientcert) for entitlement-protected repositories. TLS does not replace signatures, but it prevents metadata freeze and enumeration by an on-path attacker.
Key lifecycle and rotation
| Event | APT action | RPM action |
|---|---|---|
| Planned rotation by a vendor | Install the new keyring alongside the old one in the Signed-By file before the switch date; remove the old key afterwards | Import the new key, confirm new packages verify (rpm --checksig -v), then remove the old key with rpmkeys --delete |
| Vendor key compromised | Remove the key from the keyring, block the repository, review packages installed from it since the exposure window | Remove the key, run rpm -qa --qf '%{NAME} %{SIGPGP:pgpsig}\n' style reports to find packages signed by it, reinstall from re-signed sources |
| Your internal key rotates | Publish the new key in a keyring package first, then switch signing | Distribute the new key through configuration management before signing with it |
| Repository retired | Delete the .sources file and its keyring | Delete the .repo file and the gpg-pubkey entry |
The HashiCorp case shows what a real rotation involves. In 2021 the GPG key HashiCorp used to sign SHA256SUMS files for product downloads was exposed through the altered Codecov uploader running in its CI, and the company revoked the key, re-validated and re-signed releases and shipped Terraform releases that trusted the new key. HashiCorp stated its Linux package signing was not affected. See the HashiCorp GPG Key Exposure 2021. The lesson for repository owners is that a signing key stored as a CI environment variable is one compromised build step away from exposure.
What signing does not protect
- Malicious but legitimately signed releases: if the upstream maintainer, the build system or the distribution's signer is compromised, the signature is valid. This is ATT&CK T1195.002 Compromise Software Supply Chain.
- Tarball and source divergence: in the xz-utils case the payload sat in release tarballs and build macros rather than the reviewed git source. Distributions packaged and signed it correctly; signing could not tell them the input was hostile. It reached development and rolling releases (Fedora Rawhide and 40 beta, Debian unstable, Kali, openSUSE Tumbleweed and MicroOS, Arch) and was caught by a performance anomaly. See the XZ Utils Backdoor 2024.
- Packages installed outside the package manager:
curl | bashinstallers, language package managers (pip, npm), container images and binaries copied by configuration management bypass APT and RPM entirely. See the Malicious Package and Dependency Confusion Guide. - Post-install tampering: signatures are checked at install time. A replaced binary afterwards is a file integrity problem (ATT&CK T1554 Compromise Host Software Binary);
rpm -Vanddebsumscompare installed files with package hashes.
Complementary controls are a staged rollout from snapshots, keeping production on stable rather than rolling or development suites, SBOMs for what is installed, and reproducible builds, which the Reproducible Builds project defines as builds where any party can recreate bit-by-bit identical artefacts from the same source, environment and instructions. Debian and other distributions use them to check that a binary matches its source.
Detection and measurement
| What to look for | Where | Why it matters |
|---|---|---|
trusted=yes, allow-insecure, allow-weak, [trusted=yes] in one-line sources | /etc/apt/sources.list, sources.list.d | Disables verification for that source |
gpgcheck=0, repo_gpgcheck=0 on third-party repos, sslverify=0 | /etc/dnf/dnf.conf, /etc/yum.repos.d | Unsigned or unverified content accepted |
--nogpgcheck, --allow-unauthenticated, --force-yes | Scripts, CI jobs, cloud-init, Ansible tasks | Verification bypassed at install time |
Keys in trusted.gpg or trusted.gpg.d from non-distribution vendors | APT keyrings | Global trust for a vendor key |
| New or unexpected gpg-pubkey entries | rpm -qa gpg-pubkey* compared with an approved list | Unapproved trust anchor added |
| Writes to keyring and repo directories | auditd or FIM on /etc/apt, /etc/yum.repos.d, /etc/pki/rpm-gpg | Repository hijack attempt |
| Package verification failures | rpm -Va, debsums -c | Post-install tampering |
Metrics
- Insecure source count: hosts with any insecure repository option; target zero.
- Scoped key coverage: percentage of third-party APT sources using
Signed-By. - Approved trust anchors: percentage of hosts whose RPM keyring matches the approved fingerprint list.
- Snapshot lag: days between upstream publication and production promotion, tracked so security updates still meet patch SLAs. See the Patch Management Guide.
- Repository inventory freshness: percentage of third-party repositories with a named owner and a review in the last 12 months.
Framework mapping
| Control | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 | ATT&CK |
|---|---|---|---|
| Only install signed components | CM-14 Signed Components | PR.PS-02 | T1195.002 |
| Verify origin of software and mirrors | SR-4 Provenance, SR-11 Component Authenticity | PR.PS-02 | T1195.002 |
| Repository configuration under change control | CM-6 | PR.PS-01 | T1553 Subvert Trust Controls |
| Installed file integrity | SI-7 | DE.CM-09 | T1554 |
| Timely updates through staged mirrors | SI-2 | ID.RA-01 | T1190 |
Common failure patterns
- Copy-pasted vendor instructions: a key added to
trusted.gpg.dlets that vendor's key sign for the Debian or Ubuntu archive too. Convert every such key to a scopedSigned-Bykeyring. - gpgcheck=0 "temporarily": set to get an install working during a build and never reverted, or set in a golden image so every cloned server inherits it.
- Mirrors without staging: a pass-through mirror delivers a malicious upstream release to production as fast as to test; snapshots with a short test window catch problems before the fleet.
- Signing keys in CI variables: a repository signing key in an environment variable is exposed by any compromised build step, as happened to HashiCorp's download signing key.
- Development suites in production: rolling and unstable suites are where xz-utils 5.6.0 and 5.6.1 landed; production should track stable releases.
- Expired keys handled by disabling checks: when a vendor key expires, teams switch verification off instead of importing the renewed key. Track expiry dates and alert 60 days ahead.
Practitioner checklist
- Inventory every repository and trusted key on every host class, with owner and expected fingerprint.
- Move all third-party APT keys out of
trusted.gpgandtrusted.gpg.dinto/etc/apt/keyringswithSigned-By. - Enforce
gpgcheck=1globally andrepo_gpgcheck=1where the vendor signs metadata; useincludepkgsand priorities for third-party repos. - Grep scripts, images and CI for
--nogpgcheck,--allow-unauthenticatedandtrusted=yes, and fail builds that contain them. - Route servers through an internal mirror with dated snapshots and a test ring before production.
- Hold any internal signing key in an HSM or isolated signer and give the publishing pipeline its own monitored identity.
- Monitor repository and keyring directories for changes and run
rpm -Vaordebsumson a schedule. - Write a key compromise runbook: remove the key, block the source, find packages it signed, reinstall from trusted sources.
Standards and references
- Debian: apt-secure(8) archive authentication support for APT
- Debian: sources.list(5) and the Signed-By option
- DNF: Configuration reference (dnf.conf)
- RPM: rpmkeys(8) keyring operations
- Red Hat: Product signing keys
- Debian: Archive signing keys
- NIST: SP 800-53 Rev. 5
- Reproducible Builds: Definitions
Related NHI Mgmt Group resources: Code Signing Security Guide · Software Supply Chain Security Guide · SBOM Guide · Linux Server Hardening Guide · Supply Chain Compromise Response Playbook