BIND 9.16 Root DNS (Hidden Primary + Anycast Roots) — DISA STIG Hardened Deployment
A complete, production-tested DISA STIG hardening guide for an authoritative ROOT DNS infrastructure built on BIND 9.16 on Oracle Linux 8. Covers hidden-primary + N anycast root instances, TSIG-protected zone transfers (hmac-sha384), DNSSEC signing of the root zone, FRR/BGP anycast advertisement with per-node ASNs (RFC 6382) and health-driven withdraw, federated-tolerant SOA timers, systemd hardening, and SIEM forwarding — with every OL8-specific pitfall documented.
Air-gapped / disconnected network? See README-airgapped.md.
- Root authoritative only — recursion fully disabled on every host; serves the root zone
.(plus co-hostedin-addr.arpaand optionalip6.arpa); never answers recursive queries - Hidden-primary topology — single primary holds the root-zone source-of-truth, never on the anycast routing path, answers normal queries only from a tight management ACL
- Anycast root instances — N identical secondaries, each bound to a node-specific unicast IP AND the shared anycast service IP; per-node ASN per RFC 6382; FRR/BGP withdraws the anycast prefix automatically when the local instance is unhealthy
- Health-driven advertise/withdraw —
anycast-health.shprobesdig @127.0.0.1 . SOAand adds/removes the anycast IP fromlo; FRR'sredistribute connectedconverts that into a clean BGP announce/withdraw, so no node ever advertises a service it cannot answer - TSIG-protected zone transfers — hmac-sha384 mandatory; same key installed identically on the primary and every anycast instance; outbound transfers from anycast nodes are sourced from the node's unicast IP (never anycast)
- NOTIFY-driven propagation — primary sends NOTIFY on every root-zone change; anycast instances pick up the change within seconds via fast AXFR/IXFR
- Unicast source addressing —
query-source,notify-source,transfer-sourcepinned to the node's unicast IP on every host; the anycast address is reply-source only (the kernel mirrors the destination IP back, which is exactly the behaviour root anycast requires) - DNSSEC via dnssec-policy — root zone signed with automated KSK/ZSK rollover and re-signing on zone change, signature expiry, and key boundary events. Algorithm set restricted to RSASHA256, RSASHA512, ECDSAP256SHA256, ECDSAP384SHA384
- TLD delegation model — root delegates ONLY top-level domains; signed children publish DS records here; unsigned children get NS records only (parent NS stays signed)
- Federated-tolerant SOA timers — short refresh (1 h) so anycast instances pick up changes fast, long expire (60 d) so a partitioned instance stays usable for two months of disconnection
- Referral payload ≤ 512 octets — minimal-responses, conservative NS RRset (two records), glue A records matching authoritative A records
- Structured logging — separate channels for zone transfers, NOTIFY, dynamic updates, DNSSEC events, security denials, query errors; all categories forwarded to a SIEM via rsyslog (facility
local5) - systemd hardening drop-in — capabilities reduced to
CAP_NET_BIND_SERVICE, core dumps disabled, ReadWritePaths pinned to actual zone/log paths; OL8/systemd-239 compatible - Documented pitfalls — every non-obvious OL8/BIND-9.16/FRR-anycast failure mode explained and solved
Downstream recursive resolvers
(root.hints points here — see config/named.root)
│ UDP/TCP 53 → 198.51.100.1
▼
┌─────────────────────────────────────────────────────────────┐
│ ANYCAST PREFIX 198.51.100.1/32 (advertised from N nodes) │
├─────────────────────────────────────────────────────────────┤
│ ROOT-ANYCAST-1 unicast 192.0.2.21 ASN 4200000001 │
│ ROOT-ANYCAST-2 unicast 192.0.2.22 ASN 4200000002 │
│ ROOT-ANYCAST-N unicast 192.0.2.23 ASN 420000000N │
│ ----- │
│ - allow-query { any; } │
│ - listen-on { unicast; anycast; } │
│ - outbound source = UNICAST (never anycast) │
│ - FRR advertises 198.51.100.1/32 only while named is OK │
│ - SOA refresh 1h / expire 60d │
└─────────────────┬───────────────────────────────────────────┘
│ AXFR / IXFR (TCP 53, TSIG hmac-sha384)
│ unicast-to-unicast — never via anycast
▼
┌─────────────────────────────────────────────────────────────┐
│ HIDDEN PRIMARY 192.0.2.10 │
│ ----- │
│ - allow-query { trusted_query; } (mgmt + ops only) │
│ - allow-transfer { transfer_peers; } (TSIG-gated) │
│ - notify yes; also-notify { every anycast node unicast; } │
│ - dnssec-policy "stig-root"; inline-signing yes; │
│ - ECDSAP256SHA256 KSK + ZSK │
│ - Zones: │
│ "." (the ROOT zone — TLD delegations) │
│ "in-addr.arpa" (reverse-apex, /8 delegations) │
└─────────────────────────────────────────────────────────────┘
│
├── rsyslog (facility local5)
│ TLS 6514 (production / CUI)
│ UDP 514 (lab only)
▼
SIEM
Why two addresses on each anycast instance? The Linux kernel sources UDP replies from the IP the query arrived on. A query to the anycast IP is answered from the anycast IP — exactly what resolvers expect — without any source-IP manipulation in BIND. Outbound traffic the node initiates (AXFR pull, NOTIFY ACK, operator queries) is sourced from the unicast IP, so the BGP fabric, anti-spoof filters, and operational logs all see the unique node identity. Sending packets sourced from the anycast IP would create routing-asymmetry pathologies and is explicitly forbidden.
repository/
├── config/
│ ├── named-hidden-primary.conf # /etc/named.conf — hidden primary
│ ├── named-anycast.conf # /etc/named.conf — anycast instance
│ ├── acl.conf # Shared ACL definitions
│ ├── tsig.key.example # TSIG key template (regenerate before use)
│ ├── zones-primary.conf # Zone declarations — primary side
│ ├── zones-anycast.conf # Zone declarations — anycast side
│ ├── named-hardening.conf # systemd hardening drop-in
│ ├── rsyslog-named.conf # rsyslog SIEM forwarding rule
│ ├── named.root # root-hints file FOR DOWNSTREAM RESOLVERS
│ ├── routing/
│ │ ├── frr.conf # FRR/BGP anycast advertisement (each anycast node)
│ │ ├── frr-router.conf # FRR/BGP upstream test router (optional)
│ │ └── anycast-health.service # systemd unit for the health-check loop
│ └── zones/
│ ├── db.root # The root zone (TLD delegations)
│ └── db.in-addr.arpa # Reverse-apex zone (/8 delegations)
├── scripts/
│ ├── new-tsig-key.sh # TSIG key generator (Bash)
│ ├── new-tsig-key.py # TSIG key generator (Python 3.6+)
│ └── anycast-health.sh # Health-probe loop (advertises/withdraws)
├── README.md # This guide (internet-connected deployment)
└── README-airgapped.md # Air-gapped deployment variant
| File | Install path |
|---|---|
config/named-hidden-primary.conf (primary only) |
/etc/named.conf |
config/named-anycast.conf (anycast instances only) |
/etc/named.conf |
config/acl.conf |
/etc/named/acl.conf |
config/tsig.key.example → regenerated to tsig.key |
/etc/named/keys/tsig.key |
config/zones-primary.conf (primary only) |
/etc/named/zones-primary.conf |
config/zones-anycast.conf (anycast only) |
/etc/named/zones-anycast.conf |
config/zones/db.root (primary only) |
/var/named/db.root |
config/zones/db.in-addr.arpa (primary only) |
/var/named/db.in-addr.arpa |
config/named-hardening.conf |
/etc/systemd/system/named.service.d/hardening.conf |
config/rsyslog-named.conf |
/etc/rsyslog.d/named.conf |
config/named.root |
hand to every downstream resolver; not installed on the root servers |
config/routing/frr.conf (anycast only) |
/etc/frr/frr.conf |
config/routing/frr-router.conf (test router only) |
/etc/frr/frr.conf |
config/routing/anycast-health.service (anycast only) |
/etc/systemd/system/anycast-health.service |
scripts/anycast-health.sh (anycast only) |
/usr/local/sbin/anycast-health.sh |
scripts/new-tsig-key.* |
helper scripts — not installed on a server |
Oracle Linux 8 is assumed to be DISA STIG hardened at install time using the STIG profile available in the OL8 installer. No additional OS hardening steps are required here.
rsysloginstalled and running on every host- Outbound access to
yum.oracle.comduring installation - One unicast IP per host — the hidden primary needs one; each anycast instance needs one plus the shared anycast IP installed as a loopback alias (added/removed by the health script — see Step 7)
- One node-specific ASN per anycast instance — RFC 6382. The 32-bit private range 4200000000–4294967294 (RFC 6996) is suitable inside an enclave or private routing domain
- The upstream BGP-speaking router(s) configured to peer with every anycast instance and accept the anycast
/32
OL8 ships two parallel-installable BIND packages. The default bind package is BIND 9.11, which is outdated. Install bind9.16 instead — same service name (named.service), same config path (/etc/named.conf), but the modern 9.16 codebase (including dnssec-policy).
# Make sure the legacy 9.11 package is not also installed (the two conflict).
dnf remove -y bind bind-utils 2>/dev/null || true
# Install BIND 9.16 and matching utilities
dnf install -y bind9.16 bind9.16-utils
# Verify the version
named -v
# Expected: BIND 9.16.x (...)The package is literally named
bind9.16— there is no module stream to enable. If you previously randnf module enable bind:9.16you will seemissing groups or modules: bind:9.16; ignore it and use the plaindnf install bind9.16command above.
FRR ships in the OL8 AppStream — no EPEL or third-party repository required.
dnf install -y frrid named
passwd -l named
passwd -S named # must show: named L ...
# FRR (anycast instances only)
id frr # must exist after `dnf install frr`Steps 1–6 are common to every host (primary + every anycast instance). Steps 7–9 apply only to anycast instances. Step 10 onwards starts the services and verifies the topology.
On every host:
install -d -o root -g named -m 0750 /etc/named
install -d -o root -g named -m 0750 /etc/named/keys
# /var/log/named must be owned by 'named' — the daemon creates xfer.log,
# dnssec.log, security.log, etc. inside it. Owning it root:named with mode
# 0750 makes 'named' use group permissions (r-x) and the daemon fails with
# "isc_stdio_open '/var/log/named/dnssec.log' failed: permission denied".
install -d -o named -g named -m 0750 /var/log/named
install -d -o named -g named -m 0750 /var/named/data
install -d -o named -g named -m 0750 /var/named/dynamic
install -d -o named -g named -m 0700 /var/named/keys # hidden primary only — DNSSEC KSK/ZSK
install -d -o named -g named -m 0750 /var/named/slaves # anycast instances onlyDon't confuse
/etc/named/keys/with/var/named/keys/— they hold different material:
/etc/named/keys/tsig.key(mode 0640 root:named) — the TSIG zone-transfer key, shared identically with every peer./var/named/keys/K.+013+*.{key,private}(mode 0700 named:named on the dir, 0600 on the private files) — DNSSEC KSK/ZSK material generated and rotated by named underdnssec-policy(the hidden primary'snamed-hidden-primary.confpointskey-directoryhere). The hidden primary needs this directory because it signs; the anycast instances do not.
SELinux file contexts — the bind9.16 package's policy covers /var/named/* but does not pre-declare /var/log/named. Add the type and relabel before starting the service:
semanage fcontext -a -t named_log_t "/var/log/named(/.*)?"
restorecon -Rv /var/log/named /var/namedThe key must be identical on the primary and every anycast instance. Generate it once, then transfer the file securely to all servers.
# Bash
./scripts/new-tsig-key.sh -n zone-transfer-key -a hmac-sha384 -o tsig.key
# Python (any host with Python 3.6+)
python3 scripts/new-tsig-key.py -n zone-transfer-key -a hmac-sha384 -o tsig.keyInstall identically on every primary and anycast instance:
install -o root -g named -m 0640 tsig.key /etc/named/keys/tsig.keyThe supplied
config/tsig.key.exampleis a template only — itssecretvalue is a placeholder. Never deploy the template; regenerate.
rndc-confgen -a -k rndc-key -c /etc/rndc.key -A hmac-sha256 -b 256
chown root:named /etc/rndc.key
chmod 0640 /etc/rndc.keyinstall -o root -g named -m 0640 config/acl.conf /etc/named/acl.confEdit /etc/named/acl.conf and replace:
| Placeholder | What to set |
|---|---|
<<MGMT_SUBNET>>/24 |
The operator / monitoring management subnet |
192.0.2.21/32–192.0.2.23/32 (in transfer_peers, trusted_query) |
Each anycast instance's unicast IP — primary host |
192.0.2.10/32 (in primary_peers) |
The primary's unicast IP — anycast instances |
On the hidden primary:
install -o root -g named -m 0640 config/named-hidden-primary.conf /etc/named.conf
install -o root -g named -m 0640 config/zones-primary.conf /etc/named/zones-primary.conf
# /var/named is shipped by the bind9.16 package as mode 1770 root:named
# (sticky bit + group rwx), so 'named' group members have full write access
# there — inline-signing can create the .jnl, .jbk, and .signed files next
# to the master file. No need for a writable subdirectory.
install -o named -g named -m 0640 config/zones/db.root /var/named/db.root
install -o named -g named -m 0640 config/zones/db.in-addr.arpa /var/named/db.in-addr.arpa
# Restore SELinux labels — files copied from a non-/var/named source path
# (e.g., your home directory) carry the source context (default_t or
# user_home_t) instead of named_zone_t and named will fail to load the
# zones despite correct Unix ownership.
restorecon -Rv /var/namedOn each anycast instance:
install -o root -g named -m 0640 config/named-anycast.conf /etc/named.conf
install -o root -g named -m 0640 config/zones-anycast.conf /etc/named/zones-anycast.confEdit /etc/named.conf on each host and replace:
| Setting | What to set |
|---|---|
listen-on |
This host's unicast service IP and (on anycast instances) the shared anycast IP |
query-source address |
This host's unicast outbound address (NEVER anycast) |
notify-source |
Same as query-source — unicast |
transfer-source |
Same as query-source — unicast |
also-notify { ... } (primary only) |
The unicast IPs of every anycast instance |
server <PRIMARY_IP> (anycast only) |
The primary's unicast IP |
primaries { <PRIMARY_IP> key ... } (anycast only, in zones-anycast.conf) |
The primary's unicast IP |
Verify configuration syntax on every host before starting the service:
named-checkconf -z /etc/named.conf-z walks every zone — it catches not just syntax errors but also bad SOA timers, missing glue, and signing-policy mismatches.
For an initial bring-up it is usually cleaner to validate the basic topology (TSIG-protected AXFR, NOTIFY, SOA propagation, NS-RRset parity across every anycast instance) before layering DNSSEC on top. Two equivalent ways to do that per zone:
Edit in zones-primary.conf |
|
|---|---|
| Option A — comment out | # dnssec-policy "stig-root";# inline-signing yes; |
| Option B — explicit unsigned policy | dnssec-policy "insecure";(omit inline-signing) |
Option B is the BIND-9.16 idiomatic way to mark a zone deliberately unsigned — named will not generate keys and rndc dnssec -status reports the zone as insecure rather than missing.
The anycast instances need no configuration change — they pull whatever the primary serves (unsigned today, signed tomorrow). After enabling signing, continue from Step 11 (DNSSEC bootstrap and trust-anchor distribution).
The final production state for the root MUST be signed.
On every host:
install -d -o root -g root -m 0755 \
/etc/systemd/system/named.service.d
install -o root -g root -m 0644 \
config/named-hardening.conf \
/etc/systemd/system/named.service.d/hardening.conf
systemctl daemon-reloadanycast-health.sh is the sole owner of the anycast service address on lo: it adds the address when the local named is healthy and removes it when not. For that arrangement to survive a fresh boot, named must be able to listen-on { 198.51.100.1; } even when the IP has not yet been added — otherwise startup races the health-check by a few seconds and bind() returns address not available.
The kernel switch that allows this is net.ipv4.ip_nonlocal_bind=1 — the same sysctl used by HAProxy, keepalived, and every other anycast/VIP setup. With it set, named's socket binds successfully regardless of whether the IP is currently on an interface; once anycast-health.sh adds the IP to lo, packets start flowing into the already-listening socket.
sysctl -w net.ipv4.ip_nonlocal_bind=1
echo 'net.ipv4.ip_nonlocal_bind = 1' > /etc/sysctl.d/99-anycast.confDon't bother with NetworkManager / network-scripts persistence for the anycast IP. OL8's NetworkManager (1.40) does not manage the
lodevice —nmcli connection modify lo …returnsunknown connection 'lo'— and double-managing an IP thatanycast-health.shalready owns just creates a race. The health script is the only thing that should ever add or remove198.51.100.1.
For the very first start only, you may optionally add the IP manually so traffic flows immediately (rather than waiting for anycast-health.sh to come up and probe successfully):
ip -4 addr add 198.51.100.1/32 dev loThe health script will take over from here. On subsequent reboots, no manual step is needed — named starts (binds thanks to ip_nonlocal_bind), then anycast-health.service starts (Requires=named.service), probes, and adds the IP after ANNOUNCE_AFTER good probes.
# Enable bgpd and zebra in /etc/frr/daemons
sed -i 's/^bgpd=no/bgpd=yes/' /etc/frr/daemons
sed -i 's/^zebra=no/zebra=yes/' /etc/frr/daemons
install -o frr -g frr -m 0640 config/routing/frr.conf /etc/frr/frr.confEdit /etc/frr/frr.conf on each host and replace:
| Setting | What to set |
|---|---|
router bgp 4200000001 |
Unique ASN for this node — RFC 6382 requirement |
bgp router-id 192.0.2.21 |
This node's unicast IP |
neighbor 192.0.2.254 |
Upstream BGP peer's IP |
remote-as 65000 |
Upstream's ASN |
198.51.100.1/32 (in prefix-list / route-map) |
The shared anycast service address |
neighbor … password CHANGE_ME_BGP_AUTH |
A strong per-pair MD5 password |
Start FRR:
systemctl enable --now frr
vtysh -c 'show bgp summary'install -o root -g root -m 0750 \
scripts/anycast-health.sh /usr/local/sbin/anycast-health.sh
install -o root -g root -m 0644 \
config/routing/anycast-health.service \
/etc/systemd/system/anycast-health.service
systemctl daemon-reloadThe service is enabled after named is up — see Step 11.
On every host:
install -o root -g root -m 0644 \
config/rsyslog-named.conf /etc/rsyslog.d/named.confEdit /etc/rsyslog.d/named.conf and replace siem.example.mil with your SIEM hostname or IP.
systemctl restart rsyslog
logger -p local5.info "named rsyslog test $(date)" # verify deliverysystemctl enable --now named
systemctl status named
journalctl -u named --no-pager | tail -20Within ~30 s of the primary starting, every anycast instance should AXFR the root zone. Confirm with:
# On the primary
journalctl -u named --no-pager | grep -E 'AXFR|sending notifies'
# On each anycast instance
journalctl -u named --no-pager | grep -E 'transferred serial|Transfer status'
ls -la /var/named/slaves/ # zone files must appear here, owner named:namedOn each anycast instance, start the health-check supervisor:
systemctl enable --now anycast-health.service
journalctl -u anycast-health --no-pager | tail
# Expect: "started" and after ~2 probes: "ANNOUNCING anycast address 198.51.100.1/32 on lo"Verify BGP advertises the anycast prefix:
vtysh -c 'show bgp ipv4 unicast' # 198.51.100.1/32 should appear
vtysh -c 'show ip route 198.51.100.1/32'firewall-cmd --get-active-zones
# Wipe defaults and start clean
for svc in $(firewall-cmd --zone=public --list-services); do
firewall-cmd --permanent --zone=public --remove-service="$svc"
done
firewall-cmd --permanent --zone=public --add-service=ssh
firewall-cmd --permanent --zone=public --add-service=dns
firewall-cmd --reload
firewall-cmd --zone=public --list-allOn the hidden primary, tighten DNS ingress to operators + anycast instance unicast IPs only:
firewall-cmd --permanent --zone=public --remove-service=dns
firewall-cmd --permanent --zone=public \
--add-rich-rule='rule family="ipv4" source address="<<MGMT_SUBNET>>/24" port port="53" protocol="udp" accept'
firewall-cmd --permanent --zone=public \
--add-rich-rule='rule family="ipv4" source address="<<MGMT_SUBNET>>/24" port port="53" protocol="tcp" accept'
for ip in 192.0.2.21 192.0.2.22 192.0.2.23; do
firewall-cmd --permanent --zone=public \
--add-rich-rule="rule family=\"ipv4\" source address=\"${ip}/32\" port port=\"53\" protocol=\"tcp\" accept"
done
firewall-cmd --reloadOn anycast instances, open BGP (TCP/179) to the upstream peer:
firewall-cmd --permanent --zone=public \
--add-rich-rule='rule family="ipv4" source address="192.0.2.254/32" port port="179" protocol="tcp" accept'
firewall-cmd --reloadWith dnssec-policy "stig-root" active and inline-signing yes, named generates KSK + ZSK material under /var/named/keys/ on first load of the root zone. Wait until the keys are visible (signature publication, ~1 minute):
ls -la /var/named/keys/
rndc dnssec -status .Two distribution flows are now in scope:
-
Trust-anchor distribution (downstream) — extract the root KSK DS RR and publish it as a static trust anchor on every recursive resolver in the deployment. (Inside an enclave or private root namespace, this is the trust anchor; there is no IANA chain.)
dnssec-dsfromkey -2 /var/named/keys/K.+013+*.keyThe printed
DSline is what every resolver needs — for BIND resolvers as atrust-anchors { }clause; for PowerDNS Recursor as arecursor.luaaddTA()line. The companion repositories ship example resolver configs. -
Child DS handling (upstream from TLD operators) — each TLD operator who is signing their zone hands over their DS record. Add it to
/var/named/db.rootunder the TLD's stanza. Each child DS MUST have a matching DNSKEY in the child zone at the time you publish — otherwise validation fails the moment your update propagates.Edit
/var/named/db.root, bump the SOA serial, run:named-checkzone . /var/named/db.root rndc reload .
Within seconds the anycast instances pull the new zone via IXFR/AXFR.
If you need RSASHA256 instead of ECDSAP256SHA256 (for example, because a downstream resolver does not yet support algorithm 13), switch the zone stanza to
dnssec-policy "stig-root-rsa";before first start and re-bootstrap.
Run on the hidden primary:
# 1. Service is healthy
systemctl status named
journalctl -u named --no-pager | tail -20
# 2. Recursion really is disabled — must return REFUSED
dig @192.0.2.10 +norec www.iana.org. A | grep status
# Expected: status: REFUSED
# 3. Authoritative answer for the root — must return AA flag, NOERROR
dig @192.0.2.10 . SOA | grep -E 'flags|status'
# Expected: status: NOERROR ; flags: qr aa ...
# 4. NS RRset matches what is published as glue
dig @192.0.2.10 . NS +short
# 5. DNSSEC chain — the root is signed; expect RRSIG and DNSKEY responses
dig @192.0.2.10 . DNSKEY +dnssec | grep -E 'DNSKEY|RRSIG'
# 6. Referral payload size for a TLD — must be < 512 octets
dig @192.0.2.10 example. NS +noall +answer +authority +additional +nostats | wc -c
# Expected: comfortably under 512
# 7. Reverse-apex delegations resolve
dig @192.0.2.10 in-addr.arpa. NS +short
dig @192.0.2.10 192.in-addr.arpa. NS +shortRun on each anycast instance — via both the unicast and the anycast IP:
# 8. Zone has been transferred from primary
named-checkzone . /var/named/slaves/db.root
# 9. Anycast instance answers authoritatively for the same SOA serial as primary
dig @192.0.2.21 . SOA +short # via unicast
dig @198.51.100.1 . SOA +short # via anycast
# Expected: identical serial number, identical to primary
# 10. NS RRset returned by every authority is identical
for ns in 192.0.2.10 192.0.2.21 192.0.2.22 192.0.2.23 198.51.100.1; do
echo "=== $ns ==="
dig @$ns . NS +short | sort
done
# Expected: identical, sorted output everywhere
# 11. Anycast instances refuse recursion
dig @198.51.100.1 +norec www.iana.org. A | grep status
# Expected: status: REFUSED
# 12. Zone-transfer ACL works — unauthenticated AXFR is refused
dig @192.0.2.10 . AXFR | head -5
# Expected: "Transfer failed." (no TSIG presented)Run on the routing fabric / a neighbouring router:
# 13. Anycast prefix is advertised from every node
show bgp ipv4 unicast 198.51.100.1/32 detail
# Expected: as many paths as healthy anycast nodes, each with a different
# neighbour ASN (per-node ASN per RFC 6382)
# 14. Withdraw on failure works — stop named on one node and confirm withdraw
# within (PROBE_INTERVAL * WITHDRAW_AFTER) seconds + BGP keepalive
systemctl stop named # on root-anycast-1
# Watch the path count drop by 1 on the route reflector / RR client.Run on every host:
# 15. Logging — every category has live entries
ls -l /var/log/named/
tail /var/log/named/xfer.log
tail /var/log/named/dnssec.log
tail /var/log/named/security.log
# 16. SIEM receipt — confirm syslog forwarding
journalctl -u rsyslog --no-pager | tail -5To exercise BGP peering, ECMP, and health-driven withdraw before going to production, run a fourth OL8 host as the upstream test router. It peers with every anycast instance, installs all healthy paths into its FIB via ECMP, and forwards client traffic destined for the anycast service IP to whichever node has the best path.
+-----------------------+
| TEST ROUTER | ASN 65000
| 192.0.2.254 | ip_forward=1
+-----------+-----------+
|
+----------------+----------------+
| | |
eBGP eBGP eBGP
| | |
+----+-----+ +----+-----+ +----+-----+
| anycast-1| | anycast-2| | anycast-3|
| ASN 4..01| | ASN 4..02| | ASN 4..03|
| unicast | | unicast | | unicast |
| 192.0.2 | | 192.0.2 | | 192.0.2 |
| .21 | | .22 | | .23 |
+----+-----+ +----+-----+ +----+-----+
| | |
+----------------+----------------+
|
| advertise 198.51.100.1/32
v
+------v------+
| ANYCAST | 198.51.100.1
+-------------+
^
| dig @198.51.100.1 . SOA
+------+------+
| TEST CLIENT | (any host with route to 198.51.100.1
| / RESOLVER | via the test router)
+-------------+
dnf install -y frr
sed -i 's/^bgpd=no/bgpd=yes/' /etc/frr/daemons
sed -i 's/^zebra=no/zebra=yes/' /etc/frr/daemons
# Router must forward packets — default OL8 has this OFF.
sysctl -w net.ipv4.ip_forward=1
echo 'net.ipv4.ip_forward = 1' > /etc/sysctl.d/99-router.confinstall -o frr -g frr -m 0640 config/routing/frr-router.conf /etc/frr/frr.confEdit /etc/frr/frr.conf and replace:
| Setting | What to set |
|---|---|
router bgp 65000 |
This router's ASN |
bgp router-id 192.0.2.254 |
This router's IP on the link to the anycast nodes |
neighbor 192.0.2.21 / .22 / .23 |
The anycast nodes' unicast IPs |
remote-as 4200000001 / 02 / 03 |
Each node's per-node ASN (must match frr.conf on the node) |
password CHANGE_ME_BGP_AUTH |
Per-pair BGP MD5 password (must match the node) |
maximum-paths 8 |
At least the number of anycast nodes you operate |
198.51.100.1/32 (in ANYCAST-IN) |
The shared anycast service address |
systemctl enable --now frrfor ip in 192.0.2.21 192.0.2.22 192.0.2.23; do
firewall-cmd --permanent --zone=public \
--add-rich-rule="rule family=\"ipv4\" source address=\"${ip}/32\" port port=\"179\" protocol=\"tcp\" accept"
done
firewall-cmd --reloadvtysh -c 'show bgp summary'
# Expected: state Established for every neighbour, Up/Down > 0s
vtysh -c 'show bgp ipv4 unicast 198.51.100.1/32'
# Expected: one path per healthy anycast node, each tagged with a different
# origin-AS (per-node ASN per RFC 6382). The 'multipath' marker appears
# beside the active paths.
ip route show 198.51.100.1
# Expected (with 3 healthy nodes):
# 198.51.100.1
# nexthop via 192.0.2.21 dev <iface> weight 1
# nexthop via 192.0.2.22 dev <iface> weight 1
# nexthop via 192.0.2.23 dev <iface> weight 1From a host that uses this router as its gateway (or dig directly on the router, since it now has a route to 198.51.100.1):
# Baseline — answered by some anycast node
dig @198.51.100.1 . SOA +short
# Drain anycast-1 by stopping named (the health-script will withdraw the IP
# from lo; FRR sees the connected-route change and withdraws the BGP path):
ssh root-anycast-1 systemctl stop named
# Within (PROBE_INTERVAL × WITHDRAW_AFTER) + BGP keepalive ≈ 25 s the path
# count drops on the router:
vtysh -c 'show bgp ipv4 unicast 198.51.100.1/32'
# Expected: one fewer path
# Service remains available — answered by a surviving node:
dig @198.51.100.1 . SOA +short
# Bring it back:
ssh root-anycast-1 systemctl start named
# Within (PROBE_INTERVAL × ANNOUNCE_AFTER) + BGP keepalive ≈ 20 s, path
# re-appears in the BGP table and rejoins the ECMP group.dnf install -y bind9.16
# Drop a resolver config (or use pdns-recursor from the companion repo)
# that includes 'hint-file "/var/named/named.root";' pointing at
# 198.51.100.1 — the file in this repository (config/named.root) is exactly
# that. The whole "downstream resolver → anycast root → hidden primary"
# chain can now be exercised from a single host.Production note: the router config shipped here is a lab demonstration. Production routers will already exist as part of the operator's network, and the anycast
/32path-acceptance, ECMP, BGP MD5, and per-node ASN handling shown here must be implemented on those routers regardless of vendor (Cisco/Juniper/Arista equivalents follow the same shape). The principles — accept only198.51.100.1/32from the named anycast peers, enable multipath, require per-pair authentication, install all paths via ECMP, and trust the per-node ASN-distinguishability for telemetry — translate directly.
# 1. Edit the root zone — bump the SOA serial to YYYYMMDDNN
vi /var/named/db.root
# 2. Validate syntax and SOA/glue consistency
named-checkzone . /var/named/db.root
# 3. Reload the zone — re-signs (DNSSEC) and triggers NOTIFY automatically
rndc reload .
journalctl -u named --no-pager | tail -20Within seconds every anycast instance should AXFR/IXFR; verify with the smoke tests above.
- Confirm the child operator has signed the child zone and has a DNSKEY published.
- Receive their
DSrecord(s) via your secure-channel-of-record (signed e-mail, S/MIME, ticket workflow). - Edit
/var/named/db.root:newtld. IN NS ns1.newtld. newtld. IN NS ns2.newtld. ns1.newtld. IN A <child-NS-1-A> ns2.newtld. IN A <child-NS-2-A> newtld. IN DS <keytag> 13 2 <digest> - Bump SOA serial,
named-checkzone,rndc reload .. - Verify validation end-to-end from a resolver:
dig @<resolver> www.newtld. +dnssec +adflag # Expected: 'ad' flag set, AD answer
Same as above but omit the DS record. The parent NS RRset in the root remains signed; the child zone is served unvalidated. Resolver answers do not carry the ad flag.
dnssec-policy runs ZSK rollover unattended. ZSK lifetime ~1 year, ~1 month overlap during rollover (both old and new ZSK present in the zone).
KSK rollover (typical lifetime ~5 years) requires the offline KSK private key to be temporarily restored on the primary:
# Stage the KSK private from offline media:
install -o named -g named -m 0600 \
/media/offline/K.+013+<keytag>.private \
/var/named/keys/K.+013+<keytag>.private
# Force the rollover
rndc dnssec -rollover -key <keytag> .
# Watch progression
watch -n10 'rndc dnssec -status .'
# After the new DS record has been published to every downstream resolver
# AND the old KSK is fully retired, sanitise the on-host private file
shred -u /var/named/keys/K.+013+<oldkeytag>.privateRe-archive the new KSK private to offline media; do not leave the private file on-host outside an active rollover window. Distribute the new trust anchor to every downstream resolver before the old anchor retires from the zone.
- Bring up the OL8 host with the unicast management IP only.
- Walk through Steps 1–11 of the deployment guide on the new host.
- Add the new host's unicast IP to
transfer_peers,trusted_query, and thealso-notifylist on the primary, thenrndc reload. - Verify zone transfer and BGP advertisement of the anycast prefix from the new node.
- Allocate a fresh per-node ASN — do NOT reuse a retired one.
# On the node to be drained:
systemctl stop anycast-health.service
# anycast-health.sh removes 198.51.100.1 from lo on exit (trap).
# FRR withdraws the prefix within one keepalive interval.
# Verify the prefix is withdrawn:
vtysh -c 'show bgp ipv4 unicast 198.51.100.1/32'
# Now patch / reboot / whatever. To re-introduce:
systemctl start anycast-health.service
# The script re-adds the IP after ANNOUNCE_AFTER (default 2) good probes.| Task | Frequency |
|---|---|
Review acl.conf against current network topology |
On every network change |
| Verify SOA serial parity across primary + every anycast instance | Daily (or via monitoring) |
Inspect xfer.log and dnssec.log for unexpected failures |
Daily (or via SIEM alerting) |
Confirm rndc dnssec -status . shows healthy state on the primary |
Weekly |
Confirm the anycast /32 is advertised from every healthy node |
Hourly (route monitor) |
| Audit firewalld rules — primary ingress narrow to mgmt + anycast unicast | Quarterly |
| Audit FRR neighbour passwords and per-node ASN allocations | Quarterly |
| Rotate TSIG keys — generate, distribute, swap, retire | Annually |
Check OL8 AppStream for updated bind9.16 and FRR packages |
Monthly |
| Re-distribute root trust anchor to every downstream resolver | Per KSK rollover (~5 y) |
| Run smoke tests after any configuration change | After every change |
| Verify offline KSK media integrity (sha256 + restore drill) | Annually |
These issues are documented here to prevent recurrence.
query-source address 198.51.100.1; looks symmetric and feels right. It is wrong. Replies from an upstream root (and AXFR responses from the hidden primary) would be routed back to some anycast node — usually not the one that asked — and dropped. BGP anti-spoof / uRPF filters on the upstream peer will also drop packets sourced from a /32 that node does not own per the routing table. Always pin query-source, transfer-source, and notify-source to the unicast IP. The supplied configs do.
A node that has the anycast IP statically on lo but cannot serve the zone (named crashed, AXFR never completed, zone expired) becomes a black hole — clients route to it, get nothing, never time out enough to retry elsewhere within a session. The supplied anycast-health.sh + FRR redistribute connected combination guarantees the prefix is advertised only while a local probe (dig @127.0.0.1 . SOA) succeeds. Default-fail-safe at startup ensures the script removes the IP before its first probe runs — a stuck script cannot leave a stale advertisement.
The health-check service exists precisely to observe a failing named and withdraw the anycast prefix when probes fail. Coupling it to named with Requires=named.service defeats that: systemd will stop the watcher every time named stops, the script's exit trap removes the IP from lo (good), and then nothing is left running to detect when named comes back — so when you systemctl start named afterwards, the IP stays withdrawn until you manually restart anycast-health too. Worse, on a real named crash with Restart= not configured, the watcher is killed at the worst possible moment.
The supplied unit uses Wants=named.service (pull-in only, no lifecycle coupling) plus After=named.service (boot ordering only). The watcher then runs continuously, regardless of named's state, and the dig probe is what controls whether the anycast IP is on lo. The smoke-test sequence systemctl stop named → wait → systemctl start named should leave anycast-health Active: active (running) throughout; if it shows Active: inactive (dead) after stopping named, the unit has Requires=named.service and needs to be patched.
It's tempting to mark the anycast /32 with the no-export BGP community on the anycast nodes "so it doesn't leak beyond the directly-connected router". This is wrong: no-export blocks the route at every eBGP boundary — including the very peering between the anycast node and the upstream router. BGP on the originating node will silently keep the route in its local RIB and report not advertised to EBGP peer, the session stays Established, and PfxRcd stays at 0 on the router. The supplied frr.conf therefore does not set no-export; if you actually need to scope re-advertisement, do it on the router's outbound filter, not on the anycast originator. If you ever see Community: no-export on the anycast node's BGP RIB next to your service /32, that's the bug.
Reusing the same ASN on every anycast node makes BGP best-path selection unable to distinguish individual nodes. Operational telemetry (which node is the closest one to which client?) and incident response (which node is misbehaving?) both depend on the upstream BGP table showing different origin-ASNs per node. The supplied frr.conf makes this explicit; replace the placeholder ASN per node and never share.
dnssec-enable yes; is a 9.10/9.11 directive. In 9.16 it is removed; including it produces unknown option 'dnssec-enable'. DNSSEC is unconditionally enabled in 9.16; control is via dnssec-policy and inline-signing.
Leaving the implicit default (auto) makes named try to load the IANA root key and act as a partial validator — pointless on an authoritative-only root server (the validators are the resolvers, not the roots), and a permanent source of log noise on private root deployments. Set dnssec-validation no; explicitly (the supplied configs do).
An IP-only allow-transfer clause allows an unauthenticated peer at that IP to AXFR the zone. The TSIG requirement comes from the key clause inside the ACL, as in:
acl "transfer_peers" {
!{ !key "zone-transfer-key"; any; };
192.0.2.21/32;
};
The double-negation !{ !key ... ; any; }; reads: refuse anything that does not present the TSIG key. Without it, the IP filter is the only check — a spoofed source defeats it.
Re-running new-tsig-key.sh on each host produces a different secret each time and AXFR fails with tsig verify failure. Generate the key once, then copy the file to every peer.
If the firewall path between the primary and an anycast instance requires a specific source IP for NOTIFY (most do), notify-source must be set explicitly. The supplied primary config pins it to the primary's unicast IP; leaving it as the BIND default lets the kernel pick the routing-table source, which on multi-homed hosts is often the wrong interface.
Hidden primary still must accept queries from the anycast instances during diagnostics
allow-query { trusted_query; }; blocks normal queries from everything outside the management ACL. The transfer_peers clause is checked only on the AXFR/IXFR path — so an anycast instance that runs dig @primary . SOA gets REFUSED unless its unicast IP is also in trusted_query. The supplied ACL includes them there for operational reasons.
The root is queried by every resolver in the deployment — a delegation referral that does not fit in 512 octets forces non-EDNS clients to retry over TCP, breaking some legacy resolvers and tripling the load. Keep the NS RRset small (two records is correct for a private root; even the public root uses 13 globally), and ensure glue A records match the authoritative A records. Verify with:
dig +noall +answer +authority +additional example. NS | wc -cFiles copied into /var/named/ from a non-/var/named source path (your home directory, a checked-out repo, an approved-media drop) inherit the source SELinux context — typically default_t or user_home_t — instead of named_zone_t. named_t cannot read those, and named fails to load the zone with permission denied on the master file. Run restorecon -Rv /var/named after every install into that tree. The OL8 bind9.16 SELinux module already permits named_t to write to named_zone_t for inline-signing artifacts (.jnl, .jbk, .signed, .signed.jnl).
When inline-signing yes is active, named maintains the signed copy in db.root.signed and a journal in db.root.jnl. Editing db.root directly is fine as long as you bump the SOA serial and run rndc reload . — never edit the .signed file, and never delete the .jnl while named is running.
If the host has multiple addresses, BIND will (by default) source outbound queries from whichever interface the kernel routes the destination to. Firewalls or peering policies that expect a specific source IP will then drop the response. Always set query-source address <unicast IP>; and transfer-source <unicast IP>; explicitly. On anycast nodes especially: the kernel may otherwise select the anycast IP as source for some destination — see "never source outbound from the anycast IP" above.
The stock named.service has After=network.target. That only signals the network stack is up; it does not wait for interfaces to have IPs assigned. At boot, NetworkManager configures interfaces asynchronously, so named can race ahead and try to bind to the configured listen-on / query-source address (including the anycast IP) before the IP exists on the interface. Symptom in the journal:
named[…]: could not get query source dispatcher (10.100.100.3#0)
named[…]: loading configuration: address not available
named[…]: exiting (due to fatal error)
The supplied drop-in adds Wants=network-online.target and After=network-online.target. Make sure that service is enabled:
systemctl enable --now NetworkManager-wait-online.service
systemctl is-enabled NetworkManager-wait-online.service # → enabledThe EL stock named.service does not set User=named — named starts as root, binds port 53 on all listen-on addresses, then calls setuid(named) itself via the -u named flag. CapabilityBoundingSet= is the upper bound on capabilities for the entire service lifetime, including the root startup phase. Restricting it to just CAP_NET_BIND_SERVICE strips:
CAP_DAC_READ_SEARCH— root can no longer bypass DAC to read files it doesn't own. The zone files (named:namedmode0640) become unreadable by root, andExecStartPre=/usr/sbin/named-checkconf -zfails withloading from master file … failed: permission denied. No SELinux AVC is logged — the denial happens at the DAC layer before the LSM is consulted.CAP_SETUID/CAP_SETGID— root can no longer drop to usernamed, sonamed -u namedwould fail too.
The supplied drop-in therefore keeps four caps in the bounding set: CAP_NET_BIND_SERVICE, CAP_SETUID, CAP_SETGID, CAP_DAC_READ_SEARCH. Only CAP_NET_BIND_SERVICE is in AmbientCapabilities=, so after the setuid(named) only that one survives.
Many hardening directives implicitly force NoNewPrivileges=true, which blocks the SELinux domain transition init_t → named_t (AVC: nnp_transition). Setting NoNewPrivileges=false afterwards has no effect — the kernel locks NNP on as soon as any one of these is set:
SystemCallFilter=,RestrictNamespaces=,LockPersonality=,RestrictRealtime=,ProtectKernelTunables=,ProtectKernelModules=,MemoryDenyWriteExecute=,PrivateDevices=
The supplied drop-in excludes all of them. SELinux mandatory access control covers the equivalent ground on OL8.
Other OL8 (systemd 239) constraints:
ProtectSystem=strictis unsafe on OL8. UseProtectSystem=full.- List-type keys accumulate. Empty assignment before redefining
CapabilityBoundingSet=/SystemCallFilter=. ProtectHostname=/ProtectClock=/RestrictSUIDSGID=/ProtectKernelLogs=are not available on OL8 (systemd 239) — silent parse failures.ReadWritePaths=requires every listed path to exist at service start, otherwiseFailed at step NAMESPACE: No such file or directoryis logged with no indication of which path was missing. The supplied drop-in lists only/var/namedand/var/log/named, both created in Step 1.
Every zone-file edit on the primary must bump the SOA serial. Without a higher serial, the anycast instances treat the IXFR/AXFR as unchanged and silently skip the transfer. The supplied templates use YYYYMMDDNN (RFC 1912 §2.2) — 2026031101, 2026031102, etc.
The root delegates only top-level domains. If you co-host (say) mission.example directly in db.root for "convenience", DNSSEC validation breaks at the moment a resolver expects the chain to walk . → example. → mission.example. — the resolver looks for a DS record for mission.example in example., doesn't find one because you skipped the intermediate zone, and the answer comes back bogus. Always run mission.example as its own zone on example.'s name servers and let the delegation chain stay intact.
MIT — free to use, modify, and distribute.
Przemysław Pradela
Built through real production deployment and iterative troubleshooting on Oracle Linux 8.
Contributions and issue reports welcome.