Now let’s move on to the second part of the ‘flow’ — incoming DNS queries.

This part of the flow is the receiving DNS nameserver side — the nameserver receiving the queries in order to find records and send them to the requestor.

We covered that when sending queries, generally for non /etc/hosts/ entries, the queries go to whatever nameserver is defined in /etc/resolv.conf.

That leaves us with 3 possible nameserver situations.

(1) External nameserver not controlled by us:

nameserver 8.8.8.8

It’s going to send the queries to google’s nameserver at IP 8.8.8.8. We have no control over that server and neither does the customer.

(2) External nameserver the customer has on the same network:

nameserver x.x.x.x

This could be a bind DNS server which we might be able to help with, or it could be some 3rd party firewall or router device, which they would need to reach out to the vendor for support for.

(3) Internal localhost DNS server

nameserver 127.0.0.1

This points to localhost 127.0.0.1 on the same machine, tcp/udp ports 53.

So, if a customer has this, we need to know what kind of DNS server is running on 127.0.0.1 port 53.

The easiest way to find that out is in a sosreport. Open the sosreport and look for /sos_commands/networking/netstat_-neopa. Grep that file for ‘:53’ and it should show you what process is running on port 53:

# netstat -neopa | grep :53
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 25 769890 152824/named off (0.00/0/0)
tcp6 0 0 ::1:53 :::* LISTEN 25 769893 152824/named off (0.00/0/0)

udp 0 0 127.0.0.1:53 0.0.0.0:* 25 769888 152824/named off (0.00/0/0)
udp6 0 0 ::1:53 :::* 25 769891 152824/named off (0.00/0/0)

Here we see the bind DNS server (service name is known as ‘named’).

** If you see this, you then need to check ‘ps aux | grep named` to determine if it’s chrooted or not, because bind can run itself inside a chrooted environment.

chrooted:

# ps aux | grep named
named 210290 0.0 0.2 332740 87988 ? Ssl 11:46 0:00 /usr/sbin/named -u named -c /etc/named.conf -t /var/named/chroot

not chrooted:

# ps aux | grep named
named 210556 2.0 0.2 263100 87904 ? Ssl 11:48 0:00 /usr/sbin/named -u named -c /etc/named.conf

It’s important to determine this because if it’s chrooted, the domain record files are located at /var/named/chroot/var/named, as opposed to the standard /var/named/ folder. More on that later.

It’s also possible there could be other different services besides bind running such as dnsmasq, unbound, systemd-resolved:

# netstat -neopa | grep :53
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 0 773290 152907/dnsmasq off (0.00/0/0)
udp 0 0 127.0.0.1:53 0.0.0.0:* 0 773289 152907/dnsmasq off (0.00/0/0)

# netstat -neopa | grep :53
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 0 790618 169502/unbound off (0.00/0/0)
udp 0 0 127.0.0.1:53 0.0.0.0:* 0 790617 169502/unbound off (0.00/0/0)

# netstat -neopa | grep :53
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 193 833766 209541/systemd-reso off (0.00/0/0)
udp 0 0 127.0.0.53:53 0.0.0.0:* 193 833765 209541/systemd-reso off (0.00/0/0)

The ONLY one that will not show up as a DNS server listening on 127.0.0.1 is NSCD, because it works directly through glibc instead of as a listening nameserver.

It is possible for nscd to run alongside the others, but not recommended (conflicting caches).

In general it is not recommended to use NSCD as it is outdated and has been deprecated upstream. (https://fedoraproject.org/wiki/Changes/DeprecateNSCD)

IT’S IMPORTANT TO NOTE:

As long as the customer’s DNS server is hosted internally on software we provide, we can request all DNS files/records/configurations and attempt to reproduce the issue locally within a VM or multiple VMs.

This is what makes DNS cases one of the better/easier types of cases to handle, because it’s mostly all contained within a reproducible DNS server environment. This includes master/slave configurations.