CCATP #533- Bart Busschots on Choosing a DNS Resolver

In this “Lite” version of Chit Chat Across the Pond (within a NosillaCastaway’s definition of Lite), Bart teaches us about DNS Resolvers and helps give us the information to choose the right one for us. To get us there he starts by explaining the background technologies. He explains DNS and how there’s two kinds of servers, he explains DNS Name resolution and why caching is important, and the security problems (and solutions) of DNS. Then he explains how third-party DNS providers can solve some of these problems. He explains their motivations which will inform your own decision. Finally (at about the 1 hour mark) he walks through the solutions offered by OpenDNS, Google, Quad9 and Cloudflare. I loved this episode and you can tell Bart really loves talking about DNS.

mp3 download

What is DNS?

The Domain Name System, or DNS is like the yellow pages of the internet. DNS maps records of different types to hierarchical human-friendly names which we call domain names.

The separator within domain names is the dot (or period) symbol. We read them from left to right, but the hierarchy goes the other way. So, .com is a top-level domain (TLD), podfeet.com is a sub-domain of .com, and www.podfeet.com is a sub domain of podfeet.com.

You’ll often hear DNS described as ‘the phone book of the internet which converts domain names to IP addresses’. While it is true that DNS does serve that purpose, that’s only a sub-set of what DNS does. For a start, DNS is not just for the internet, DNS also plays an important role on home and corporate networks. Even leaving that aside, DNS also does a lot more than just converting names to IP addresses. There are actually many different kinds of DNS record, and they all serve different ends. Here’s a list of some of the most commonly used DNS record types, and what they’re used for:

• A and AAAA records map domain names to IP addresses (A records for IPv4, and AAAA records for IPv6).
• CNAME records act as domain name aliases — the value of a CNAME record for any given domain is another domain name. For example, if you pay Bart’s Widgets to host your domain’s email you might create a CNAME record to map mail.yourdomain.com to mail.bwidgets.com.
• MX records are used to specify one or more email servers for a domain.
• PTR records, in-conjunction with the special domains in-addr.arpa and ip6.arpa, map domain names to IP addresses (in-addr.arpa for IPv4 & ip6.arpa for IPv6)
• NS records are used to delegate control of a sub-domain to another set of DNS servers. This is how the hierarchy is managed. The name servers for the .com domain use NS records to delegate control of the sub-domain podfeet.com to Allison’s DNS servers.
• SRV records enable service auto-discovery — these records are heavily used by Microsoft, enabling things like automatic license server discovery on corporate networks.
• TXT records allow arbitrary text strings to be associated with domain names. Because they are such generic records they’re used for all kinds of things, including domain ownership validation, SPF records for mail sender white-listing, DKIM records for distributing email-signing public keys, and DMARK records for publishing domain-specific email-handling policies for use by spam filters.

There are Two Kinds of DNS Server

DNS is very much a game of two halves — there are DNS servers which publish DNS records, and there are DNS servers which perform lookups on behalf of apps, computers, and devices. Both kinds are colloquially referred to as DNS Servers, but they are completely different entities, so without enough context to figure out which one people mean, the term is meaningless.

The servers that publish information to the network are properly called Authoritative DNS Servers, and the only people that need to worry about those are the owners of domain names. They are not what Windows, MacOS, iOS, or your home router mean when they label a text box on a network settings page ‘DNS Server’.

The DNS servers our router sand OSes care about, and, the ones we’ll be focused on today are the ones that look up DNS records on behalf of users. They are properly called DNS Resolvers.

DNS Name Resolution

Because domain names are hierarchical, turning a domain name into a DNS record is actually quite a complex task. DNS Resolvers have a lot of work to do!

To turn a domain name like www.podfeet.com into an IP address a DNS resolver has to work through the following process:

1. A client sends a request to the resolver for the A record(s) for the domain www.podfeet.com
2. Randomly pick the IP address for one of the so-called DNS Root Servers (authoritative DNS servers that know the IP addresses for the authoritative servers responsible for every top-level domain who’s IP addresses are shipped as part of just about every OS for just about every internet-connected device) and send it a request for the A record(s) for www.podfeet.com.
3. The chosen authoritative root server will not know the answer, but it will know the IP addresses for the authoritative DNS servers for the .com TLD, so it will reply with a so-called referral, i.e. with a list of NS records.
4. The DNS resolver then randomly picks the IP address from one of the received NS records, and sends a query for the A record(s) for www.podfeet.com to that authoritative DNS server.
5. Again, the .com authoritative server will not know the final answer, but it will now which authoritative DNS servers are responsible for the podfeet.com domain, so it will reply with another referral, this time contains the NS records for podfeet.com.
6. The DNS resolver then randomly picks the IP address from one of those NS records and sends the query for the A record(s) for www.podfeet.com to that authoritative DNS server.
7. The authoritative DNS server for podfeet.com does know the IP address(es) for www.podfeet.com, so it will finally reply to the resolver with the A record(s) containing the IP(s) for www.podfeet.com.
8. The DNS resolver replies to the original client with the answer.

That’s a lot of work! The task is also recursive, which is why DNS resolvers that perform these kinds of lookups are known as DNS Recursors, or Recursive DNS Servers.

Not every DNS resolver does all that hard work — some DNS resolvers out-source the work! So-called Stub Resolvers just accept requests from clients and then pass them on to another DNS resolver which will then do the work, report back, and allow the stub to answer the client (and take all the credit!). Home routers typically act as stub resolvers, passing the real work back to your ISP’s DNS server which is probably a full recursor.

Caching, Caching Everywhere!

If your ISPs DNS server had to do a full recursive lookup each and every time any customer browsed to any web page the internet would grind to a halt! So why doesn’t it? The one-word answers is simply caching. There are DNS caches just about everywhere!

Apps that use the network extensively, like browsers, will often keep their own DNS cache (though not always). Operating systems, which provide a DNS lookup API for use by all apps, will keep their own DNS cache. And finally, just about every resolver, stub or recursor, will keep their own cache. Any DNS resolver that keeps a cache is known as a caching resolver, and just about all DNS resolvers are caching resolvers.

I can now tell you that your home router almost certainly contains a caching stub resolver, and your ISP almost certainly provides a caching recursive resolver. Finally, I can tell you that third-party DNS servers like Google’s 8.8.8.8 are caching DNS resolvers.

Sticky Caches

According to the DNS spec, the length of time any DNS record should be cached by any DNS resolver is determined by the owner of the domain the record belongs to. When an authoritative server returns a result it attaches a so-called Time To Live, or TTL to the response. All caching resolvers should ignore all cache entries older than their TTL.

That’s the theory, but in reality, unscrupulous DNS resolver operators can force their resolvers to keep cached data for longer than that. Many ISPs do this in order to reduce the load on their resolvers. This has a nasty side-effect, when a website owner updates a DNS record, users of DNS resolvers with sticky caches will continue to see the old records for much longer than they should.

Another way in which a record can become sticky is much darker. If an attacker succeeds in sending a falsified DNS response, they will almost certainly specify an extremely long TTL in their fraudulent response, resulting in the intentionally incorrect entry hanging around in caches for as long as possible. This is known as cache poisoning, and to mitigate against this danger the operators of caching DNS resolvers can specify a maximum TTL for the cache. This means that any result with a TTL above the max will be treated as if its TTL was the max. I.e. if I configure my resolver with a max TTL of 1 hour, then if if your domain specifies a TTL of 1 year, my server will only cache the result for 1 hour.

Good DNS resolvers have short max TTLs and never cache any records for longer than their official TTL. Bad DNS resolvers will have very long max TTLs, and will hold on to records even after they expire.

Default DNS Setup

Assuming you’ve a regular users, and you’ve not changed your DNS settings, your DNS world probably works like this:

1. Your devices are set up to automatically determine which DNS resolver to use when they connect to a network using the Dynamic Host Configuration Protocol, or DHCP.
2. At home your home router responds and configures your device to use the home router as its DNS resolver.
3. Your router is configured as a stub resolver, not a recursor. When it was powered up it used DHCP to ask your ISP what resolver it should use to fulfil DNS requests from your devices. It was almost certainly instructed to use your ISP’s recursive DNS resolver.

So, when you go to www.podfeet.com your browser asks your OS to go fetch the IP for that domain, your OS asks your router, your router asks your ISP’s recursive resolver, your ISPs recursive resolver does all the heavy-lifting of actually determining the IP for you, it then replies to your router which replies to your OS which replies to your browser. There are caches all along the way, so most of the time, that sequence can be short-circuited.

Most public wifi works in exactly the same way, so you can think of a coffee shot as someone else’s house. As you move your mobile devices from network to network they use different DNS resolvers.

Security Problems

The domain name system is old, very old, so it comes from a more innocent time, where the internet only contained good people, and security threats were abstract hypothetical thought experiments!

The big three security problems with DNS are:

1. No guarantee of authenticity — DNS operates over UDP, not TCP, so there’s no three-way-handshake to ensure that the response really did come from the IP address it claims. Any device at any IP address can send a UDP packet that has a from address of any other IP. Since a DNS client will believe the first response it gets that sees to be from the server it asked the question to, an attacker who gets their fraudulent reply in first can have it accepted as the answer.
2. No guarantee of integrity — DNS provides no protections against tampering. A man-in-the-middle can arbitrarily change the content of any DNS query or response and neither the client nor server can possible detect that this has occurred.
3. A complete absence of confidentiality — all DNS queries and all DNS responses are sent over the network in plain text, so, even if you do all your actual browsing over HTTPS, a man-in-the-middle can tell where on the net you’ve been by watching your DNS traffic.

Authenticity

The authenticity problem has been strongly mitigated by a technique called source port randomisation which has been retro-fitted into the DNS. DNS servers of all kinds listen on UDP port 53, so that can’t change, but the source port on the device sending the query can be randomised, and replies that come back to the incorrect port ignored. This means an attackers who’s not a man-in-the-middle has to guess the random port before the legitimate reply arrives. Not impossible, but very difficult, and, requires a lot of network traffic which is very characteristic, and easy for firewalls and other security devices to filter out.

However, a man-in-the-middle sees the out-going request’s source port number, so they can send fraudulent replies without needing to guess. They can also drop the original query packet, then take their time constructing their fraudulent response, and lie to the requestor at their leisure!

Integrity

Cryptographically, digital signatures are how we prove that digital content of any kind has not been altered since it was signed. The core DNS protocols don’t provide for digitally signing responses from authoritative servers, DNSSEC, an extension to the original spec does.

DNSSEC has been slowly rolling out for years now, and it will probably take many more years before it becomes universally used.

Today, some important domains, including US Government domains, have DNSSEC running on their authoritative servers. This means that any resolver or client that supports DNSSEC to can cryptographically verify that an answer from one of these authoritative servers has not been altered in any way as it travelled across the internet.

DNSSEC comes at a cost — it puts a much heavier load on authoritative servers who have to sign their records, it also puts a much heavier load on resolvers and clients because they have to do a whole bunch of crypto to validate each signed response, and key management is notoriously difficult, so most authoritative DNS server operators avoid going down that rabbits warren altogether.

DNSSEC is coming, but the chances are high that today, most of the domains you visit don’t support it, and even if they did, your ISP’s DNS resolver probably doesn’t verify signed responses.

Finally — note that DNSSEC provides zero confidentiality/privacy because even signed responses are still in plain text, they are digitally signed, not encrypted!

Confidentiality

Source port randomisation is universally deployed to mitigate against the authenticity problem quite well, DNSSEC is making strides at dealing with the integrity problem, all be it slowly, but the confidentiality problem is nowhere near being solved yet.

One small concrete technology that is in use today and is making things marginally better for users of DNS resolvers that deploy it is EDNS. Normally, to facilitate location-aware-answers from authoritative servers, recursive DNS resolvers include the IP address that’s requesting the DNS record in the query sent to the authoritative servers. This makes it possible for large websites to return a European IP address for people in Europe, but an American one for people in America. This is a very useful feature, but it has obvious privacy implications.

EDNS lets us have our privacy cake while eating our efficiency cake! Rather than sending full client IP addresses to the authoritative servers, just the geographical important subnet part of the full 32bit address is sent. This means the authoritative server will know I am in Ireland, but not what my actual IP is.

It’s up to DNS resolvers to use EDNS, and today, support is far from universal, but I expect that to improve over time.

While EDNS is a step in the right direction, it doesn’t provide any protection at all from sniffing of your queries between you and the DNS resolver you’re using. For that we really do need fully encrypted DNS. At the moment there are two candidate technologies to provide just that, but they’re both in the early stages of development, and nowhere near being widely deployed or supported.

Both candidates involve wrapping DNS queries using existing encryption standards — there’s DNS over TLS, and, the one that seems to be marginally ahead at the moment, DNS over HTTPS, known as DoH.

These technologies are not designed to protect DNS packets for their entire journey between client, authoritative server, and back. Instead, they protect the packets between apps and recursive resolvers, client devices and recursive resolvers, or stub resolvers and recursive resolvers, which is generally the most sensitive part of the journey.

If you want to get adventurous, there are 3rd-party DNS resolvers that support both DNS over TLS and DNS over HTTPS out there, and, there are plugins for some browsers, apps for some OSes, and firmwares for some routers that support both. But remember, this is experimental cutting-edge stuff, it’s not likely to work perfectly smoothly just yet!

Having said all that, it seems likely that within the next few years one or both of these protocols will get widely adopted, and we will finally have a viable option for DNS privacy.

What’s Wrong With the Default DNS Setup? Why Change it?

We’ve seen the standard configuration — device to router to ISP recursor. Why change it? It just works right?!

These are some common reasons you may want to start actively managing your DNS settings:

• Speed & reliability — many ISPs are just bad at running DNS servers! They are often slow, and many are configured with very long max TTLs, and even to ignore the TTLs specified in the responses from authoritative servers. Big ISPs in particular often have very sticky DNS resolvers.
• Accuracy — in theory, if a requested domain name does not exist, a DNS resolver should respond with a so-called NX Domain response (non-existent domain). However, some ISP DNS are configured to lie, and return the IP for a web server they control instead of an NX domain so they can show you ads.
• Security — many ISPs are very bad at applying patches to their DNS servers, so they could be quite easy to poison (extra bad if they are also sticky!). Most ISPs also don’t implement DNSSEC on their resolvers.
• Conflict of Interest/Lack of Trust or Confidence — depending on where you are in the world, your ISP’s business model may set up conflicts of interest you are not comfortable with. If your ISP makes money selling you to advertisers, do you want them seeing every domain every app on every device on your network ever accesses?

Value-Added DNS

You can think of the list above as push factors driving you away from your ISP’s DNS resolver, but there can also be pull factors drawing you towards a third-party DNS resolver, like:

• Content filtering — a third-party DNS resolver can offer to lie to you in a way you like! It could do they opposite to replacing NX domain responses with IPs of ad-infested websites, and instead replace results for porn or gambling sites with NX domain responses!
• Malware Protection — third-party DNS resolvers can also return NX domain responses for domains associated with known-malware like infected websites, or command-and-control servers. This is absolutely positively not a replacement for other kinds of malware protection, but it can provide a useful extra layer of defence.
• Added Security/Future Proofing — many third-party DNS resolvers already implement DNSSEC, and some already implement either DNS over TLS or DoH. A number of third-party resolvers also use other advanced techniques to audit their caches and make extra efforts to be sure they’re not being spoofed.

Third-Party DNS Providers

Today, there are a number of big-players in the third-party DNS space. They have different focuses, and they are run by different kinds of organisations. Depending on what’s most important to you, you’ll make different choices.

OpenDNS — 208.67.222.123 & 208.67.220.123 (Primary Focus is Filtering)

OpenDNS are a for-profit corporation who started off independent, but are now owned by networking giant Cisco. They sell DNS services, and operate on a freemium model — basic features are free, more advanced features are paid.

OpenDNS have been around for many years, have a good track-record in terms of speed, reliability, and security, and are well regarded in the community.

I can’t see any obvious conflicts of interest in their business model, or any sign that they are up to anything undesirable.

Unfortunately, their website is terrible, so I was not able to find any clear information on what they do with their logs, or even what specific technologies they support. It seems likely they have fill DNSSEC support, but I wasn’t able to find that stated in black-and-white anywhere.

I was able to test for myself that they do properly return NXDOMAIN for non-existent domains.

Aside — you can use the following terminal commands to test if any DNS server returns NXDOMAIN for non-existent domains (replacing the IP address after the @ with that of the server you want to test): dig bloopbleep.podfeet.com @208.67.222.123 | grep NXDOMAIN. If that command returns output like this then the server does deal with non-existent domains properly: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 43966.

Links

• Corporate home page — www.opendns.com/…
• Family Shield, their freemium product for home users — www.opendns.com/…

Google Public DNS — 8.8.8.8 & 8.8.4.4 (Primary Focus is Speed)

We all know who Google are, and their DNS resolvers are free for everyone to use.

Google’s explanation of why the offer this free services is that a faster internet is good for Google, so by offering the world fast and reliable DNS name resolution, they are helping more people use the internet more, and hence, interact with Google services more, and hence boost Google’s bottom line.

I’m not aware of any evidence that Google are doing anything undesirable with logs from their DNS service, and their website explicitly states that they do not:

“We built Google Public DNS to make the web faster and to retain as little information about usage as we could, while still being able to detect and fix problems. Google Public DNS does not permanently store personally identifiable information”

But, the potential conflict of interest remains a concern for the future because of Google’s chosen business model — profiling people and then selling access to those people to advertisers. DNS is a potentially very lucrative source of data, so there will always be at least some temptation to start mining that data.

While their primary focus has always been speed Google have continued to add ever more security features to their DNS service, including some clever back-end protections against cache poisoning, DNSSEC, EDNS, and DoH.

Google’s DNS servers also correctly respond with and NX Domain when you look up a non-existent domain name.

Links

• The Google Public DNS home page — developers.google.com/…
• Google’s privacy page regarding their public DNS service — developers.google.com/…
• Google’s description of their security features — developers.google.com/…

Quad9 — 9.9.9.9 (Primary Focus is Malware Prevention)

The Quad9 DNS resolution service is free for everyone to use, and run by the Global Cyber Alliance, a non-profit founded by law enforcement agencies and security researchers and paid for by industry partners.

The idea is to harness the knowledge and information at the disposal of IT security researchers and firms to drive a DNS black-list which will protect users from malware and hence make the internet safer for everyone. In exchange for contributing to this project, industry partners and researchers get access to anonymised statistical data on attempts to access blocked domains. This gives the industry real-time data on which security threats are real, and which are just hypothetical at any given moment. This is how Quad9 describe the data they share with partners:

“Quad9 gives anonymized telemetry back to the TI providers only for the malicious domains they share with Quad9. This telemetry does not include source IP information of the users.”

Quad9 have a very clear policy when it comes to logging personal data, including IP addresses — they don’t! Here’s how they put it:

“We store details of the DNS records queried, timestamp, and the city, state, and country from where the query came. We do not store source IP information of end user queries.”
…
“Quad9 does not and never will share any of its data with marketers, nor will it use this data for demographic analysis. Our purpose is fighting cyber crime on the Internet and to enable individuals and entities to be more secure.”

Quad9 correctly returns NXDOMAIN, supports DNSSEC, supports EDNS, and supports DNS over TLS.

The biggest beef people are likely to have with Quad9 is their connection to law enforcement. There is no evidence they are doing anything untoward, but if you don’t trust law enforcement, I doubt you’ll feel comfortable trusting Quad9.

Links

• The Quad9 home page — www.quad9.net/…
• Quad9’s very clear FAQ — www.quad9.net/…
• A nice article describing the service from when it launched a few months ago — arstechnica.com/…

Cloudflare — 1.1.1.1 & 1.0.0.1 (Primary Focus is Privacy)

Cloudflare are the most recent entrant into this space with their new public DNS name resolution service.

Cloudflare are a for-profit company that sell content delivery services using a freemium model. Their most popular service is their freemium Content Delivery Network (CDN).

Cloudflare have partnered with APNIC, the non-profit responsible for managing IP address allocation in the Asia-Pacific region, to deliver this service.

There is data sharing as part of the partnership with APNIC, but confusingly, explicitly not DNS data that will be shared. The IP address 1.1.1.1 has been effectively unusable for decades because people who don’t know what they’re doing type it into IP address fields on settings pages on all sorts of things. It’s basically under permanent DDOS attack from the ignorant and ill-informed from all over the planet! But, because Cloudflare are a world-class DDOS protection specialists, they are one of the few companies that can successfully host the 1.1.1.1 IP. So, in exchange for letting APNIC know about all the non-DNS traffic being hurled at the IP from all over the world, Cloudflare can host their DNS service on what has to be one of the most memorable IP addresses in existence.

Cloudflare are pretty blunt about their priorities for this service — privacy first, closely followed by speed:

“We will never sell your data or use it to target ads. Period.”

“We will never log your IP address (the way other companies identify you). And we’re not just saying that. We’ve retained KPMG to audit our systems annually to ensure that we’re doing what we say.”

“Frankly, we don’t want to know what you do on the Internet—it’s none of our business—and we’ve taken the technical steps to ensure we can’t.”

Benchmarks also show that 1.1.1.1 is the fastest public DNS resolver out there today.

Cloudflare’s DNS servers correctly return NXDOMAIN, support DNSSEC, support EDNS, and support both DNS over TLS and DoH.

When it comes to following the money there is no obvious conflict of interest, Cloudflare do not make money from creating or selling profiles or from gathering or selling personal data, but you do have to believe them when they say they are doing this because they thing it’s right thing to do. To a cynic is sounds quite hippy-dippy, but, Cloudflare have a track-record on doing this kind of thing, as their CEO pointed out in his blog post announcing the launch of the service:

"The answer to why we built the service goes back to our mission: to help build a better Internet. People come to work at Cloudflare every day in order to make the Internet better, more secure, more reliable, and more efficient. It sounds cheesy, but it’s true.

When, in 2014, we decided to enable encryption for free for all our customers a lot of people externally thought we were crazy. In addition to the technical and financial costs, SSL was, at the time, the primary difference between our free and paid service. And yet, it was a hard technical challenge, and clearly the right thing to do for the Internet, so we did it. And, in one day, we doubled the size of the encrypted web. I’m proud of the fact that, three and a half years later, the rest of the industry is starting to follow suit.“
…
”When, last year, we made DDoS mitigation free and unmetered across all our plans a lot of people again scratched their heads. But it was the right thing to do. You shouldn’t have to have a big bank account to stand up to hackers and bullies online. “
…
”Part of the reason we’ve been able to hire such a great team is that we take on big challenges like this when they’re the right thing to do. Walk around the office and our team’s laptops are adorned with 1.1.1.1 stickers because we’re all proud of what we’re doing. That alone made building this a no brainer."

Links

• The 1.1.1.1 homepage — 1.1.1.1/…
• Cloudflare CEO Matthew Prince’s Blog Post announcing and explaining the service — blog.cloudflare.com/…
• APNIC’s blog post explaining the research outcomes they’re getting from this partnership — labs.apnic.net/…
• An good article explains the service from when it launched — nakedsecurity.sophos.com/…

How to Alter your DNS Settings?

So, you’ve decided you do want to switch to a third-party DNS provider, and you’ve picked the one you want to use, now what?

You have a few options:

1. Update your router’s DNS settings so it uses your chosen resolver instead of your ISPs resolver. In one fell swoop all devices on your network, even ones you can’t configure individually like IoT devices will be using your chosen resolver. You also don’t have to make any changes on any of your devices, which is handy! The only down-side is that once you leave your home, you’re back to the default setup, and using what ever DNS resolver the owner of the network you’re on is choosing to use.
2. Update each of your devices’ DNS settings so they use your chosen resolver rather than the resolver returned by the network’s DHCP service. This involves making a change on every single device! However, the changes you make will stick with the device where ever it goes. Some corporate firewalls may block DNS packets not passing through their resolver, so this configuration might be a little brittle. It’s also a lot of work, and not all devices are configurable, particularly IoT devices.
3. A hybrid Approach — update your router and your mobile devices. Anything that doesn’t leave the house is covered by your router, so only spend your energy updating the device settings on devices that actually leave your home.

Addendum (11 May 2018)

The theoretical ideal solution remains the hybrid approach, but in reality it’s not as straightforward as it should be on phones or tablets. On a laptop setting a system-side DNS resolver is easy, but that’s not true on either an iOS or an Android device. Both of the most popular mobile OSes only allow you to set DNS resolvers on a per-network basis, so the customisation would have to be explicitly applied to every network you connection to. It’s feasible to do that on networks you connect to very often like the one in work, school or your local coffee shop, but it’s unlikely you’d do it on every network you ever connect to. Thanks to listener Allister for bringing this limitation to my attention.

Useful Links

• How to update the DNS settings on your iPhone — www.imore.com/…
• How to Change DNS Entries on the Mac — www.macobserver.com/…