Home DNS for Fun and Privacy

The technology industry loves a good metaphor. If there’s one thing better than a good metaphor, it’s a metaphor stretched so hard it bleeds. One that’s shed a good deal of blood over the the last half decade or so is that of pets versus cattle. This has encouraged the technology industry run around murdering their pets and using the carcasses to feed herds of cattle. These days, it’s been decided that the cattle lack sufficient value to be kept in fields and are being moved into factories where they are housed in little ever tinier boxes controlled by Greek sailors. You have to wonder what’s going to happen to all the waste. That’s the problem with metaphors, sometimes they lead you to dark and unpleasant places.

One of the delights of the pets that everyone wants dead was and is figuring out what to call them. I am one of those unpleasant people who genuinely enjoyed a good long discussion about a naming scheme and typically found myself veering between ever more byzantine combinations of purpose, location, quantity and idiosyncrasy, and just giving them all serial numbers (because they might be pets but they’re still going to be looked after by people who don’t care about them). Those discussions ended in lots of different places and schemes; one mostly as daft as the next. But the other place they ended up was in the endlessly fascinating domain name system. I’ve long enjoyed tinkering with DNS and as I don’t get to play with it a great deal professionally these days, I have found various justifications for vastly over-engineering my home technology setup to allow me to tinker on.

Critically important justifications for tinkering

The desire, common in technology circles, to be invisible while simultaneously taking part in society and writing things on the internet leads to all sorts of opportunities to get in a huff about being spied on. Doing these things while protecting what remains of ones’ privacy opens up all sorts of interesting avenues for tinkering with technology. As it happens, DNS is one of the technologies that offers the voyeuristic tech companies, ISPs, governments, advertisers (you know… everyone…) powerful ways to hoover up information about us. But in a happy example of circularity, it also offers opportunities for the privacy minded tinkerer to rub the noses of the voyeurs in a little of the farm waste referred to above.

If privacy weren’t justification enough, DNS is just all round important and helpful in a network of any complexity. And as any good tinkerer knows, the opportunity for tinkering increases proportionally with the level of complexity in a network. The self-reinforcing nature of this dynamic means that effective name resolution of devices in a home network is simply not optional.

Creating problems to solve

With justification in hand, I can start getting specific about what problems should be solved. As we all know, finding one problem always leads to another and tech tinkerers are typically unable to resist the urge to dive headlong into the rabbit burrows that suddenly become visible when they starts looking. The trick is knowing at what point sufficient problems have been identified, with sufficient complexity to allow for a good few months of weekend and evening tinkering, without being so endlessly complex that nothing is ever achieved. It would be nice to think I was good at knowing what sufficient looked like…

The problems that I wish to solve are the following:

  1. I need to be able to resolve the names of device in my home network as well as do reverse lookups of their IP addresses. Everything in my home gets a name. As I said, I like naming things. I have accumulated a number of domains over the years and use a real domain with no real-world records for my home.
  2. I want to block resolution of various things. Ad blocking at the DNS layer works rather beautifully and has the great advantage of minimizing the amount of time I have to spend configuring desktops, laptops, phones and the like with ad-blocking / filtering (I like complexity and tinkering, not work). It’s not just ads though, as malware, data hoovering services (e.g. crash-dump services) and others can be profitably blocked.
  3. I want to prevent my ISP capturing my DNS resolutions for their purposes or anyone else’s on their DNS servers or their sneaky transparent DNS proxies.
  4. I want to do analytics on my own (and that of my family and visitors for I am in fact, just as bad as all the other data obsessed entities in this weird world we live in). This is mostly because I’m interested in what devices do when we’re not watching but also because why not.

That’s a nicely constrained set of problems. I say constrained because there’s nothing actually stopping you going off down a number of privacy rabbit holes apart from time and the need ultimately to get on with life. Some of those rabbit holes don’t emerge until solutioning starts but some are evident in the problems themselves. The question of filtering can be fraught: should blacklists or whitelists be used? Is it really ethical to block all ads if you’re enjoying content whose production is paid for them? Who controls the lists you’re downloading and are they using them to redirect legitimate traffic to bad sites? There are lots of ways to spend a lot of time thinking (and typing…) rather than just getting on with building and documenting an actual solution.

Is there much to figure out?

As it turns out, the technology industry is full of people who think in remarkably similar ways. Many problems like those described above turn out to have been solved many times by a great many people in a great many ways. It’s worth saying again. A love of tinkering does not necessarily imply a love of hard work. Poncing off the hard work and success of others is a virtuous behaviour on which the success of the human race has been built. In solving these problems, it turned out that a lot of other people had already indeed done a lot of work.

Over the years, I’ve made great use of the documentation and tutorials built up at the wonderfully named, calomel.org. It was at Calomel that a few years ago, I discovered the combination of the Unbound caching, recursive DNS server with nsd for local name resolution (both Unbound and nsd are development by NL Net Labs) and network level ad-blocking (I use the consolidated blacklists from Stephen Black). The combination stems from the simple but sensible idea that the roles of looking up the IP addresses of unknown and untrusted devices on the internet should be separated from that of looking up the names of devices you own and trust. And so, I followed the guides set out there to:

  1. Setup Unbound as a caching, recursive resolver responsible for DNS resolution for all the devices in my home network
  2. Setup Unbound to filter ads and other things on the basis of a regularly updated blacklist
  3. Setup nsd (on the same machine - I’m not made of money, can’t be bothered with virtualization and more and am still unhappy about the young people messing up my lawn with their containers) to handle resolution of my home devices

The Calomel guides really are excellent. If you’ve read this far, I’m afraid to say that likely or not, you should just have gone to Calomel and been done with it. But if you came here for tinkering (or more likely because DuckDuckGo found some interesting keywords a little further on from here), there is plenty of that to be had yet.

Tinkering on other people’s work

The Calomel people (person?) did great and helpful work. When I was re-building my mobile VPN server recently (subject of another essay with a small amount of useful information in it at a later date), I got to wondering again about the question of DNS over TLS. Unbound as a recursive resolver is great but all those requests to root servers and onwards are in the clear. It turns out that our ISPs have been quietly sniffing these packets for a good long time, building up pictures of our internet use for what are mostly benign and even mundane purposes but occasionally no doubt for more nefarious reasons.

The simplest solution to this and one which I’ve pondered in the past is to use Unbound’s built-in, forward-zone capabilities to simply send requests to one of the big DNS providers (e.g. 1.1.1.1 / 8.8.8.8 - you know who…) over TLS. This is a solution that is catered for in one of the Calomel guides but while it solves one problem, it (rabbit hole!) creates a new one. Our ISP can no longer sniff or proxy our requests but we’re now back aggregating all our requests with one of these suppliers. Since (for me at least) making requests directly to root servers was a way of distributing my data exhaust (another of those metaphors that can be made to bleed or rather, choke in this instance), this felt like a step backwards.

That got me to researching alternatives. It turns out there are a great many, some standardized, some partially standardized and some that even work. One solution in particular seems to have a good deal of support from people who actually use DNS. This is the dnscrypt protocol and the dnscrypt-proxy software that implements it in a fully-featured way. Of course, dnscrypt and dnscrypt-proxy have their own warren of burrows to explore but on balance I decided it looked like a good solution and having already installed it on my VPN server (which does not require nsd as I don’t need to worry about my local zones while reading the paper on a railway platform) in combination with Unbound, I thought it would be a straightforward trick to integrate it into my home setup.

Tinkergrating dnscrypt-proxy

This wouldn’t be a blog about tinkering if that was actually the case. As it turns out, I couldn’t find much evidence that many people had combined Unbound, nsd and dnscrypt-proxy in the way I wanted to use it:

  1. Clients connect to Unbound running on 53 on my dns servers.
  2. Unbound checks its blacklist and returns 0.0.0.0 if it gets a hit.
  3. Unbound checks its stub-zones and forwards requests to nsd, listening on 127.0.0.1:xx53 (or so I thought)
  4. nsd responds with the address of the stub-zone device or fails
  5. Unbound responds to the client with the local zone IP address or the failure
  6. Unbound forwards any requests not matching the blacklist or stub-zones to dnscrypt-proxy listening on 127.0.0.1:yy53
  7. Dnscrypt-proxy performs lookups over TLS according to it’s round-robin policy and returns the requested IP address to Unbound
  8. Unbound returns the requested address to the client.

I should draw this up as a picture tells a thousand words. Unluckily for anyone reading this, I like words and dislike Visio.

On the Unbound side, the meat of the configuration consists of two sets of configuration attributes. The first of these are the stub-zone entries that refer Unbound to nsd for local domains. The second is the forward-zone entry that tells it where to go for forward lookups. This second attribute replaced the root-hints configuration that I had been using up to then.

Don’t use this config. It’s borked!

stub-zone:
        name: "[my_local_domain].co.uk"
        stub-addr: 192.168.0.200@xx53
[...other stubs...]
stub-zone:
        name: "[my-is-range].in-addr.arpa."
        stub-addr: 192.168.0.200@xx53
[...]
# Forward non-local requests to dnscrypt listening on 127.0.0.1@yy53
forward-zone:
        name: "."
        forward-addr: 127.0.0.1@yy53

Having set up dnscrypt-proxy and tested it (dig @localhost -p yy53 yahoo.com), and done the same with nsd (dig @localhost -p xx53 [local.addressed.thingy.co.uk]) I found that Unbound was failing in a somewhat unhelpful way. In the logs, I could see requests for resolution but no evidence that much in the way of resolution was actually being attempted. There was no evidence of requests and none of failure except that dig failed to return results. 

\[1566978437] unbound\[26325:0] info: \[REQUESTING CLIENT IP] news.bbc.co.uk. A IN
\[1566978437] unbound\[26325:0] info: resolving news.bbc.co.uk. A IN
\[1566978440] unbound\[26325:0] info: \[REQUESTING CLIENT IP] gateway.icloud.com. A IN
\[1566978440] unbound\[26325:0] info: resolving gateway.icloud.com. A IN
\[1566978440] unbound\[26325:0] info: \[REQUESTING CLIENT IP] news.bbc.co.uk. A IN
\[1566978447] unbound\[26325:0] info: \[REQUESTING CLIENT IP] www.apple.com. A IN
\[1566978447] unbound\[26325:0] info: resolving www.apple.com. A IN
\[1566978450] unbound\[26325:0] info: \[REQUESTING CLIENT IP] www.apple.com. A IN
\[1566978459] unbound\[26325:0] info: \[REQUESTING CLIENT IP] sun.com. A IN
\[1566978459] unbound\[26325:0] info: resolving sun.com. A IN
\[1566978462] unbound\[26325:0] info: \[REQUESTING CLIENT IP] sun.com. A IN
\[1566978477] unbound\[26325:0] info: \[REQUESTING CLIENT IP] www.apple.com. A IN
\[1566978477] unbound\[26325:0] info: resolving www.apple.com. A IN
\[1566978480] unbound\[26325:0] info: \[REQUESTING CLIENT IP] [LOCALLY.ADDRESSED.DEVICE]. A IN
\[1566978480] unbound\[26325:0] info: \[REQUESTING CLIENT IP] www.apple.com. A IN
\[1566978483] unbound\[26325:0] info: \[REQUESTING CLIENT IP] [LOCALLY.ADDRESSED.DEVICE]. A IN

Getting it wrong

An underapreciated aspect of tinkering is the degree to which getting things wrong is an important part of the process. I believe that proof of my efficacy as a tinkerer can be seen in the remarkable frequency with which I get things wrong. Troubleshooting this was no exception.

It’s worth noting that I was not dealing with a virgin installation. I originally built the Unbound / nsd configuration on an out of support, QNAP 419PII using Debian Jessy in 2017 (since upgraded to Stretch). I’m a fan of little ARM devices as dedicated servers and the QNAP has given fantastic service having been bought many years ago. Debian and OpenBSD are both great at supporting such things (the sets of devices they support intersect but are very different). I have to mention Martin Michlmayr for his website which contains a wealth of information on getting these devices working with Debian (and fixing them when one breaks them. Which I do. A lot).

Searching for the combination of Unbound and dnscrypt-proxy led to a few hits which on the face of it suggested that I was doing the right thing. I started to suspect that the nsd / stub-zone configuration was the problem and that there might be a conflict between the stub and forward (which uses a wildcard ’.’). More searching seemed to confirm this when I found more than one mailing list entry which suggested that stubs and forwards don’t play nicely together.

And this is where in order to get things right, I had to first get them wrong. I eventually decided that the three-way setup I was using couldn’t work. That got me investigating other options. One would be to use dnscrypt support that can be (but on Debian, is not) built into Unbound. That would require uninstalling the Unbound packages and compiling from scratch. That sounded like work to me but more importantly, the dnscrypt support in Unbound is basic and doesn’t offer capabilities like using multiple downstream servers. I also looked at using dnscrypt- proxy itself to forward local domain requests to nsd and to act as a caching resolver directly. That looked like a goer so I set it up.

To enable this, I added the following line to dnscrypt-proxy.toml:

forwarding_rules = '/etc/dnscrypt-proxy/forwarding-rules.txt'

and created the forwarding rules using the rather good dnscrypt-proxy documentation.

Next I tested dnscrypt-proxy with dig and it was able to correctly distinguish local and internet resolutions and worked perfectly. I then went back to my Unbound installation and created a simple configuration based on one from Calomel:

## Simple recursive caching DNS, UDP port 53
## unbound.conf -- https://calomel.org
#
server:
  access-control: 10.24.11.0/24 allow
  access-control: 127.0.0.0/8 allow
  access-control: 192.168.8.0/24 allow
  access-control: 172.24.11.0/24 allow
  cache-max-ttl: 14400
  cache-min-ttl: 1200
  hide-identity: yes
  hide-version: yes
  interface: [server IP address]
  interface: 127.0.0.1
  prefetch: yes
  rrset-roundrobin: yes
  use-caps-for-id: yes
  verbosity: 2

forward-zone:
   name: "."
   forward-addr: 127.0.0.1@yy53

This is bound to work I thought… Nope.

This is the point in tinkering where despair starts to set in. Everything looks right. It should be working. But it’s not. You start asking whether you should just junk the complexity and go with something simple like a pure dnscrypt-proxy setup. But that would have meant re-engineering my filtering setup and anyway would be defeatist. The fact that this configuration failed had provided some important information. It showed that the supposed conflict between stub and forward zones may not be the problem because what I was seeing in the Unbound logs was identical to what I’d been seeing in the previous configuration

At this point I decided to take the time honored step of leaving it alone for a bit to see if inspiration would hit. It did. After a delicious lunch of smoked mackerel and beetroot salad, I got back to searching and eventually found this which led me to read the unbound.conf man page on the subject of the do-not-query-localhost attribute:

[...]
Do-not-query-address: <IP address>
       Do  not  query  the given IP address. Can be IP4 or IP6. Append /num to indicate a classless delegation netblock,
              for example like 10.2.3.4/24 or 2001::11/64.

do-not-query-localhost: <yes or no>
       If yes, localhost is added to the do-not-query-address entries, both IP6 ::1 and IP4  127.0.0.1/8.  If  no,  then
              localhost can be used to send queries to. Default is yes.
[...]

The lightbulb went off. I added do-not-query-localhost: no to the simple configuration and suddenly all was well. At this point, Unbound was caching and ad-blocking, and dnscrypt-proxy was forwarding to local and internet zones. I decided to see if I could go back to my original configuration would work with the new found configuration attribute. It did…. mostly. After disabling the forward rules in dnscrypt-proxy and going back to stubs in Unbound, Unbound was able to do internet lookups but the stub-zone rules were failing. This was confusing but looking at the Unbound configuration I had a realization.

stub-zone:
     name: "[my-is-range].in-addr.arpa."
     stub-addr: 192.168.0.200@xx53

My stub-addr: entries were not using 127.0.0.1. They were using the server’s IP address (I don’t use a 192 really, they’re boring). During the tinkering, I’d noticed that nsd was configured to listen on both the server IP and 127.0.0.1. Of course, I fixed this and in the process left Unbound trying to talk to a server that was no longer listening. I headed back to the Unbound configuration and replaced the stub-addr: entries with 127.0.0.1@xx53, reloaded the Unbound configuration and found everything was working perfectly.

Wonderful. I now have exactly the configuration I wanted (whether it’s a good configuration is a different question but I like it). But this last fix begged the question as to why I had configured nsd to listen on the server’s real IP and Unbound to talk to it there rather than 127.0.0.1. The answer of course was that this was not my first rodeo. In setting up the server back in the days of Jessie, I had no doubt come across the do-not-query-localhost issue but instead of actually fixing it, had worked around it by having nsd listen on the server’s externally facing interface.

Tinkering and professionalism

And that right there is why tinkering is fine at home and for learning but leads to problems in a professional context. Separating out responsibility for different kinds of resolution reduces the risk associated with each of the components. Having nsd and dnscrypt listen only on local interfaces means that the set of capabilities exposed directly on the external interfaces is reduced. In the jargon of the industry, we’ve minimized our attack surface. By sticking nsd on the external interface to make it work I had undermined that intent and left the misconfiguration in place for the best part of two years. That’s fine at home where the greatest security risk is generally me but in a professional context, on the internet it would have been far less acceptable.

An apology

This was meant to be a quick note to remember the do-not-query-localhost issue but thinking through the whole process has been valuable. Tinkering is a real aspect of what we do in technology. It’s part of what makes technology attractive and fun. The boring part is often taking the correct configuration and formalizing it. But it’s also the most important aspect. So yes, I must apologize to the modern day technology factory farmers. They’re doing it right. But doing it wrong is still a great way to learn and I have no doubt that I and my pets will keep doing it wrong for a long time yet. And of course, I am writing this rather than getting my server configurations into Ansible…

Postscript

I found myself wondering why my VPN server just worked when Unbound there is forwarding requests to dnscrypt-proxy listening on 127.0.0.1. The answer is that my configuration there already had do-not-query-localhost: no in it’s configuration which I pinched from this site. Which just goes to show that the answer to problems is usually staring you in the face.

Postscript 2

You may well ask why I don’t just use PiHole or similar. The answer is that I do. Duh! Everything connected to my IoT network uses PiHole because I don’t care so much about them. They really are cattle.