Week 2 of GoCardless. I started writing code this week; just some fairly simple refactoring and configuration changes, but it was nice to start to get hands-on. I’m definitely not the sort of person who likes to spend weeks just reading documentation or pairing before starting to do some independent productive work.
If you take reddit as gospel, you’d think that it takes 6+ months for someone to start to be able to independently contribute, which feels insane to me. If anything, I’d have preferred to start even sooner, last week rather than this.
I did still have a bunch of things to read, and onboarding sessions to go to, and of course I’m still lacking a lot of domain knowledge, but I’m getting there.
This week I read:
Volumes 10, 11, and 12 of So I’m a Spider, So What? by Okina Baba
There’s two strands of story in this series, separated by 15 years. It’s mostly been following the earlier strand for the last several volumes, but now the two are almost in sync: they’re only a few months apart. I think it’s worked well as a storytelling device: all the reincarnations who were born as monsters got a significant head start on those who were reincarnated as humans, because monsters mature much more quickly, so they’re more mature and are making decisions more consciously, whereas the human reincarnations are still a bunch of teenagers who, it seems, are being used by their more nefarious elders.
Kind of weird to read a story where the demon lord is the good guy. But it’s enjoyable.
I got there: I’m using my DNS server for my LAN DNS: it’s acting as an authoritative nameserver for the
lan. domain, under which I have all my other hosts (like
nas.lan.), and recursively resolving other queries. This is very cool.
Of course, after actually switching over to it for real, I found some bugs. The worst two were excessive copying of DNS blocklists (kind of bad when the blocklist I use has 99,000+ entries) and a weird inability to resolve domains where Cloudflare is the authoritative DNS.
But those are fixed now, and I’ve implemented a bunch of other improvements too.
This week I merged the following exciting PRs:
I realised that I was talking about hosts files in various parts of the code, but they didn’t actually exist as a type. Hosts files were parsed & merged into the zone structure in one go. So, to make the logic more closely fit the explanation, I made them a thing.
This was a big step. With this, I could get rid of the janky and verbose configuration needed to implement static records, and read zone files directly! Woo!
Not quite enough to read the real root hints file yet though, as that also contains AAAA records which I need to either support or ignore.
With zone files supported, the configuration file was just a list of paths and the name of a network interface. Since my longer-term goal was to support reloading hosts and zone files, I decided to just pass those directly as command-line arguments.
This was a nice and simple RFC, just defining the AAAA record type. I still haven’t implemented the ability to query upstream nameservers over IPv6, because I don’t have IPv6 at home.
Is this the single coolest PR of the week? Perhaps!
The change is straightforward: it’s just committing a new zone file to the repository. But it’s the upstream zone file from IANA.
I won’t lie, it was a bit dissatisfying having to fix issues with my zone file parser so soon. Especially as the changes in this PR are pretty significant But it’s better to have found the issues than not.
I’m now going to fuzz test parsers as a matter of course. Fuzz testing is just unreasonably effective.
The humble hosts file, so straightforward, surely my implementation of this is perfect? Nope!
For a while now I’d planned to add a program to convert a hosts file into a zone file (called
htoz, naturally). But when I started writing it I realised “hey, there are two two-state variables here, surely I should have four programs?”
Since hosts files can only hold A and AAAA records,
ztohis lossy (unless you pass a flag to make it error on unconvertable records instead). And while
ztozsound useless, they do at least normalise the output, which could be handy.
Despite adding four new programs, a refactoring in this PR actually ended up reducing the side of the repository, which I’m kind of proud of.
We got there in the end, one of my earliest goals: the ability to reload the configuration, asynchronously, without restarting the server! I worried about the performance impact of this, because it means sticking the configuration inside a read-write lock, but it turns out that the performance penalty of copying the configuration on every request, which I was doing, was way worse.
You know what’s slow? Going to three or four different nameservers to resolve a hostname. That costs precious milliseconds! So, to speed things up,
resolvedwill cache records.
How many records? 512. That’s a big number, right?
Turns out no. When you take NS and CNAME records into account, you get a lot of records! And also 512 takes up almost no memory at all! The cache size on my home server is now 1,000,000.
A big title for a big PR. This is basically just a refactoring PR, but it solves a pretty fundamental problem in the code.
There are two (well three, but we’ll get to that later) resolvers: the recursive resolver and the non-recursive resolver. Essentially the difference is that the recursive resolver queries upstream nameservers, whereas the non-recursive resolver only queries static records and the cache. But we don’t want the recursive resolver to always query remote nameservers: what about cached records? So before making any such queries it also queries the non-recursive resolver
Unfortunately, if it got an answer from the non-recursive resolver, it would then just return that to the user. That sounds good, but is actually bad! For example, let’s resolve
$ dig cache.nixos.org ... ;; ANSWER SECTION: cache.nixos.org. 3595 IN CNAME dualstack.v2.shared.global.fastly.net. dualstack.v2.shared.global.fastly.net. 55 IN A 18.104.22.168
CNAMErecord has a much longer TTL than that
Arecord. In fact, this looks to be a common way to handle CDNs. I guess for failovers? The CDN operators can change the address
cache.nixos.orgpoints to, without also having to control the
So, what happens if we query
cache.nixos.org, wait for a couple of minutes, and do it again? The
Arecord will have expired, the non-recursive resolver will return only the
CNAMErecord, and the recursive resolver will return just that to the client. Making the client unable to get an IP address for
Now, the recursive resolver already handles incomplete responses from upstream nameservers just fine. So… I made the non-recursive resolver effectively just an upstream nameserver as far as the recursive resolver is concerned (albeit one it accesses by calling a function, not one it accesses over the network). I had to do some restructuring of the non-recursive resolver to make that happen, but I think it’s worth it.
Finally, I have reached feature-parity with the Pi-hole! By adding this feature I’ll never actually use. This is the third resolver I alluded to before.
If a forwarding address is given,
resolvedwon’t use its standard recursive resolver for recursive queries. Instead it’ll just pass the query on to the given nameserver to handle, and cache the result.
Another RFC down. This one is just a configuration change.
Well, ok, it’s just a configuration change because I put in the work to properly support authoritative zones. In fact, seeing this RFC is why I decided to implement a zone file parser and support being an authoritative nameserver in the first place.
I only discovered this bug after switching my LAN over to using
You don’t appreciate the little things in life, like speedy DNS resolution, until you’ve lived without it for a few days.
For some bizarre reason, Cloudflare’s authoritative nameservers reject queries where the RA flag is set. I don’t think the RA flag means anything in queries, so since
resolvedis a recursive resolver I was just setting it in every message, whether it was a query or not. Cloudflare doesn’t return an error, it just doesn’t respond at all.
I spent so many hours trying to debug this. It was especially infuriating since
digworked fine. I spent so long staring at packet captures from wireshark, and missed this one-bit difference between what
resolvedwas sending and what
digwas sending. I thought maybe it was a firewall or OS issue, and tried other variations of those. Nope. I was completely lost.
As often happens, after getting some sleep and returning to the problem in the morning, I spotted that one byte
digwas sending was 0 and that same byte in my query was 255. Bingo.
Really, I should have stopped debugging this and taken a break sooner than I did. Computers aren’t magic, so as soon as I found myself thinking “these queries are identical, and yet are somehow being distinguished”, I should have realised it was time to stop and come back with fresh eyes later. We can always improve our practices.
The final nail in the coffin for “being an authoritative nameserver is out of scope”.
You could get almost the same effect by just not giving any root hints or a forwarding address, which would make all queries which couldn’t be answered non-recursively fail, but the RA flag would still be set in responses. So this PR switches off the RA flag and doesn’t recursively resolve no matter what the other configuration has.
And the following less exciting PRs:
- Refactor: rename
clippyexit code in CI, and also lint tests
- Make domain name handling more consistent: they’re ASCII and end with a dot!
- Fix panic in benchmarks
- Remove unnecessary serde dependency
- Ignore interface-specific addresses in hosts files
And opened several issues, all but one of which have already been closed:
So it’s been a pretty productive week! And it’s certainly very cool to be running my own DNS server in my LAN. There are still issues to resolve, and RFCs to implement, but I could stop now and this would already be more than good enough.
This project has definitely been a good learning experience. DNS is a huge eventually-consistent distributed database based on a hierarchical web of trust, but it isn’t actually that complex. I’m planning to write a “how DNS works” memo soon, before I start to forget all the little details.