#004

I drew a bunch of architecture diagrams:

• One for our documentation about the architecture of transition:

  

• One for a talk I gave about how we handle files uploaded by publishers, comparing it to how we handle content:

  

• Another for the same talk, about the internal workflow in the asset-manager itself:

  

On Wednesday I started looking into our postgres backup system, WAL-E. We actually have two backup systems, one which takes a full backup as a SQL dump every day, and WAL-E which takes a full backup every day and incremental backups throughout the day. We want to switch to using only WAL-E, and remove the old system¹. But we can’t do that until a few outstanding issues have been solved. It took a little while to work out exactly what was wrong, and we also had our weekly sprint planning and retrospective, so I didn’t have much chance to make significant progress.

WAL-E work continued on Thursday and Friday. I found a number of different problems:

• The replicas didn’t recover after the primary restored a backup. The solution was to make it possible to also restore the backup to the replicas, after which point the normal postgres replication keeps up.

• One of our staging machines was failing to push backups to S3, as the permissions on its user were incorrect. The solution was to redeploy our Terraform configuration.

• Restoring our backups is a bit finnicky, because we encrypt them. I wrote some documentation.

• One of our staging machines (and both of our production machines!) can’t restore the backup at all, as they’re missing some permissions. We’re currently working out the best way to modify our Terraform to grant the necessary permissions.

When the final issue is done we’ll test our production backups (by restoring them to our staging environment) and, if that works, I think we can have confidence in our WAL-E backups and turn off the old system.

I picked up a change to our “getting married abroad” smart-answer. Unlike most of GOV.UK, making a change to the content of a smart-answer requires a developer to deploy the change. Content designers submit pull requests, we do some necessary fiddling, and make it live.

I fixed a piece of tech debt I’d left around. A while ago I made a change, it caused problems, and the change got reverted. Unfortunately this change added a field to a database table, and the pull request was simply reverted, with no migration to remove the field. This field has been sitting around harmlessly but confusingly in our production database since then. I got rid of it with a slightly weird conditional migration.

I upgraded all of my machines to NixOS 18.09, which worked perfectly. Good job, NixOS team!

I played Minecraft for the first time in a long while, and made a server for some friends, with a bunch of mods. For a long time I’ve found it hard to enjoy vanilla Minecraft, as there isn’t really any challenge to the game once you’ve set up a self-sustaining base; after that everything you want is just a matter of time. Mods add enough depth to offset the tedium for me.

Some new tea I ordered arrived. This is a welcome change from the Earl Grey I’ve been drinking for the past week and a half. I use an online tea shop called What-Cha, which generally has a pretty good selection, and all of my orders so far have included a small sample of something else.