Guide 7 of 7 · Staying healthy
How it stays working
Settings files do not crash when they break. A feature just quietly stops, and you find out weeks later. So this repository tests its settings the way an app tests its code.
Smoke detectors, and someone to test them
In plain words
A test is a smoke detector: it sits there quietly and shouts when something goes wrong. But a smoke detector with a flat battery is also quiet. So this repository does two things. It fits detectors everywhere, and it regularly lights a match under each one to make sure it still shouts.
Four tiers, each asking a bigger question
The quick tests run first and catch the silly mistakes. The slow ones run last and catch the subtle ones. make test runs the first three in about 35 seconds.
- staticDoes every file even read correctly? Syntax, linting, valid configs and a scan for secrets.make test-static
- unitDoes each small decision go the right way? One guard or one function at a time.make test-unit
- integrationDoes a real shell start up properly? The whole startup chain, in a throwaway home folder.make test-integration
- vmDoes it work on a Mac that has never seen it? A full install onto fresh macOS in a virtual machine.make test-vm
Nothing is tested on your real home folder
Tests that start a shell build a temporary home folder first and link the repository into it, the way script/setup would. A safety check refuses to run against anything that is not a temporary folder, so a test can never damage the files it is testing.
The tests also bring their own pretend base layer: stand-in ~/.zprofile and ~/.zshrc files that behave like a managed setup. That way the layering is tested on every Mac, with and without one, whatever that Mac really has installed.
A test has to be able to fail
Most of the real bugs in this repository hid behind a test that could not fail. It passed when the feature worked and it passed when the feature was broken, which makes it worse than no test at all. Two scripts hunt for exactly that.
-
make mutate
Does this test still work?
Reads a list of deliberate breakages from
test/mutations.tsv. Each row names a file, how to break it, and the test that must notice. Every row is a bug that really shipped, or the failure a test was written to catch. -
make sweep
Is this line tested at all?
The opposite question. It switches off each line of config in turn and reports the ones no test noticed were gone. It is slow, so it works to a time budget and picks up where it stopped.
Watching the outside world
Dotfiles depend on things other people maintain: Homebrew packages, shell plugins, macOS itself. Those change without warning, so two scripts keep an eye on them.
| Command | What it looks for |
|---|---|
make audit | Packages that were renamed or deprecated, plugin projects that were archived, tools the config uses but the Brewfile does not install, and config left behind for tools that are gone. |
make brew-drift | What this Mac has installed that the Brewfile does not list, and the reverse. Advice only: you decide each one. |
Skipping is not passing
When the audit cannot run one of its checks, it prints NOT CHECKED and says which. It never reports a check it skipped as a check that passed.
Robots on GitHub
Four workflows run on GitHub's disposable Macs, so problems show up as a notification and not as a broken shell.
| Workflow | When | What it does |
|---|---|---|
test | Every pull request, every push to main, and Mondays | Runs the static, unit and integration tiers against both the built-in zsh and Homebrew's, then the mutation list. |
provision | Mondays, or on demand | Runs the install scripts for real: setup, the Brewfile, macos, remove, and bootstrap from start to finish. |
cold plugin cache | Mondays, on demand, and any pull request that changes a plugin's pinned commit | Installs every shell plugin from nothing, the way a new Mac does, and checks each one arrived at its pinned commit. |
audit | Mondays, or on demand | Runs make audit. |
A fifth workflow, pages, is not a test: it publishes these guides whenever a change to them reaches main.
The Monday runs matter because Homebrew packages are not pinned. A tool can change on a Tuesday, and the following Monday's run is what notices.
The shell plugins are pinned: config/zsh/pins.zsh lists the exact commit of each. A robot called Renovate watches them and opens a pull request, at most once a week, when any has something newer. The tests run on that pull request, so a new version is tried on GitHub's Macs before it reaches one of yours. tmux's plugins are the exception and still install at whatever is newest.
The habit
Before committing a change, both of these should come back clean.
make test
make audit
And when a change claims to make something faster, the claim comes with a measurement. The startup time has a test of its own for that reason.