Guide 7 of 7 · Staying healthy

How it stays working

Settings files do not crash when they break. A feature just quietly stops, and you find out weeks later. So this repository tests its settings the way an app tests its code.

Smoke detectors, and someone to test them

In plain words

A test is a smoke detector: it sits there quietly and shouts when something goes wrong. But a smoke detector with a flat battery is also quiet. So this repository does two things. It fits detectors everywhere, and it regularly lights a match under each one to make sure it still shouts.

Four tiers, each asking a bigger question

The quick tests run first and catch the silly mistakes. The slow ones run last and catch the subtle ones. make test runs the first three in about 35 seconds.

  1. staticDoes every file even read correctly? Syntax, linting, valid configs and a scan for secrets.make test-static
  2. unitDoes each small decision go the right way? One guard or one function at a time.make test-unit
  3. integrationDoes a real shell start up properly? The whole startup chain, in a throwaway home folder.make test-integration
  4. vmDoes it work on a Mac that has never seen it? A full install onto fresh macOS in a virtual machine.make test-vm
Wider means slower and more realistic. The VM tier runs on your own Mac only, because it needs a genuinely fresh copy of macOS.

Nothing is tested on your real home folder

Tests that start a shell build a temporary home folder first and link the repository into it, the way script/setup would. A safety check refuses to run against anything that is not a temporary folder, so a test can never damage the files it is testing.

The tests also bring their own pretend base layer: stand-in ~/.zprofile and ~/.zshrc files that behave like a managed setup. That way the layering is tested on every Mac, with and without one, whatever that Mac really has installed.

A test has to be able to fail

Most of the real bugs in this repository hid behind a test that could not fail. It passed when the feature worked and it passed when the feature was broken, which makes it worse than no test at all. Two scripts hunt for exactly that.

1 · Break it on purpose 2 · Run the test it must fail 3 · Put it back then the next one
If step 2 passes, the test is blind, and that is the thing to fix.

Watching the outside world

Dotfiles depend on things other people maintain: Homebrew packages, shell plugins, macOS itself. Those change without warning, so two scripts keep an eye on them.

CommandWhat it looks for
make auditPackages that were renamed or deprecated, plugin projects that were archived, tools the config uses but the Brewfile does not install, and config left behind for tools that are gone.
make brew-driftWhat this Mac has installed that the Brewfile does not list, and the reverse. Advice only: you decide each one.

Skipping is not passing

When the audit cannot run one of its checks, it prints NOT CHECKED and says which. It never reports a check it skipped as a check that passed.

Robots on GitHub

Four workflows run on GitHub's disposable Macs, so problems show up as a notification and not as a broken shell.

WorkflowWhenWhat it does
testEvery pull request, every push to main, and MondaysRuns the static, unit and integration tiers against both the built-in zsh and Homebrew's, then the mutation list.
provisionMondays, or on demandRuns the install scripts for real: setup, the Brewfile, macos, remove, and bootstrap from start to finish.
cold plugin cacheMondays, on demand, and any pull request that changes a plugin's pinned commitInstalls every shell plugin from nothing, the way a new Mac does, and checks each one arrived at its pinned commit.
auditMondays, or on demandRuns make audit.

A fifth workflow, pages, is not a test: it publishes these guides whenever a change to them reaches main.

The Monday runs matter because Homebrew packages are not pinned. A tool can change on a Tuesday, and the following Monday's run is what notices.

The shell plugins are pinned: config/zsh/pins.zsh lists the exact commit of each. A robot called Renovate watches them and opens a pull request, at most once a week, when any has something newer. The tests run on that pull request, so a new version is tried on GitHub's Macs before it reaches one of yours. tmux's plugins are the exception and still install at whatever is newest.

The habit

Before committing a change, both of these should come back clean.

make test
make audit

And when a change claims to make something faster, the claim comes with a measurement. The startup time has a test of its own for that reason.