Upgrading
An upgrade is two commands, and they update two different things. Running only one of them is the most common way to end up confused about what version you are on.
bothy upgrade the BOX - moves the checkout to a newer release and applies it
bothy self-update the SCRIPT - replaces the bothy on your PATHTwo paths: with the host updater, and without#
bothy upgrade does one of two things, depending on whether the host updater is
installed (just install-updater, which puts it in
~/.local/lib/bothy-updater/current):
Installed - the normal case on a box that runs Settings > Updates. It is the same plan and executor the Apply button drives (docs/plans/updates.md step 6), from a shell:
git fetch, then pick the newest release tag onorigin/mainthat is ahead of the checkout and whose commit's CI check runs all passed (asked throughghwhen it is logged in, else GitHub's API unauthenticated). Notmain's tip: a tag is whatrelease.ymlcuts only on a green commit.- Show the plan - release notes, commits, what is rebuilt, which other
stacks' files come along unapplied, whether the updater itself changes -
and ask (
--yesskips the question; a plan touching compose, edge or other stacks asks you to typebothy). - Build
bothy-web:<sha>,bothy-files:<sha>andbothy-ops:<sha>from a temporary worktree of the tag, before anything running changes. - Arm a rollback timer (
systemd-run --user), fast-forward the checkout, bring up bothy-files, bothy-ops and bothy-web - web last - on those images, and verify: the images and their labels,/version.json, the/healthzbodies, the catch-all'sindex.html. - Verify passing disarms the timer. Verify failing - or the updater dying, or
WSL restarting - lets it fire:
git reset --hardback to the previous commit (safe because step 1 refused a dirty tree) and the previous images, without a build. That release is then not offered again.
It refuses, and says why, on a dirty or untracked tree, a branch other than
main, a checkout that has diverged fromorigin/main, a tag that is not onmainor not green, and a release that touches the socket-proxy boundary. It updates Bothy's own three services only: other stacks' pins that arrive with the checkout are then offered on their own rows in Settings > Updates.Not installed - a fresh install, CI, a box that never ran
just install-updater. The original path below:main's tip, pulled and applied, with no rollback. It says so when it runs.
If a release changes the updater itself, the new copy is staged and not
switched: the program doing an update never replaces itself mid-run. Settings >
Updates and just update-status say a switch is pending; just install-updater
makes it.
What bothy upgrade does without the updater#
Four steps, and it stops at the first that fails:
- Find the checkout (The
bothycommand explains how) and enter it. git pull --ff-only. Fast-forward only, so a checkout with local commits or local edits refuses to upgrade rather than merging. That is deliberate: the alternative is a merge conflict inside a directory that is also a running system.- If the commit did not move, it says so and stops. There is nothing to apply.
just up.
That last step is the whole of "apply". There is no down, and no -v.
Bothy's own code is rebuilt, not just restarted#
Most of the stack is third-party images named by a pinned tag, and a new pin is
applied by up pulling it. Bothy's own three services - bothy-web, bothy-files
and bothy-ops - are different: they are built from this checkout, and
up on its own never rebuilds an image that already exists. So just up-apps
(which just up runs) builds them first, then brings them up:
docker compose -f apps/bothy/compose.yml build # a no-op when nothing changed
docker compose -f apps/bothy/compose.yml up -d --wait ...The build is cached layer by layer, so an upgrade that did not touch the apps
costs seconds. It does need the network when their sources or base images
changed (npm ci, and the pinned node, nginx and python bases), which is
why bothy download builds them too.
Before 2026-09-18 this step was missing, and an upgrade applied every new compose file and none of the new app code - the old images kept running, healthy, with nothing to say so.
Each image is labelled with the commit it was built from, so you can check:
docker inspect -f '{{index .Config.Labels "org.opencontainers.image.revision"}}' bothy-web
git -C ~/stacks rev-parse HEAD # should match
curl -s http://<box>/version.json # the same, as a browser sees itThe label is HEAD at build time, so every new commit relabels the three
images and just up recreates those three containers - a few seconds of
downtime for Bothy's own pages, and nothing else restarts.
The image tag is the commit too: bothy-web:<sha> (compose reads
BOTHY_IMAGE_TAG, which just up-apps sets to HEAD's sha). The previous
commit's images therefore survive a build under their own name, which is what
lets the updater's rollback run them again without building anything. up-apps
keeps each image's three newest commits and never removes one a container runs.
An open tab notices an update by itself: it polls /version.json and shows
"Bothy updated - reload" when the served revision is no longer the one it
loaded. A lazy-loaded page part that no longer exists reloads the tab once.
Update, and Apply#
Two different actions, and confusing them is the other common way to be unsure what version you are on:
- Update moves a pin in
main- a Dependabot PR, reviewed and merged. It happens on GitHub. The box takes no part in it, deliberately: git stays the only record of what this box should run (docs/plans/updates.md, step 4). - Apply makes this box run what
mainalready pins. That is what Settings > Updates does, and all it does.
A box can be fully up to date on the first and far behind on the second - that is the normal state the morning after a batch of Dependabot PRs merges - so the page counts them separately and leads with the second.
Several pins in one recipe#
just up-<recipe> is one docker compose up over the whole project, and it
recreates every service whose definition or image changed. That is why
Settings > Updates refuses to apply ONE component while another service of the
same project also has a merged pin waiting: that other one would be recreated
too, with no snapshot of it - and if it is Grafana or Keycloak, its first start
migrates a database that the old version cannot read.
Where that happens the page offers the group: the whole recipe at once, with
every member's own snapshot taken first (the one-way ones first of all), the
strictest confirmation any member needs, and a rollback that puts all of them
back together - there is no keeping the ones that worked, because that would
be a combination of versions no commit on main describes.
From a shell, just update-groups prints what each recipe would apply and, for
one it refuses, exactly which service it will not carry and why. just backup
then just up-<recipe> by hand is the other answer, and the only one for a
recipe whose other services are things the updater does not deploy at all (the
auth boundary, the edge).
Your data survives and the new code runs, and both are tested#
just up recreates containers whose definitions or images changed and leaves
volumes alone. That this preserves data and applies the new code is asserted in
CI by the Upgrade
workflow (.github/workflows/upgrade.yml) on every merge to main. It installs
two starting points - the previous commit, and the newest release, which is
what your box is most likely running - writes a row to the database, upgrades
to HEAD, and then checks three things:
- the row is still readable;
- no volume was recreated - compared by volume ID, not by name, because a recreated volume keeps its name and comparing names would pass exactly when the data was lost;
- the running code is
HEAD's - bothy-web, bothy-files and bothy-ops run images labelled with the new commit,/version.jsonnames it, and theindex.htmlbothy-web serves is byte-identical to a fresh build ofHEAD's sources. The data checks alone passed for months while no upgrade applied app code at all.
So the thing that would break this is not an ordinary upgrade. It is somebody
adding a down -v to the apply step, renaming a volume in a compose file, or
dropping the build from up-apps - which that workflow exists to catch. A
release is only tagged once that workflow, the install workflow and CI have all
passed on the exact commit being tagged.
Take a backup anyway before an upgrade you are unsure about. See Backups; it is one command and the restore path is documented.
Upgrading with no network#
bothy download pre-fetches every image named by every compose file in the
checkout - not only the ones up starts, because the point of the command is
that a later up needs no network at all, and that includes the tiers you may
start by hand. pull skips Bothy's own three images, which are built rather
than pulled, so it also builds them - which fetches their base images and
the portal's npm packages - leaving the build inside up all cache hits. It
also runs npm ci for the portal's web sources if npm is present, for local
development; the image build does its own.
The order for an offline or slow-link upgrade is:
bothy upgrade # pulls the checkout and builds the apps (needs the network)
bothy download # pulls the images
bothy upupgrade already runs up at the end, so on a good link the middle step is
just insurance. If the link drops partway, run bothy download and then
bothy up: every build step that already finished is cached.
After upgrading, update the CLI too#
The bothy on your PATH is a copy of scripts/bothy, made when you
installed. upgrade pulls the repository and does not touch it. That drift is
harmless right up until the day a subcommand is added, removed or renamed - and
then a dispatcher is resolving what you typed against a justfile it no longer
matches.
bothy self-updatescripts/checks/cli-commands.sh asserts that every subcommand the CLI names
still resolves - but it asserts it inside a checkout. Nothing checks the
copy on your PATH, which is exactly why the command exists.
Checking what you are on#
bothy versionVersion, short commit, (modified) if the working tree is dirty, and the
checkout path. The (modified) marker is the one that matters when a box
behaves unlike its version number says it should: it is the only thing that
distinguishes two machines claiming the same release.
When an upgrade goes wrong#
pull failed - the checkout has local changes or has diverged. Expected, and the right answer is not--force. Look atgit statusin the checkout. If the change is yours and wanted, commit it on a branch; if it is not, it is usually an edit made through Bothy Files, and the file's history is in the reader.- The pages look unchanged after an upgrade. Compare the revision label
(above) with
git rev-parse HEAD. If they differ, the build did not run or failed -just up-appsagain shows why. A browser tab that was already open keeps the old page until it reloads. - A container will not start after an upgrade.
bothy doctorfirst - it covers containers, ports, routes, targets, DNS and disk, and the failing line is often not the one you expect. See Troubleshooting. - A setting you changed has stopped taking effect. Editing is not applying. A changed label or port does nothing until the container is recreated; see The files you will actually edit.
Next#
- Backups - what to take before an upgrade you are unsure about
- Troubleshooting - when the box comes back up wrong