FreeBSD 15.0 has been out for a while, with issue#1036 pending resolution. The CI update is easy, but the CI worker rosemary needed an upgrade and new base images first.
A quick poke around on rosemary confirmed the starting state.
# freebsd-version -kru
14.3-RELEASE-p3
14.3-RELEASE-p3
14.3-RELEASE-p4
The first requirement is to install the latest patches as it’s a bit behind. The host is a generic kernel, no /usr/src, single-disk EFI boot (260M ESP, UFS root on da0p2, ZFS obuilder pool on da0p3), 87 packages, and 13 obuilder build jails currently running. Nothing custom in rc.conf or loader.conf beyond a serial console and the worker service.
A 15.0 host kernel will happily run the existing 14.3 base image jails via COMPAT_FREEBSD14, but the opposite is not true.
Pausing the worker
Before any updates, pause rosemary so the cluster builds can finish in an orderly way, and outstanding jobs will remain queued.
$ ocluster-admin --connect ~/admin.cap pause --wait freebsd-x86_64 rosemary
Waiting for jobs to finish…
rosemary: Running jobs: 2
rosemary: Running jobs: 1
rosemary: Running jobs: 0
Success.
Catching up on patches
With the worker paused, install the pending patches to get to a clean baseline before the major upgrade.
# freebsd-update install
src component not installed, skipped
Installing updates...
Restarting sshd after upgrade
Performing sanity check on sshd configuration.
Stopping sshd.
Waiting for PIDS: 1967.
Performing sanity check on sshd configuration.
Starting sshd.
done.
Reboot to pick up the new kernel.
reboot
After the reboot, freebsd-version confirms a fully-patched starting point:
# freebsd-version -kru
14.3-RELEASE-p11
14.3-RELEASE-p11
14.3-RELEASE-p11
Upgrading to 15.0
The major-version upgrade is the usual three-phase freebsd-update procedure: stage the patches, install the kernel, reboot, install the new userland, rebuild ports, then a final cleanup pass. With no src component installed, there are no source merges to worry about.
# freebsd-update -r 15.0-RELEASE upgrade
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 14.3-RELEASE from update2.freebsd.org... done.
...
The following components of FreeBSD seem to be installed:
kernel/generic world/base
The following components of FreeBSD do not seem to be installed:
kernel/generic-dbg world/base-dbg world/lib32 world/lib32-dbg
Does this look reasonable (y/n)? y
...
To install the downloaded upgrades, run 'freebsd-update [options] install'.
rosemary has no locally-modified files in /etc that freebsd-update cares about, so the merge phase passed silently.
A small detour: forgetting to reboot
The correct flow is install -> reboot -> install again, with the first call installing the new kernel, the reboot activates it, and the second call installing the userland onto a kernel that already understands the new syscalls. I accidentally ran the two installs back-to-back without rebooting, which merged both stages into one:
# freebsd-update install
src component not installed, skipped
Installing updates...
Restarting sshd after upgrade
Performing sanity check on sshd configuration.
Bad system call (core dumped)
Completing this upgrade requires removing old shared object files.
Please rebuild all installed 3rd party software (e.g., programs
installed from the ports tree) and then run
'freebsd-update [options] install' again to finish installing updates.
The new 15.0 sshd had been started against the still-running 14.3 kernel and immediately tripped a syscall that the old kernel doesn’t have. As I said, COMPAT_FREEBSD14 in the 15.0 kernel handles old binaries on a new kernel; there’s no compat layer in the other direction. However, a reboot was all that was required to get back on track.
reboot
Rebuilding ports against the 15.0 ABI
The pkg ABI moves from FreeBSD:14:amd64 to FreeBSD:15:amd64, so every installed package needs to be reinstalled. pkg itself goes first, since the new binary is what understands the new ABI:
# pkg-static install -f pkg
pkg-static: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
...
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):
Installed packages to be UPGRADED:
pkg: 2.5.1 -> 2.6.2_1 [FreeBSD-ports]
...
[1/1] Upgrading pkg from 2.5.1 to 2.6.2_1...
[1/1] Extracting pkg-2.6.2_1: 100%
Then force a reinstall of every other package:
pkg upgrade -f
87 packages, a few minutes. The only post-install message worth noting was a reminder that tmux needs to be detached and restarted after a binary swap.
Final state
The normal workflow expects a final freebsd-update install to remove obsolete shared libraries left over from the old userland. But because of the missed reboot, pkg upgrade -f didn’t find any dangling 14.x .so references for the cleanup pass to find as they’d already been removed:
# freebsd-update fetch
...
No updates needed to update system to 15.0-RELEASE-p6.
# freebsd-update install
src component not installed, skipped
No updates are available to install.
Run 'freebsd-update [options] fetch' first.
freebsd-version confirms a clean 15.0 baseline:
# freebsd-version -kru
15.0-RELEASE-p6
15.0-RELEASE-p6
15.0-RELEASE-p6
Updating the EFI loader
freebsd-update does not touch the EFI System Partition, so the loader binary on the ESP is still the one included with the original 14.x install. The kernel is happy to run under an older loader.efi, but it should be updated.
rosemary keeps the ESP permanently mounted at /boot/efi, so the comparison and copy is straightforward:
# ls -la /boot/loader.efi /boot/efi/EFI/freebsd/loader.efi /boot/efi/EFI/BOOT/BOOTx64.efi
-rwxr-xr-x 1 root wheel 660992 Jul 29 2025 /boot/efi/EFI/BOOT/BOOTx64.efi
-rwxr-xr-x 1 root wheel 660992 Jul 29 2025 /boot/efi/EFI/freebsd/loader.efi
-r-xr-xr-x 2 root wheel 665088 Apr 29 08:27 /boot/loader.efi
Two files need replacing: EFI/freebsd/loader.efi is what the FreeBSD NVRAM boot entry points at, and EFI/BOOT/BOOTx64.efi is the removable-media fallback the firmware tries when no NVRAM entry matches.
cp /boot/loader.efi /boot/efi/EFI/freebsd/loader.efi
cp /boot/loader.efi /boot/efi/EFI/BOOT/BOOTx64.efi
Another reboot and resuming the worker
A final reboot to confirm the host comes up under 15.0 with the replaced EFI, then unpause the worker:
$ ocluster-admin --connect ~/admin.cap unpause freebsd-x86_64 rosemary
The four queued jobs picked straight up on the existing freebsd-14.3-ocaml-* base images, with the 14.3 jails running against the 15.0 kernel without issue.
Building 15.0 base images alongside 14.3
The next job is to build a fresh set of freebsd-15.0-ocaml-* base images on obuilder while the 14.3 ones are still serving live jobs. Three changes to ocurrent/freebsd-infra:
- Change
BSDINSTALL_DISTSITEURL to15.0-RELEASE. - Change the dataset-name template in the role from
freebsd-14.3-ocaml-tofreebsd-15.0-ocaml-. This is the image name in the obuilder spec(from freebsd-14.3-ocaml-5.4)and since it includes the versionfreebsd-15.0-ocaml-5.4can sit on disk next tofreebsd-14.3-ocaml-5.4. - Set every entry in
playbook.ymltodefault: falsefor this run. Thedefault: trueentry runs azfs cloneintoobuilder/base-image/freebsd, which already exists from the 14.3 build; with all entriesfalsethe clone step is skipped, the existing default alias keeps serving the worker, and the new datasets are just inert until promoted. Only opam-health-check uses this default alias and that’s not scheduled to run at this moment.
I dropped 5.3.0 from the version list at the same time as nothing calls it and it was only left as an upgrade-compatibility fallback. The list is now busybox, 4.14.3, 5.4.1.
The busybox image doesn’t include the FreeBSD version, so the role’s zfs destroy -R -r obuilder/base-image/busybox step does fire, and the existing 14.3 busybox is replaced with a 15.0 build. However, this is really fine as busybox is only used internally by ocluster’s worker periodic health check.
bsdinstall changes
Running the playbook against rosemary revealed that the role’s bsdinstall script has been broken by a change in bsdinstall itself. This script was originally copied from a FreeBSD installation many years ago and things have moved on:
$ pgrep -fl bsdinstall
42582 /bin/sh /usr/libexec/bsdinstall/script jail /obuilder/base-image/busybox/rootfs
42602 /usr/libexec/bsdinstall/scriptedpart
$ procstat -f 42602 | grep "0 v"
42602 scriptedpart 0 v c rw------ 9 0 - /dev/pts/1
scriptedpart was sitting on /dev/pts/1 waiting for input. The cause is that bsdinstall script now splits the script file at the first #! shebang into a preamble (sourced for variables like PARTITIONS, DISTRIBUTIONS, BSDINSTALL_DISTSITE) and a setup script. The role’s jail script was no longer in the right format to continue.
I could have figured out the new layout, but the easy fix is to stop using bsdinstall script and use the purpose-built bsdinstall jail instead. It does exactly what we want for a chroot install: no partitioning, supports nonInteractive=YES to skip the interactive dialogues, and reads the install URL from BSDINSTALL_DISTSITE.
- name: Install FreeBSD in jail //base-image//rootfs
shell: bsdinstall jail //base-image//rootfs
environment:
nonInteractive: "YES"
BSDINSTALL_DISTSITE: "https://download.freebsd.org/ftp/releases/amd64/amd64/15.0-RELEASE"
DISTRIBUTIONS: "base.txz"
Note that it’s important to wipe the cached download directory /usr/freebsd-dist as this currently holds the 14.3 installation. I have a block in update.yml for this and copied it over into playbook.yml.
Result
NAME USED
obuilder/base-image 8.11G
obuilder/base-image/busybox 746M
obuilder/base-image/freebsd 8K
obuilder/base-image/freebsd-14.3-ocaml-4.14 1.56G
obuilder/base-image/freebsd-14.3-ocaml-5.3 1.47G
obuilder/base-image/freebsd-14.3-ocaml-5.4 1.47G
obuilder/base-image/freebsd-15.0-ocaml-4.14 1.49G
obuilder/base-image/freebsd-15.0-ocaml-5.4 1.40G
obuilder/base-image/freebsd/rootfs 8K
Five base-images are now available in the same pool: the 14.3 set still serving live work, the 15.0 set ready for promotion, and obuilder/base-image/freebsd (the default alias) still pointing at the 14.3-5.4 snapshot.
The infrastructure changes are bundled in ocurrent/freebsd-infra#21.
Testing the new image
Before promoting the new image, I built a known working package mtelvers/mandelbrot targeted explicitly at freebsd-15.0-ocaml-5.4:
ocluster-client --connect ~/mtelvers.cap submit-obuilder \
--pool freebsd-x86_64 --local-file mandelbrot.spec \
https://github.com/mtelvers/mandelbrot 14e08f30f087994a19822546a55405d078acd0d3
The spec’s first step is (from freebsd-15.0-ocaml-5.4), so ocluster picks up the new dataset directly. Result:
...
FreeBSD 15.0-RELEASE-p6
The OCaml toplevel, version 5.4.1
2.5.0
...
Job succeeded
OCaml 5.4.1 runs against the 15.0, opam pulls dependencies, dune builds the package and runs the tests showing that the base image is functionally complete.
Promoting the new default
Since the image is working the default image obuilder/base-image/freebsd can be repointed at the 15.0-5.4 snapshot.
zfs destroy -r obuilder/base-image/freebsd
zfs clone -p obuilder/base-image/freebsd-15.0-ocaml-5.4@snap obuilder/base-image/freebsd
zfs clone -p obuilder/base-image/freebsd-15.0-ocaml-5.4/rootfs@snap obuilder/base-image/freebsd/rootfs
zfs snapshot -r obuilder/base-image/freebsd@snap
CI services
opam-repo-ci and ocaml-ci each have a hardcoded freebsd-X.Y distro string that needs updating. Both follow the same template as the previous 14.2 -> 14.3 pass.
For opam-repo-ci, PR#472 updates three files: opam-ci-check/lib/variant.ml (the freebsd distro constant), doc/platforms.md (the supported-platforms list and the matrix table), and test/specs.expected. The expected output is regenerated to match the new constant; dune runtest confirms the diff is consistent.
For ocaml-ci, PR#1051 updates two files: lib/variant.ml (the freebsd_distributions list) and service/conf.ml (the per-OCaml-version platform record and the fetch_platforms match that special-cases distros backed by ZFS snapshots rather than Docker images).
With both PRs green on CI, they were merged, and the changes were pushed to the live branches; the workers now submit jobs against the freebsd-15.0-ocaml-* base images by default.