day10 now builds on every platform obuilder does: Linux, Windows, macOS and FreeBSD; CephFS spent the week being slow; monteverde’s replacement drive has the same fault as the one it replaced; and Greenland is nearly done.
day10 #
System parity with obuilder #
day10 now supports Linux, Windows, FreeBSD and macOS, and these are running in parallel with the obuilder jobs in opam-repo-ci.
The results are better than 90% identical across the platforms. The remaining differences are largely due to the base commit differing between this week, when the day10 job ran, and potentially many months ago, when some of the obuilder jobs ran.
FreeBSD was written when day10 started, but wasn’t deployed until now. However, the macOS support was newly written this week.
macOS: a VM per build #
The macOS backend uses a new tool, day10-vm, to run each build in a Virtualization.framework VM. Unfortunately, there is no layered filesystem. There are external mounts, which are basically network shares, but as we have found on FreeBSD and Windows, these filesystems do not behave the same as a local filesystem in corner cases.
The image is built from an IPSW: macOS 15.6.1, the command-line tools, and an opam user. When a job starts, the snapshot filesystem is mounted, and the layers, the command (to be run by a one-shot LaunchDaemon in the VM) and a timestamp marker are copied over. The volume is unmounted, and the machine is booted. The command runs in the VM, with its output redirected to the virtual serial port for real-time logging. Once the command finishes, the machine is stopped, the filesystem is remounted, and the differences are determined by comparing the timestamp on the marker file.
Test builds of astring, zarith (with a Homebrew depext) and --with-test passed, and a cancel exits with 130 and leaves nothing behind. m1-worker-04 was then converted from an obuilder worker to a dedicated day10-macos-arm64 worker: its ZFS pool was destroyed and replaced with a 227 GiB APFS volume. The Virtualization framework allows at most two macOS VMs per host, which isn’t much, but it is better than obuilder’s single concurrency. Its host is on macOS 15.5 while the image is 15.6.1. It needs updating to the latest macOS, as do the obuilder workers.
Of 596 jobs, 573 matched obuilder. Of the 23 differences, 16 are base drift, 3 came from a layer-merge bug (a symlink against a directory in Homebrew’s share/gettext/its) that has since been fixed, and 2 are odnnr, which differs because day10 runs opam with --disable-sandboxing.
FreeBSD: out with unionfs #
thyme was netbooted into mfsBSD and reinstalled as a FreeBSD 15.1 day10 worker on UFS. After months of problems with ZFS-backed obuilder (now worked around), I was pleased to be using UFS.
Comparing 632 jobs with obuilder, 443 of 488 matched pairs agreed. 38 of the 40 cases where day10 failed and obuilder passed had a single cause: pkg failing with Failed to rename ... -> /usr/local/lib/gdk-pixbuf-2.0: Invalid argument. On FreeBSD’s unionfs, renaming a directory over an empty directory returns EINVAL. No combination of mount options changes it.
So FreeBSD now uses the same model as Windows and macOS: assemble a real root, build, and collect the changes with find -newercm. Real copies cost about 6 s per build. With them, conf-gtk4, conf-gdk-pixbuf, ocgtk, conf-gnuplot, conf-r and conf-mysql all build.
That exposed a problem unionfs had been hiding. With several layers merged, pkg saw only the database from the topmost layer. Each layer now records a manifest, and the database is reassembled with pkg register inside the jail, which unfortunately takes about 8 s.
Windows in opam-repo-ci #
opam-repo-ci now tests Cygwin and MSYS2 with OCaml 4.14 and 5.5 via day10. Over 53 PR heads, 952 of 960 jobs completed: 233 succeeded, 292 had no solution, 195 had a failed dependency, and 228 failed. Windows never passes where Linux fails.
Once real jobs were flowing:
dune-sitefailed on a 262-character path against Windows’ 259 limit, so opam’s root moved fromAPPDATAtoC:\opam, and navajo’s D: was reformatted as ReFS while I was at it.- The MSYS2 mirrors return HTTP 429 when they see a burst, so the base now fetches one package at a time and retries after a pause.
- A dependency cycle exited with opam’s code 60, which bypassed the classifier and gated the job. It now reports
[ERROR] dependency cycleand exits 0.
x-ci-accept-failures caused some frustration on #31195 and #31240. The convention from #28856 is cygwin-mingw and msys2-mingw, with bare cygwin meaning pure Cygwin, but day10 was matching the bare names, so flint needed both. day10 now matches only the -mingw forms. That newly silences a few packages and un-silences ortools_solvers, which lists only the bare names.
Finally, the layers are now filled and read from the host through wclayer rather than copied inside the container. Timing showed no difference, but I preferred this as the model matched the other architectures such as macOS and FreeBSD.
Cancelling a job #
Cancelling a job now works the same way on every platform: ocluster-worker interrupts, then kills. Linux sends SIGINT rather than SIGTERM, and Windows uses a small console-interrupt helper.
The Windows job logs also had dozens of blank lines. Some were CRLF from Windows stdout, including the odd \r\r\n. The remainder were from robocopy, which now logs everything to a file except for failures.
Smaller fixes #
- Arch: openssl 4.1.0 broke fresh base builds. The base had been correctly rebuilt from scratch, but the builder stages ran
pacman -Syand then-S, which is a partial upgrade. They now use-Syu. - Fedora 45: dnf5 keeps its repository files in
/usr/share/dnf5/repos.d, so themetadata_expireedit matched nothing. day10 prune --tempreclaims orphanedtemp-*directories.
Windows FsDepends leak #
Last week’s leak, with thousands of orphaned FsDepends minifilter instances and tens of GB of kernel nonpaged pool, continues to be tracked.
The model appears to be that when a single instance isn’t released, all future occurrences queue behind it and aren’t released either. Neither concurrency nor uptime seems to predict it.
Searching online suggested that EnablePerVolumeLazyWriter=0 may address this. I tried it on ReFS, and it prevents the volume from mounting at boot time. Testing is continuing on NTFS.
ltsc2025-1 and -2 on thyme have been retired. These are replaced by ltsc2025-5 and ltsc2025-6, which have much larger disks. thyme has been repurposed for FreeBSD.
Other CI work #
OxCaml health check #
My fork of opam-health-check ignores extra-repositories: for day10 builds, so instead of plumbing it through three layers, I wrote the oxcaml check as a mtelvers/ferry pipeline on doris: the oxcaml overlay in front of opam-repository, 1,972 package versions against 9 compilers, 17,748 combinations in all. The results are at oxcaml.mint.caelum.ci.dev.
opam and OCaml versions #
ocaml-ci (#1083) and opam-repo-ci (#494) now take their opam versions from Dockerfile_opam.Opam_version. ocaml-version has OCaml 4.14.5 (#95), and ocaml-dockerfile 8.4.8 adds openSUSE Leap 16.1 (#279).
macOS #
The Apple Silicon obuilder workers moved to OCaml 5.5.2. Intel macOS has also left opam-repo-ci (#495), following OCaml-CI last week.
jupiter2 #
A Milk-V Jupiter 2, with a SpacemiT K3 implementing RVA23, ran 362 jobs as a RISC-V worker before it overheated: the NVMe reached 95C. Ubuntu’s 7.3 kernel has no driver for its fan controller, and the vendor 6.18 kernel hung. It is powered off.
Cambridge machines #
CephFS: pins, strays and a crash #
The shared filesystem was slow for several days due to three reasons.
On Monday, a single metadata server was doing all the work. Our first step was to add more MDS servers and enable distributed pins for /maps and /papers. The resulting migration storm triggered heartbeat failovers, but did eventually converge and appeared to make virtually no difference to the performance.
On Wednesday, Anil’s copy of /papers to comanche was crawling. A fresh look showed most of rank 0’s work was now requests forwarded between ranks. The load came from a 9-way parallel indexer running over /maps, at 166k requests/s, which the distributed pins had spread over the same ranks as /papers. We went back to what we had started with: /papers on rank 0, /maps on rank 1, everything else on 2 and 3.
On Thursday, the real problem emerged: 24.7 million strays, hard links whose primary name had been deleted, were being reintegrated at about 10 per second. Behind each request, the cache worked hard to keep the active set in memory. Prometheus showed it had been going on for at least a week, and rank 0 already held 29 million strays in week 45. mds_dir_prefetch false took rank 0 from 52 to about 3,400 requests/s, and the strays drained from 24.6M at 07:50 to 587k by 21:11. With this set, the MDS used more RAM, which exceeded the high water mark and I feared it would continue to grow without bound, but it levelled off at double the configured setting.
Ceph housekeeping #
The Ceph daemons restarting early on Friday turned out to be an unattended upgrade of libc6, which restarted every Ceph daemon on nine hosts.
I decided to be proactive and update all 20 Ceph hosts to kernel version 6.8.0-149, to address CVE-2026-98432. I used an Ansible playbook to update and reboot each host in turn, ensuring that the cluster was stable before moving on.
Each client machine now has a single key, rather than a key per mount. This makes sense, as everything mounted on a machine is visible to all users, regardless of which key was used to mount it. POSIX permissions block access. Users still have keys in the cases where I have no access to the machine.
After a mammoth effort, the UID renumbering on daintree, kinabalu and sherwood is complete. All users across the machines now have unique UIDs.
I’ve moved 55 unclaimed items, totalling about 27 TiB, from the top level of /maps to /maps/_unclaimed for their owners to claim.
A short mains outage on Friday morning took down daintree, kinabalu and marpe. I took the opportunity to install the HWE 7.0 kernel to address the CephFS issue reported in week 43.
monteverde #
The failing PS1010 was evacuated with zpool remove, copying 1.29 TB in five minutes. Dell’s replacement is another PS1010 with the same 1.1.0 firmware. It ran 764 GiB of raw I/O without an error, but as a ZFS pool it logged 313 errors in eight minutes: 18 rejections in 1.07 million SGL-mapped commands and none in 331,000 PRP-mapped commands.
Tessera #
Greenland #
The EPFL run is at 1,041 of 1,122 items: 2017 and 2019 are complete, and 2023 is at 335 of 412.
tessera-inference #
The C++/CUDA and oneDNN encoder from the Endeavour post is published at mtelvers/tessera-inference. It matches PyTorch with a cosine similarity of 0.99997 on the L4, using the published v2 large checkpoint from Hugging Face.
Other #
32-bit OCaml #
My nightly rebase of the 32-bit backends onto trunk hit conflicts in runtime/power.S three days running. Once those were resolved, it failed for two more nights in the tests on remote workers: a stray #endif left over from the conflict resolution broke PPC32 and PPC64BE. This morning’s run, still going as I write, is the first with the fix.