Greenland is being processed into the Zarr store on a single L4; day10 stopped refreshing its base images and started carrying every package it might need; a five-year-old lwt workaround turned out to be the bug; monteverde’s drive was failing after all; and a kernel CVE was mitigated across 90 hosts in an evening.
day10 #
Base refresh breaks the invariant #
day10 keeps a base image per platform and builds a layer per opam package on top of it. The whole design rests on every layer having been installed from the same package index as its siblings.
However, we need to keep the distributions, or at the very least the mirror list, up to date. To do this, ocluster-worker periodically ran day10 refresh-base which effectively runs apt update or equivalent for each distribution in each base image.
PR #31032, adding imguiml, failed 88 of 307 jobs and showed both consequences. On Debian and Ubuntu, 64 jobs died with apt reporting “Unmet dependencies”: the conf-freetype layer had pulled libbrotli1 1.0.9-2+deb12u1, sibling layers still carried the base’s +b6 stanza, and the merge of dpkg’s status files kept whichever came first. On Arch, four jobs died with pacman reporting a duplicate database entry: a layer’s whiteout over the base’s brotli-1.2.0-1 directory had been dropped by the merge, because the base is mounted beneath the merge as a lower layer of its own and the merge only looked at itself.
Those two are fixed in 0a54c2b, which keeps the highest version per package in the dpkg union using opam’s Debian-style comparison with the epoch split off first, and edaf118, which keeps a marker unless a later layer really provides the path.
Is refresh-base conceptually flawed? #
obuilder uses a weekly refresh, which isn’t perfect, as there are days where mirrors or packages are unreachable/unobtainable. However, this does mean that the images themselves are consistent, whereas with day10, I think that isn’t always true when we start overlaying packages in unintended ways. With Ubuntu and Debian, I think it could work, but distributions with continuous updates, such as Arch, will continue to cause problems.
prune --base --days N replaces refresh-base #
day10 now wipes the base image and all layers which have a depext every 7 days and rebuilds them. Consistency over the 7 days is guaranteed by prefetching all depexts at the point the base image is built, so any opam calls to install depext find the package already cached.
ocluster-worker runs the command on a schedule. Previously, the idle window actually detected the end of the idle window: the worker sat blocked on a Cap’n Proto call until a job arrived, and then the maintenance hook ran exactly when there was work to do. Now a thread wakes once a period, runs the prune if nothing is building, and otherwise logs that it will try again in an hour.
day10 cp #
The copy step was originally a perfect sudo cp applied oldest to newest, but unfortunately, this isn’t so perfect when whiteout character markers are left. This is largely caused by the original refresh-base command, but the copy step is growing in complexity with an initial sudo find followed by a sudo cp with sudo rm -rf applied to marks. Some of the later changes also required day10 to run as root, which was a big step back. sudo cp was one thing, but requiring sudo day10 seemed excessive and broke my own day10 build . use!
Effectively, the updates have been recreating the overlayfs semantics in day10. Interestingly, the overlayfs limit of exactly one memory page for the mount options has been removed from the Linux kernel, so I can now stack unlimited layers rather than creating the hard-linked tree. I decided not to do that as the RISC-V machines still run 5.10 kernel (in some cases).
day10 now provides its own version of cp via day10 cp, which implements the exact semantics it needs. A trivial benchmark showed that day10 exec -- tessera-mosaic --help took 2.37 s after the change, down from 2.46 s before, so marginally faster and certainly not worse.
fetcher #
The extra downloading of distributions, coupled with the HTTP GET storms the CI workers generate when the test matrix fans out to N distros and two compiler versions, would benefit from a cache.
I did a quick review of proxy caches which were available looking for these three specific properties: files live at <root>/<host>/<path>, unhashed, so anything can be found and deleted with ls and rm; one upstream fetch per URL however many clients ask at once, the rest waiting on the first; and nothing is ever refreshed or expired until an operator deletes it.
Clients are not proxied, because nearly every mirror is https and an https_proxy setting makes them tunnel TLS straight through, so the cache sees nothing. Instead, the URL is rewritten: https://deb.debian.org/debian/... becomes http://cache:8080/https/deb.debian.org/debian/..., and the cache terminates TLS to the origin itself.
opam can be configured to use it and similar tricks are available in the package managers.
opam option download-command='["fetcher" "download" "--cache" "10.0.0.1:8080" "%{url}%" "-o" "%{out}%"]'
It is on GitHub at mtelvers/fetcher. It is not deployed yet.
Other CI work #
A CVE across everything #
CVE-2026-98432 is a use-after-free in the IPv4 TCP handshake path. tcp_v4_syn_recv_sock hands the request socket’s IP options to the child socket without copying them, and a retransmitted SYN arriving on another CPU can read them after they are freed. It is reachable by any unauthenticated remote peer that can send a SYN carrying IPv4 options, it affects every kernel since 4.4, and the kernel CNA rated it 9.8. IPv6 is not affected.
Of the 121 machines surveyed, every one but monteverde was running a vulnerable kernel, and for almost all of them no fixed kernel had been released. Debian had shipped 6.12.113, so dopey, the only Debian host, was properly patched and rebooted that evening. monteverde happened to be on Ubuntu 26.10, whose 7.3.0-10 carries the fix, and was never vulnerable. For everything else, Ubuntu’s 6.8.0-149 for 24.04, 5.15.0-203 for 22.04, and 7.0.0-43 for 26.04 were all still sitting in -proposed.
The interim mitigation is an nftables table on the prerouting hook, so Docker-forwarded traffic is covered too, that counts and drops inbound IPv4 TCP carrying IP options, plus net.mptcp.enabled = 0.
Just under 100 machines were mitigated this way, and will stay that way until the fixed kernels are promoted.
A symlink, a stat, and a fix from 2021 #
Marek asked for symlinks on the Windows runners as odoc’s tests had been failing with a syntax error in cases/ocamlary.mli. git had checked the symlink out as a text file containing its target path. Git for Windows is installed with /VERYSILENT, which takes the installer’s defaults, and the default is core.symlinks=false. The catch is that git bakes that into every repository’s .git/config at clone time, so setting it system-wide fixed nothing for the 1,170 repositories already cached across the four workers. Unset in all of them, live, without pausing anything.
odoc’s checkout now had a real symlink, and the next run failed differently: Unix_error(ENOENT, "stat", "...\\test\\generators\\cases\\ocamlary.mli"), Internal error, the worker crashing while deleting the build context. Lwt_io.delete_recursively walks with lstat, and on Windows, when it finds a symlink, it then calls stat on it to decide between rmdir and unlink. stat follows the link. If the target has already been deleted, which it has whenever a link points at another file inside the tree and readdir visits the target first, that raises ENOENT and nothing catches it. odoc has exactly one symlink, from test into src, and src is traversed first, so every build of it failed, deterministically.
The first fix caught ENOENT and tried unlink then rmdir. However, a quick test on a Windows worker showed the rmdir leg never fired, as unlink always worked. That is weird because the entire Windows specialisation in that function was based on the assumption that rmdir was needed to remove Windows directory links and that unlink was needed for file links. Perhaps it was just that broken links could be removed with either? Testing on the worker showed this was not the case: unlink always worked. Of course, on Windows, there is a risk in assuming that the one machine you are using is representative across all Windows machines, so I was reluctant to make a wholesale simplification.
Searching the git history for ocaml/ocaml found d9cfda1e1a: Antonin Décimo, 16 September 2021 at 16:54, “making Unix.unlink remove directory symlinks on Windows”, which was released in 4.14.0, so the behaviour did change. Interestingly, earlier the same day Antonin modified the function in lwt.
As lwt requires 4.14, the entire special case is redundant on every runtime lwt supports, making the function identical on all platforms with one lstat per entry.
Raphaël’s review asked how we know the test reaches the bug, given it depends on readdir order, which was a fair criticism of my test. Based upon what I’d seen, NTFS always returned a sorted file list regardless of the creation order. However, Microsoft does not guarantee that, it just happens to be true for NTFS. If the underlying file system was FAT then it wouldn’t be. I rewrote the test to have two sibling directories cross-linked at each other, so that whichever is removed first leaves the other’s link dangling. ocsigen/lwt#1132.
core.autocrlf=true is the same installer default and was fixed at the same time. It had been putting a carriage return at the end of set -e in every shell script in every checkout.
macOS, again #
On Tuesday morning, two more of the rebuilt arm64 workers had corrupted overnight: m1-worker-04 with its first adding existent segment to range tree panic and m1-worker-03 with a data abort inside the ZFS kext. That made three of four within five days of the 21 October rebuild.
The Intel side is quieter. The /private/tmp sweep from #62 was sized against 16 GB a day; under a large batch the real rate was a 3.8 GB Homebrew source tree an hour, and i7-worker-04 was back down to 32 GB free. #63 halves the window to six hours.
OCaml 5.5.2 and new distributions #
The rollout that follows an OCaml release. ocaml-dockerfile 8.4.6 went out with the Ubuntu 26.10 change and is in opam-repository; docker-base-images#362 picks it up along with ocaml-version 4.1.5, adding 5.5.2 and 26.10 and dropping 25.04, 25.10 and Fedora 42; opam-repo-ci#490 and ocaml-ci#1076 follow, and the day10 branches of both were rebased on top.
FreeBSD, forty days on #
Before rosemary was rebuilt with OCaml 5.5.2, one last reading of the vnode fix from September: vnlru_failed frozen at 25 for forty days, through two more traffic surges that took the load average to 33 with eight jails and the store to 21,183 datasets before it pruned itself back to five thousand. No warning ever logged.
grumpy #
“193 updates could not be installed automatically” turned out to be a release upgrade from jammy to noble that had stopped halfway: base-files, dpkg and libc6 from noble, apt, systemd and the kernel still jammy, both sources lists present, and noble’s t64 library renames needing removals that unattended-upgrades will never do. The root filesystem was 7.5 GB with 1.3 free, and the volume group had 149 GB unallocated, so 40 GB online and the upgrade finished, then on to 26.04.
A side find for every 26.04 host: the garbage on the serial console at boot is systemd 259’s OSC 3008 context escape and an XTGETTCAP query that serial-getty@ sends because TTYReset=yes. A drop-in with TTYReset=no stops it.
Tessera #
Greenland for EPFL #
With monteverde back in service, I have started processing #408: dark ice on the Greenland ice sheet, a bounding box from 55 to 42 degrees west and 59 to 71 north, for 2017, 2019 and 2023.
I used mtelvers/ferry to create a pipeline similar to the one I used for 17MPU. This will process 412 shards across UTM zones 21 to 24, 5,513 live sub-windows of 6,592. 114 of the 2017 shards were already in the store from earlier work, so 1,122 remain.
The ice sheet itself is interesting in the logs. Interior shards get no usable Sentinel-1 at all, 165 of 165 dates unavailable, and the model falls back to Sentinel-2 only; the download log has 224,846 lines rejecting HH/HV polar-mode scenes against the model’s VV/VH.
As of 8am this morning: 67 shard-years uploaded, 17.3 minutes a shard, no failures overnight, about 83 shards a day, finishing around 15 November.
dClimate’s engine against mine #
On a call with dClimate earlier in the week, we discussed the scarcity of GPUs across cloud providers. We suggested using the AMX to do inference on CPUs. However, when they looked at the numbers, dClimate’s optimised Python pipeline was significantly quicker.
I could not remember a recent benchmark on their inference engine, so when monteverde was available, I tested the inference on a single tile vs my CUDA encoder on the NVIDIA L4.
| engine | per tile |
|---|---|
| dClimate v2 Large, pipelined | 129.0 to 130.3 s |
| dClimate v2 Large, serial loop | 129.8 to 131.6 s |
my CUDA encoder, TC_CHUNK=64 |
117.0 s |
Cambridge machines #
monteverde #
monteverde was reporting NVMe errors despite a spotless SMART record and the drive firmware at the latest available. I gathered logs so the issue could be reported to Dell. However, after a single reboot, the box came up in the initramfs: the PS1010 now failed every read, including LBA 0, with Internal Error rather than the SGL rejection. A controller reset, a physical reseat and a full power cycle changed nothing.
Dell replaced the drive with a Samsung-based PM1743. The rebuild on Saturday had its own obstacles. The department’s PXE rescue entries fail in GRUB with “out of memory”, so the route was a manual-install entry, a shell, a static address typed by hand because nothing offers monteverde DHCPv4, and the 26.04 server ISO written raw onto the new drive and booted from the old GRUB. Root on the BOSS mirror this time as that is a hardware RAID mirror. tank as a stripe of the two data drives, 27.9 TB, with home, scratch and the container stores on it.
On Saturday, Anil upgraded it to 26.10 with do-release-upgrade -d, and within an hour the console was logging Data SGL Length Invalid again. Always on the surviving PS1010, never on the PM1743, on the same bus. We are currently running with nvme.sgl_threshold=0, which forces PRPs for everything. Hopefully, Dell will give us a preemptive replacement.
hopper: three releases in a day #
hopper, the Grace Hopper box, was on 22.04 and needed to be updated to 26.04. This has to be done on-site, as the fans are so loud that we need to ensure the server room is empty before rebooting. The upgrade was an uneventful step through 24.04 to get to 26.04 with the latest NVIDIA drivers.
daintree’s crash was a power cut #
daintree had been restarted, which I initially assumed was an OOM crash which turned into a reboot. The amber light on the front led to racadm, and the iDRAC’s lifecycle log reported: “iDRAC firmware was rebooted with the following reason: ac”, at 22:17 on 25 October.
hopper’s previous boot ended uncleanly at 22:14 the same night, marpe’s at 22:14, and grumpy rebooted at 22:22. The common factor was that all these machines were in the same rack.
On the way back up daintree ran four firmware jobs that had been staged in May, including a BIOS from 1.12.4 to 1.17.1. The amber light itself is “Unable to authenticate System FPGA/CPLD” at every power-on since at least September. And the reboot did not change the kernel: still 6.8.0-63, with 447 updates pending.
marpe’s backplane #
marpe came up from the same power cut halted at a BIOS prompt: “Comm Error: Backplane 1”. The third backplane assertion in three months, roughly monthly. Via the serial console, I accepted the configuration to boot it, then found the one combined power and signal cable that runs from the backplane through the latch between the fans to the front edge of the board, and reseated it. If the sensor stays quiet until December, the cable was the fault.
UID standardisation #
daintree’s power cut was the opportunity to renumber ae491 and sj514. monteverde’s reinstall meant recreating its users, so they went in at their canonical numbers from the start, with mte24 finally renumbered there too. zf281 was back on daintree with 27 processes before his turn came, and remains the other half of the 1012 collision. cae37’s R jobs are still running.
Other #
conservation-copilot: the R hang #
On the VM, the Celery worker was in a fork-crash loop: 494 kills in ten minutes. Restarting everything brought the service back, but why did it fail? Fortunately, there are detailed logs, so I could replay the last query, which recreated the crash.
The issue is that when the query runs for more than the hard time limit, the billiard.Pool tries to kill the process, but the R worker blocked in native code didn’t exit and everything piled up behind it.
I replaced the nested pool with a plain subprocess: pickle in, JSON out, SIGKILL on timeout, which R cannot ignore. PR #24, merged. The R hang itself is still unexplained.
fuel #
I revisited my fuel web app, as the data was missing. Claude wrote it, and I asked for a fix and a review of the project, explaining that it was written six months ago by an earlier version of the agent.
The app was substantially rewritten, taking it from a single file to an API client, a station cache, and a thin server, each with an interface, and moving refresh into a background fibre that publishes an immutable snapshot.
The client adds per-request timeouts, re-authentication on 401 or 403, backoff retries on 429 and 5xx, result-typed errors instead of escaping exceptions, and a connector that tries IPv4 before IPv6 and falls through on failure. A failed or partial refresh now keeps the previous snapshot and retries with backoff, the frontend lives in its own HTML file embedded at build time and polls a status endpoint until data is ready, logging uses proper Logs sources, the stations endpoint supports ETag and 304.