This week, six million Tessera S3 objects were renamed. ocaml-gdal was extended to handle vector formats. FreeBSD CI performance investigations and OCaml compiler updates. Further work on day10 and OCaml CI.
FreeBSD CI worker rosemary
On Monday, jobs were failing on the copy opam-repository step, and there was a single git process running at 100% CPU, blocking the ocluster worker event loop. Obviously, git wasn’t the culprit; it was the manifestation of the actual problem. The git process eventually finished, but operations that normally took less than a second took 10 or more minutes. The immediate symptom was vnode starvation, with the kernel’s vnode reclaim thread consuming hours of CPU and free vnodes down to 526. The cause of that was that the obuilder store had grown to roughly 46,000 ZFS datasets over six weeks. It was not an I/O problem, since the disk was doing zero IOPS and the ARC hit rate was 98.9%. I raised the prune threshold to 50% to clear some of the backlog, but waiting on ZFS to delete the thousands of datasets was very slow. Since I have Ansible playbooks to rebuild the machine that start with zpool destroy -f, I took that as the fastest option.
Three days later, Hannes tagged me on an opam-repo-ci commit, noting that the FreeBSD workers had issues again. This time, vnodes were not exhausted, with around 108,000 free, and yet git fetch took 57 minutes, tar 25 minutes and opam set-url 8 minutes, each pinning a core while the box sat 68% idle and the disk did about 19 MB/s. The store had already regrown to 29,913 datasets in three and a half days.
Sampling the kernel with DTrace put all of the non-idle time in two places. The builds were sitting in fstatat, down through kern_statat and ending in lock_delay, which is the kernel spinning while it waits for a mutex. The code holding that mutex was vput_final into vdropl into vdbatch_process, which is the path taken every time a vnode is released.
That mutex is vnode_list_mtx, and there is exactly one for the whole system. Every file a process touches gets a vnode, and every vnode goes back onto a single global list when it is let go. At ordinary rates, this is fine, and it goes unnoticed. It is quite different when a build stats around 150,000 files to copy opam-repository, because each of those is a vnode taken and dropped, and each drop grabs that one lock. Two concurrent builds are enough to saturate it.
So the bottleneck is not the disk, not a shortage of RAM, and not git or opam. Nor is it strictly a ZFS bug: the contended lock is generic VFS code, though it is the dataset-per-build-step design, with a snapshot and a nullfs mount alongside, that generates the object count in the first place. It is a single serialisation point in the VFS layer, and what drives it is the count of filesystem objects and the rate at which they churn rather than the number of bytes involved.
The machine’s 206 GB of RAM makes this worse rather than better. FreeBSD sizes the vnode cache based on available memory, so this box has a ceiling of 6.3 million vnodes and a standing target of keeping 1.57 million free. The reclaim thread continuously pursues that target, but roughly 1.2 million vnodes are pinned by the live datasets and cannot be reclaimed, so it never arrives and simply keeps scanning.
I tried raising kern.maxvnodes, which did drop the reclaim path from the hot stacks, but the builds kept spinning and the lock delay still dominated, so it didn’t really work, and I reverted it rather than leaving it in place. Without a clear path forward, I raised the prune threshold again to 90% and redeployed. It’s not a particularly satisfactory solution, but it seems to be one of those cases where our usage falls outside the optimised path.
It got me thinking, as I do from time to time, about writing my own filesystem specifically for the CI workload. Obuilder supports many options: BTRFS, ZFS, XFS, rsync, etc., but none of these are perfect. What I really want is something designed to be a layer store rather than implementing one on top of a filesystem.
The 32-bit compiler and POWER frame pointers
The nightly rebase of my 32-bit fork failed. Upstream merged frame-pointer support for POWER on the 12th, which firstly moves the allocation pointer off r31 so that register can serve as a frame pointer, and adds 405 lines to runtime/power.S along with changes to the emitter, the register description and the stack frame code. My seventeen-commit stack carries its own heavy rewrites of exactly those files for PPC32 and PPC64 big-endian, so three of the seventeen collided.
The real work wasn’t shuffling conflict markers; it was the register renumbering. Removing r23 from the allocatable pool shifts every physical register index, so the PPC32 destroyed_at_c_call and max_register_pressure lists had to be rewritten, and then rewritten again when the next commit in the stack added another reserved register. The governing rule throughout was to adopt upstream’s r23 move and frame-pointer machinery for PPC64 while keeping it entirely excluded from PPC32, which works out because configure never enables frame pointers on the 32-bit targets.
Pushed on the 16th, and the manual run then passed the full test cycle with all four backends green: i386 and 32-bit ARM at 1,614 tests, PPC32 at 1,611 and PPC64 at 1,619, no failures.
Should the 32-bit backends gain frame-pointer support too? The r23 move was a Power-specific change to free r31. ARM32 and i386 keep their allocation pointers elsewhere. Frame pointers are opt-in profiling, not a correctness fix, and register pressure means I can’t afford a dedicated one: ARM32 has nine allocatable integer registers, and I’ve already hit “function too complex” during the earlier atomics work, and i386 has eight in total. OCaml’s own frame descriptors already handle GC and exception unwinding. The risk is that upstream later adds something that depends on the frame pointers existing.
Prometheus 1.4
Prometheus 1.4, which I had a hand in, moves the Lwt-typed functions out of the core package into a new prometheus-lwt and deprecates the originals. That now raises a deprecation alert in the CI on some other projects.
The change is simply adding the dependency on prometheus-lwt and changing Prometheus to Prometheus_lwt, e.g.
- let* data = Prometheus.CollectorRegistry.(collect default) in
+ let* data = Prometheus_lwt.CollectorRegistry.(collect default) in
ocurrent/opam-health-check#112 applies the change which fixes OCaml CI, along with updating the Dockerfile pin for opam-repository to a commit containing Prometheus 1.4, so the deployability check could solve the new constraints.
ocurrent/ocluster#265 was the same migration with a little bit more of a tidy up in the Dockerfiles as these hadn’t been updated in a few years!
Store specs
Three small PRs, all of which got better through review.
ocurrent/ocluster#264 came from Thomas noticing that the worker command in our own README dies with an uncaught exception. --obuilder-store had silently become mandatory, even though a worker without a store is a supported configuration that simply rejects OBuilder jobs. The fix parses the store into an option and turns the invalid combinations into ordinary command-line errors. Thomas’ review suggested delegating almost everything to obuilder’s own parser, which produced a cleaner second commit where only “no store at all” stays local.
ocurrent/obuilder#218 is something I spotted while writing #264. In the code that produces that message, overlayfs:/path was missing from one of obuilder’s two store lists and docker: from the other, though both are perfectly acceptable stores. And an --rsync-mode given alongside a non-rsync store fell through to the same branch, so it complained that no store type had been supplied when in fact a valid one had. That case now gets its own message. Thomas then pointed out the better fix for the missing-store case is to let cmdliner mark the option required.
ocurrent/obuilder#219 is a follow-up to Thomas’ observation that if the option is required, then the parameter needn’t be an option at all, which removes the wrapper and an unreachable case. Eleven lines in, twelve out.
day10 on more than Debian/Ubuntu
mtelvers/day10 Linux base-image generation was hard-coded to Debian, which is fine until you want it to serve ocaml-ci’s matrix of Debian, Ubuntu, Fedora, Alpine, openSUSE and ArchLinux.
The interesting decision was whether to adopt ocurrent/ocaml-dockerfile’s own image generation rather than keeping day10’s. I started out constrained to no functional change at all, relaxed that to no behavioural change, and ended up leaning against adoption. The Dockerfile it generates is 78 lines compared with day10’s 35. The sudoers heredoc also needs BuildKit. The opam installation is also more complex, as it efficiently builds all versions of opam, which is overly complex, where make cold would be sufficient for day10.
I used Distro as a lookup table only for distro_of_tag, package_manager and base_distro_tag, but not its generators, with a fallback so a distro newer than the linked library still works. The distribution-specific parts are a plain record of functions rather than a first-class module, with instances for apt, yum, apk, zypper and pacman.
Golden tests confirmed the entire refactor changes exactly one token, apt install -y <pkgs> instead of apt install <pkgs> -y, so that all three stages can share one function; the Debian output is otherwise byte-identical. Fedora, Alpine, openSUSE and Arch all build their final stage; Debian failed, and that turned out to be a red herring caused by a cached apt update layer against an updated mirror list, which prompted folding update, upgrade and install into a single RUN so a stale index can’t outlive the mirror.
I also tried it on a riscv64 machine, as I wanted to see whether the hard-coded x86-only seccomp architecture list produced an error on non-x86 platforms. It doesn’t: the generated Dockerfile came out correctly, the base stage built clean with the right uid and gid, and all three runc variants started and returned, so that list is inert there rather than fatal.
OCaml-CI with day10
The routing change itself is written: a single-file change to lib/cluster_build.ml adding an OCAML_CI_USE_DAY10 opt-in, a gate on Linux and x86_64 reflecting current single worker availability, and one match arm that sends qualifying project builds down the day10 path while lint, fmt, doc, opam-monorepo, macOS and everything non-x86_64 keep the OBuilder path. The work continues.
images.ci.ocaml.org rebuild
Last week’s base-image work was deliberately held back rather than forcing a complete rebuild immediately after having just completed one. Saturday is the natural rebuild cycle, so it went out then.
There are three changes in these new images: gc.autoDetach false, which is what unblocks opam repo add on the distributions that have moved to Git 2.55; Alpine 3.24 replacing 3.23; and the removal of Debian 12 s390x builds, which should stop the Marist workers failing on a manifest that no longer exists. So the Git 2.55 fix is now in production. This week opam-repo-ci will need to be updated to use the new Alpine release.
Two smaller CI items
conf-capnproto.2 had depext entries for macports, SUSE, FreeBSD and nixos but nothing for OpenBSD, so opam installed nothing there and the build check failed. Before opening the one-line PR I checked the conventions, which turned up three things worth knowing: bare package stems dominate OpenBSD depexts in that repository by about 35 to 14 over full pkgpaths, pkg_add fetches a prebuilt binary rather than building from ports (the gate is PERMIT_PACKAGE in the port Makefile, and capnproto permits it), and unlike the Debian and Fedora lines OpenBSD ships the headers and libraries in the same package, so no -dev companion is needed. Both currently supported releases have it.
The other is ocurrent/opam-health-check#113. FreeBSD and openSUSE have never built that repository, for want of the system dependencies on pixz, so every CI run was red regardless of the change under review, which makes a real failure indistinguishable from an unsupported platform. An available: filter fixes it, following the same approach as opam-repo-ci. The first attempt was instructive: FreeBSD disappeared and openSUSE didn’t, because opam takes os-distribution from the ID field of /etc/os-release, which is opensuse-leap, while ocaml-ci’s variant name truncates it to opensuse. Matching on os-family instead covers Leap, Tumbleweed and SLES at once and follows an idiom with several hundred packages of precedent.
OGR bindings for ocaml-gdal
My mtelvers/ocaml-gdal bindings have been raster-only. Enumerating tiles from a shapefile needs vectors, so this week I added a Vector module covering OGR: layers, features, attribute fields, geometry trees, spatial and attribute filters, and reprojection. The alternative was shelling out to ogr2ogr, which just felt untidy. The existing two-layer structure held up: a thin ctypes FFI underneath and an idiomatic wrapper returning result on top, with no hand-written C stubs.
tessera-tiles
mtelvers/tessera-tiles is a new command-line tool that turns a region of interest into a list of Tessera grid cells. It reads anything OGR can read, reprojects to WGS84, and emits grid_<lon>_<lat> names.
This tool replaces the ad-hoc Python scripts which I’d been hacking up each time a request for an ROI arrived. bin/geo.ml is a faithful port of mtelvers/genesis’s own enumerator, which means the tool and the server now agree by construction rather than by coincidence. The output format now follows ls, one tile per line to a pipe and padded columns to a terminal. The test suite is 54 cram assertions with every fixture inline.
Losing the last of the compute
We are massively grateful for the support from Vultr, but sadly, we had to return the four loan machines.
The wind-down was orderly. I added Restart=no systemd override onto all four boxes, so each of the 64 workers finished the tile it was on and then exited instead of respawning. The last worker finished 2017/grid_35.75_-13.95 and the machines were powered off.
A portable worker container
Before the Vultr machines were returned, I captured the Python environment and built a container.
Two packaging issues surfaced: Installing torch with --index-url .../whl/cpu makes pip resolve every dependency through the PyTorch index, whose repackaged wheels carry non-canonical metadata, so the resolution breaks on things like typing_extensions against typing-extensions; the fix is to install the PyPI dependencies first and then the torch wheel with --no-deps. Also, python:3.10-slim strips libexpat1, which breaks rasterio. I tested the container on monteverde.
Re-versioning source.coop
The geotessera library will now read from the Source Cooperative, so before the new client is released, the prefixes need to be renamed. Three operations, all under s3://tessera/tessera/:
| operation | objects | elapsed |
|---|---|---|
npy/v2/ to npy/v2-2B-L~beta1/ (move) |
415,004 | ~76 min |
npy/v1.1/ to npy/v1.1-cam/ (move) |
3,883,968 | ~16.5 h |
landmasks/v1.1/ to landmasks/v2/ (copy) |
1,724,656 | overnight |
As S3 has no rename, each of these is copy, verify and delete. Now was the time to do it, since the geotessera client doesn’t reference these paths yet. Only Aneesh needed to be retargetted!
Adding --checksum-algorithm CRC64NVME to aws s3 cp forces a client-side read, and on the test copy 155 MB was duly streamed through the Cambridge machine. Across 415,000 objects, roughly 22 TB would have been pulled out of source.coop’s bucket. An alternative was to drop to boto3, but that surely wasn’t necessary, and the way to settle it was to watch the network counters during a plain CLI copy and check the metadata afterwwards. This showed 0.0 MB crossing the machine despite the progress bar reporting activity. It turns out that plain copies also preserve the checksum, because the source object already carries it. Everything ran server-side at zero egress.
A full diff against both listings before the delete showed: twelve grids, 24 objects had been silently skipped. Also, the parallel delete using delete_objects with Quiet=True left 415 out of about three million objects behind. However, that aside, the boto3 delete was impressively fast, removing 3,013,797 objects in 1,813 seconds, vs roughly 55 minutes per year for serial aws s3 rm.
Reviewing the global campaign plan
I was asked to read through the plan for the global campaign, which is 1,008 zone-years across ten Ray clusters, about five days and a substantial budget, writing into a single store.
The S3 request rate will be huge, with roughly a billion objects over five days, equating to about 2,360 PUTs per second against a ceiling near 3,500 per prefix, on a keyspace S3 has to split reactively and then hold for the whole run.
One client, 8.73 TB
Caddy on the download server looked extremely busy with the CPU at 122%, driving about 8.7 Gbit/s. One client accounted for 99.2% of it. Over two hours and fourteen minutes, a host at Oxford’s ARC cluster made 219,649 of the 221,358 requests and pulled 8.73 TB, with peaks exceeding 3,200 requests per minute.
The two parquet registry files were each fetched 432 times, and individual tiles 27 to 36 times each. The client sends Connection: close and Accept-Encoding: identity, so there is no keepalive reuse and nothing is cached. There were also 11,113 404s, so part of the job is probing for tiles that don’t exist.