Finally, a benchmark that compares day10 with obuilder, making day10 die cleanly and porting opam-repo-ci. Plus Tessera MODIS results showing reduced temporal stability.
day10 against OBuilder, measured #
Last week, I said what I was missing was a good A/B comparison between OBuilder and day10. A single RISC-V worker running the live OCaml-CI traffic tells you very little, because you never see the same work done twice.
It’s trivial to set up an ocurrent/ocluster scheduler, so I started one on my machine with two pools, day10 and obuilder, which results in pool-day10.cap and pool-obuilder.cap. These were deployed to two identical Scaleway RISC-V bare-metal workers.
Searching the logs of my existing OCaml-CI worker, I extracted the last 36 build jobs covering 36 git repositories. These used OCaml 5.5.0 against opam-repository 6de1938 on Debian 13. Both workers started with an empty cache.
OBuilder initially seeds its cache by pulling a prebuilt Docker base image that includes the OCaml compiler. day10 starts with the published Debian Docker image and treats the compiler as just another layer. Therefore, day10’s first build layer is the OCaml 5.5.0 compiler, which is substantially longer than the Docker extraction. However, after that initial layer, day10 is able to share the building of common packages, whereas OBuilder is only able to do this when the entire dependency set is the same.
day10 finished all 36 builds in 96 minutes. OBuilder took 255 minutes. At the moment day10 finished, OBuilder had completed 8 of the 36. 2.6 times faster.
Making day10 die cleanly #
When OCluster cancels a job, it kills day10 with SIGKILL. This didn’t give day10 any time to tear down cleanly, which left temporary directories, runc containers and overlay mounts behind. Eventually, the cluster worker got stuck at Cancelling day10 job and stayed there.
I updated OCluster to initially send SIGTERM and, after a delay, to send SIGKILL. Then I added a handler in day10 to intercept SIGTERM and tidy up before exit.
Reviewing the entire day10 codebase shows that the child process was invoked in multiple ways. Killing day10 with Control-C was also problematic, as when day10 runs things through Sys.command, day10 ignored SIGINT and SIGQUIT. Now everything goes through one Os.spawn, pipe and redirection descriptors are closed on exec, and pruned cache entries are deleted in batches rather than one xargs invocation at a time.
day10 –only-packages #
prometheus provides prometheus-lwt and prometheus-eio from the same repository. Building it with OCaml 5.5.0 is fine, but at 4.14.4 the Eio package cannot solve, because it needs OCaml 5.0 or later. OCaml-CI already works out which packages can be built for a given variant and passes that subset to OBuilder; it just had no way to tell day10. I have added a --only-packages flag to day10, and OCaml-CI now passes the compatible subset through.
opam-repo-ci on day10 #
A single HiFive RISC-V board using day10 can keep up with OCaml-CI traffic. The next target must be opam-repo-ci. This builds individual opam packages against a modified opam-repository rather than a project checkout. The closer analogue is opam-health-check, whose day10 branch is essentially the template.
Every opam-repo-ci job is a merge of master and the PR head: the worker receives two commits and runs git merge inside the OBuilder container. day10 reads a single commit from the worker’s mirror via git archive. The Day10 schema uses opamRepositoryBase @3 as the base commit for a PR merge.
day10 already supports multiple --opam-repository sources. Repo.create indexes them into one hashtable keyed by name.version, and the combine rule is first-source-wins, while every version seen across every source is accumulated. So passing the PR head first and the base second gives exact merge semantics for additions and modifications, with one exception: an overlay can never remove a package, so a deletion PR would falsely pass.
The neat part is that the exception does not need additional handling. A PR head commit is a complete opam-repository tree, not a diff. It already contains master at the branch point, minus the deletion, plus the PR’s changes. So for a deletion PR you pass the head alone and get the right answer for free. opam-repo-ci’s analysis already has an explicit Deleted case, so it knows which PRs those are.
Running opam-repo-ci locally for testing showed that valid day10 jobs were generated and built successfully.
The Tessera temporal axis #
Last week, I closed out Trentino with a green light: the MODIS BRDF correction delivers its measured 35% reduction where the swath artefact actually lives, and is a no-op within noise in mild regimes. That still holds. What it missed is an axis none of the spatial tests cover.
James ran downstream land-cover classification on v2 Trentino embeddings. Same-year nested cross-validation is neutral, matching the label-free null exactly. But temporal transfer degrades under the correction. Training on 2018, and testing on 2019, weighted F1 goes 0.843 to 0.802, and macro F1 goes 0.586 to 0.511, a 12.8% degradation, where the uncorrected embeddings actually improved across years, macro F1 0.576 to 0.626.
I confirmed it label-free on the 308-tile Alps block, same pixels, both runs. Per-pixel cross-year embedding drift is higher under MODIS: a paired difference of +0.028 between 2018 and 2019 and +0.037 between 2019 and 2020. And it is geographically diagnostic.
Red is more cross-year drift under MODIS, blue is less. The signal is +0.07 in the southern valley and plain agriculture, and about zero in the high Alps. It is not snow, it is cropland.
MODIS on Tessera v2 is now three-sided: Spatially, 35% reduction in the boundary artefact where it exists and a no-op elsewhere. Spectrally, collateral damage of about zero. Temporally, a real and class-structured cost: small in norm, two to four percent of cross-year drift, but concentrated in exactly the year-varying agriculture where rare-class discrimination lives.
That makes it a product decision rather than a correctness one. Single-year mapping and cross-swath spatial work favour the correction; multi-year model reuse and change detection over cropland favour native. Future work should train the model on corrected data rather than just correcting during inference.
Capping a 10 G firehose #
okavango was pushing about 9.8 Gbit/s outbound, and maxing out some of the interconnects in the University. Almost all of the traffic was to one host in Oxford, which was pulling from Caddy with over 1,860 connections. bond0 is 2 x 10G LACP with no shaper on it at all.
Rather than rate-limit in Caddy, I put an HTB qdisc on bond0 with two classes: a default class at 20 Gbit/s that nobody notices, and a 2 Gbit/s class with an fq_codel queue for traffic to that one destination address, matched with a u32 filter. Total outbound dropped from about 9.8 to about 2.2 Gbit/s, which is the 2 Gbit cap plus a couple of hundred megabits of everything else, and the 1,860 connections now share the cap fairly instead of monopolising the link.
Ultimately, dl2.geotessera.org needs to be retired in favour of the source.coop S3 bucket.
The branch that never got the fix #
Hannes reported that the MirageOS deployer was still trying to update 147.75.84.37, the old Equinix address. I opened ocurrent-deployer#269 to repoint it, then found that master already carried the fix from my earlier PR#268 and PR#267. However, it seems that these were never actually deployed to live.
The deployer self-deploys from the live-mirage branch, and live-mirage was simply behind master. So there was no code to write. I closed #269 as redundant and fast-forwarded the branch, and it worked end-to-end in about ten minutes: the deployer rebuilt and replaced itself on chives, having been running the same image for three months, and the new one immediately deployed a fresh www to dopey and restarted the unikernel. That also replaced a copied March 2025 image with a genuinely fresh build.