I was on holiday for most of week 29, so just a few high-priority issues to report on.
check.ci.ocaml.org
Kate nudged me to look at the question raised on discuss.ocaml.org about ocurrent/opam-health-check, as it hadn’t produced a completed run since October 2025.
Every scheduled run aborted early at the opam list --available --installable step for the oldest compilers (4.08.1, 4.09.1, 4.10.2), which simply no longer solve against the opam repository. I switched the run from the 13 versions defined to the curated list, which ocurrent/ocaml-version sets as the “significant” set: 4.11.2, 4.14.4, 5.2.1, 5.4.1, 5.5.0.
With those cleared, a run finally went the distance. All 25,805 package × compiler builds completed, followed by a crash in the post-build metadata phase with Unix.EMFILE. The container’s ulimit -n was only 1024, and 3.4 GB of results were sitting there unfinalised. Rather than throw away two and a half days of compute, I reproduced move_tmpdirs_to_final by hand, building a pixz archive of the temp logs into a valid archive. The permanent fix was to raise the service nofile limit to 1,048,576, although it’s not entirely clear why this worked previously with more switches, but I suspect the next paragraph is at least part of the story.
The check.ci.ocaml.org was timing out on a 12.8-second load, which was caused by a crawler requesting every log from /log/. Very reminiscent of the crawl of opam-repo-ci and ocaml-ci last week. I switched to my custom ocurrent/caddy-ratelimit and added a redirector for @badbots -> 403 and a per-IP limit on /log/. This took the front page back to ~0.4 s with the server idle. A fresh run is now going through with all the fixes in place; on Saturday it was briefly fair-queue-starved behind a ~400-job opam-repo-ci backlog, which is expected behaviour, not a fault, so I left it to drain.
I also merged and deployed Kate’s PR ocurrent/opam-health-check#109, backporting a comma-separated package-name glob filter and a v2 REST API (/api/v2/latest/packages, with a log path per status) from opam-health-check-ng.
RISC-V, mirage-crypto’s entropy source, and U-Boot
Romain opened ocurrent/obuilder#217, a one-line change adding -cpu max to obuilder’s QEMU sandbox, as mirage/mirage-crypto’s Fortuna RNG has an open PR which would require the RISC-V Zkr scalar-crypto entropy-source extension, which the default QEMU riscv64 CPU doesn’t expose.
There are two different ways obuilder and QEMU get combined:
- obuilder runs natively inside a QEMU VM. We boot an emulated QEMU machine and obuilder uses its ordinary native backend inside the guest (Linux/runc with BTRFS snapshots; the same native backends exist for FreeBSD/jail and Windows/HCS). The
-cpuflag belongs to whoever launches the VM - for us that’s mtelvers/riscv_worker’srun.sh. - obuilder drives QEMU itself. obuilder’s own QEMU backend runs each build step in an emulated VM it manages over SSH, snapshotting to layered qcow2 files. This is how OpenBSD support works, and it’s the backend Romain’s PR patches.
Our linux-riscv64 pool, which is a mix of real hardware and QEMU-emulated machines, uses the first arrangement. So #217’s -cpu max change to obuilder’s second backend doesn’t actually touch our fleet; the flag we care about lives in riscv_worker, not ocurrent/obuilder. Thus, from my point of view the PR is in the wrong place, but the concern it raises is real wherever the workers run, so I still wanted to get to the bottom of it.
The 30-worker emulated riscv64 fleet runs -cpu rva23s64, so any package pulling in mirage-crypto could in principle fault fleet-wide. Before restarting 30 workers, I wanted to see the problem in action.
At the hardware level, rva23s64 exposes the vector-crypto bits but not zkr, and a userspace probe reading the Zkr seed CSR SIGILLs on it.
mirage/mirage-crypto#283 removes the rdtime-based seed and reads the Zkr CSR instead, with a hard #error when __riscv_zkr is undefined, and re-enables the test_entropy test on riscv/arm64. Zkr isn’t in the RVA23 -march baseline, so it won’t compile, and forcing it past the #error SIGILLs at runtime. Crucially, this isn’t emulation-specific as __riscv_zkr isn’t in the baseline that Ubuntu 26.04 targets, so #283 would break mirage-crypto on riscv64 broadly, including real SBCs and bare metal. Web research confirmed the wider point that essentially no shipping silicon implements Zkr yet (RVA23 was only ratified in 2024, and even then Zkr is optional even within it).
Testing the -cpu max on one worker hit an unexpected issue. The worker failed to boot with -cpu max, hanging in U-Boot before the kernel with initcall sequence ... err=-28 (ENOSPC). The distro U-Boot is 2022.01 (the emulation host is still Ubuntu 22.04), and -cpu rva23s64,zkr=on fails identically. So it isn’t specifically the size of max, it’s the firmware.
So I cross-built the current U-Boot. navajo is x86_64, and building riscv64 U-Boot v2026.07 with the same qemu-riscv64_smode_defconfig and jammy’s gcc-11 was easy. The new firmware clears err=-28 for both rva23s64,zkr=on and max:
-cpu |
U-Boot 2022.01 | U-Boot 2026.07 |
|---|---|---|
rva23s64 |
boots | boots |
rva23s64,zkr=on |
err=-28 |
boots |
max |
err=-28 |
boots |
I committed that to mtelvers/riscv_worker as a make uboot target plus UBOOT=/CPU= option in run.sh, with the defaults left untouched.
Tessera: the v2 “student” models
A new request came in to run a test matrix over the latest Tessera v2 distilled “student” models across four checkpoints, nano_1M (1.07 M params), small_7M (7.11 M), medium_21M (21.0 M) and large_44M (43.83 M). I ran this over the Austrian Crop 2022 data. I initially ran it on gpu4.vultr, which has seven AMD/HIP GPUs, one model per GPU, with a smaller download parallelism (the pipeline is GPU-gated) and the usual thread caps. Overnight, Planetary Computer 429s and SAS-token expiry on the heaviest ~780-scene tiles caused some stalls. Frank is now reviewing the results.
Tessera compute at ~1,750 concurrent jobs
With multiple workers per d32 instance discussed last week, the number of concurrent jobs is around 1,750. This generates embeddings at a rate of between 1,250 - 3,000 per hour, depending upon the ROI and the eviction rates.
westeurope is in a league of its own: 0.0010 evictions/box-hour. Only nine evictions across 120 boxes and 8,640 box-hours, ~50–70 times better than anywhere else. francecentral is a clear second (0.053); the other seven regions are a tight, noisy pack (0.069–0.072).
Alongside all that, the geotessera issue queue kept moving through genesis’s import -> /priority -> rescan -> close loop: UFPR (#318), a western-India request (#319, 77 tiles), a Togo request (#320, where the researcher’s low counts traced back to the legacy v1 archive), and an HKU Amazon Basin request (#321). The Tanzania backfill (Geocledian #317) continued back through 2021->2017, with Cameroon, Chad, Mozambique, Burkina Faso and Zambia queued behind it. Hopefully, Brazil will finish this week, allowing me to close a number of long-standing issues.
Source Cooperative copy
Approximately 500TB of Tessera v1 data has been mirrored to the Source Cooperative with v1.1 in progress.
Ceph
A quick look at the Scaleway RGW cluster this week. The buckets.data resize looked like it was “still splitting”, but that was a misread: pg_num is fixed at 512 (the split is done), and it’s pgp_num (471/512, climbing ~18/day) that’s still advancing as placement catches up. The ~38% “misplaced” figure hovers by design — it’s the target_max_misplaced_ratio = 0.38 throttle: backfill drains toward 38%, the manager releases the next pgp_num step, which reintroduces a little misplaced, and it drains again. Recovery is healthy at ~156 MiB/s with 60 OSDs up and no re-flaps (the heartbeat-grace fix from last week holding). The 38% cycle finished on Friday, and then a linear decrease to zero began. 18% was recorded at the time of writing.