The main threads: making day10 a native OCluster job type, porting opam-health-check to use day10, keeping the CI dashboards ahead of both a scraper farm and the opam 2.6 alpha, and a lot of Tessera. New v2 “student” models, the v1.1 mirror to Source Cooperative continues, and diagonal banding in the Amazon mosaic.
day10 as an OCluster job type
The ocurrent/ocluster cluster runs Docker and OBuilder jobs. OBuilder caches and shares identical steps between jobs, but for shared opam dependencies each is built from scratch. For example, opam install dune foo and opam install dune foo bar would rebuild both dune and foo in order to add bar. mtelvers/day10 instead builds each dependency once as a content-addressed layer and merges layers on demand, so the goal was to make the OCluster worker accept day10 jobs the same way it already accepts Docker and OBuilder ones.
I started by surveying what the farm actually runs. I pulled the job specs that journalctl records across five Linux workers, and confirmed what I expected to see: the ~1,800 daily jobs reduce to just four spec families: opam-repo-ci (1,396), opam-health-check (972, overnight), ocaml-ci (283) and mirage-ci (134), plus rare one-offs. That identified opam-health-check as a good basis for testing and showed what a day10 job specification should look like.
Each worker already keeps a shared git mirror of opam-repository under /var/cache/obuilder/ocluster/git/, which is populated lazily whenever a job names opam-repository as its repository. Combined with day10’s new support for reading a specific commit straight from the git object database (piece below), the ~30,000-file tree never has to be copied or checked out as git archive streams the exact commit to an in-memory tar file, which is indexed with Tar.fold. The worker gains a Day10 case in its custom-job dispatch, leaving Docker and OBuilder untouched: a new Day10 payload in schema.capnp, a day10_dispatch.ml, a checkout-free ensure_opam_repository reusing the existing mirror, and a --day10-cache worker flag that gates day10 jobs and keeps day10’s own cache separate from the OBuilder store.
Getting there needed three small day10 fixes, each surfaced by real use:
- The “read a commit from the git DB” form worked for
build/ci/health-checkbut notlist, which simply wasn’t updated with the new:shafunctionality that was added. Reroutingrun_listthrough the sameRepopath as the other subcommands fixed it, and also fixed a cryptic failure mode where a bad SHA surfaced as an opaquetar fold failed.Os.runhad been discarding the exit status ofgit rev-parse, so a failed resolve returned an empty string thatgit archivecouldn’t use. Sys.getenv "HOME"runs early to read the global~/.day10, but raises under systemd units where$HOMEisn’t set. Now it’s anOption.iterovergetenv_opt.- The
tarversion 3.5.0 now makes theRead/Really_read/Seeksignatures carryint64; I matched the guard-then-convert idiom tar-unix uses rather than a bare truncation.
Running the health check through day10 also meant adapting ocurrent/opam-health-check. At the same time, I updated to the redesigned docker_hub 0.2.0 API, which, as a side benefit, skips the “unknown” attestation manifests that recent images like ocaml/opam:debian-13 carry and that 0.1.1 couldn’t parse. This is an actual issue on the real server which has been quietly falling back to the existing SHA rather than updating to the latest.
Then the payoff. I borrowed doris into a test pool and ran opam-health-check through the day10 path, configured to mirror the live check.ci.ocaml.org switches exactly: 4.11.2, 4.14.4, 5.2.1, 5.4.1, 5.5.0 on debian-13, ~4,500 packages each:
| Run | Mechanism | Duration |
|---|---|---|
| Live check.ci.ocaml.org (production pool) | OBuilder | 40h 12m |
| day10, cold cache, one worker | day10 | 4h 51m |
| day10, warm cache, same opam-repo commit | day10 | 1h 36m |
Note: this isn’t true like-for-like testing, other than the number of concurrent jobs being the same. The real health check was using a contended obuilder pool, where the day10 version had a dedicated one. Furthermore, the day10 version used obuilder for rev-deps, and the warm-cache results should be much better. However, it’s a starting point. check.mint.caelum.ci.dev
check.ci.ocaml.org outage
Midweek, the opam-health-check site was overwhelmed by a distributed scraper farm hammering /log/* with spoofed browser user agents. In opam-health-check, /log/ requests spawn a pixz to extract the logs, but the client hangs up early, saturating the single-core server. Per-IP limits and UA blocks both miss a spoofed, distributed farm, so I moved /log/ to a global rate cap (5/s at Caddy, excess 429’d before it reaches the backend). I’ll need to revisit this with a proper fix. The issue is the same as on opam-repo-ci. We can’t block scrapers, so we must make scrapes cheap.
opam-repo-ci and opam.2.6.0~alpha1
Base images are built and released slowly to opam-repo-ci due to the way various schedules interact. Jan started noticing a new class of failure:
/home/opam: (run (network host)
(shell "opam init --reinit --config .opamrc-sandbox -ni"))
Configuring from /home/opam/.opamrc-sandbox, then /home/opam/.opamrc, and finally from built-in defaults.
Checking for available remotes: rsync and local, git.
- you won't be able to use mercurial repositories unless you install the hg command on your system.
- you won't be able to use darcs repositories unless you install the darcs command on your system.
This version of opam requires an update to the layout of /home/opam/.opam from version 2.1 to version 2.6~alpha, which can't be reverted.
You may want to back it up before going further.
Perform the update and continue? [Y/n] y
[NOTE] The 'jobs' option was reset, its value was 1 and its new value will vary according to the current number of cores on your machine. You can restore the fixed value using:
opam option jobs=1 --global
Upgrading the internal repository format for 'default'...
Upgrading the internal repository format for 'ocaml-patches-overlay'...
Format upgrade done.
<><> Rerunning init and update ><><><><><><><><><><><><><><><><><><><><><><><><>
<><> Updating repositories ><><><><><><><><><><><><><><><><><><><><><><><><><><>
[ocaml-patches-overlay] no changes from git+https://github.com/ocurrent/opam-repository#patches
[default] no changes from git+file:///home/opam/opam-repository
Update done, please now retry your command.
"/usr/bin/env" "bash" "-c" "opam init --reinit --config .opamrc-sandbox -ni" failed with exit status 10
2026-07-25 11:06.25: Job failed: Failed: Build failed
This is because opam-dev is now 2.6.0~alpha1, and on first use it migrates the root from 2.1 to 2.6 and prints “please now retry your command” and deliberately exits 10, which OBuilder treats as a build failure. Because lib/build.ml sets opam_version to the `Dev variant globally, this will eventually break every build. I reproduced it and confirmed no flag avoids the abort (-y, --confirm-level=unsafe-yes, OPAMNOAUTOUPGRADE=1, --cli=2.1 all still exit 10); only a second, identical init returns 0.
I’ve added a hack opam init --reinit … || opam init --reinit …, with || specifically to avoid double-fetching the overlay on every job. This is in ocurrent/opam-repo-ci#483, also cherry-picked onto live. Before deploying, I traced the cluster cache key ({pool, commit, variant, ty} so the spec text and base image aren’t hashed) to confirm the change doesn’t rebuild everything.
Unrelated to opam, I also merged and deployed ocurrent/opam-repo-ci#481 (reject CRLF line endings in opam files, with Shon’s In_channel.with_open_bin suggestion).
Tessera
Tessera v2 has a family of distilled “student” models (nano/small/medium/large, 1.07M–43.83M parameters). The headline is speed: running on MI325X, v1.1 median inference of 101 s/cell drops to ~21 s for the large student. So roughly 4–6 times (20.7 vs 83.1 s/megapixel normalised for tile mix). v1.1’s cost was the sequential per-timestep GRU attention-pool; the student replaces it with a cheap per-pixel forward.
I split my monolithic worker Python file into a common dpixel.py (ROI -> download S1+S2 -> preprocessed cube), which could be imported by either v1.1 or v2. Obviously, any changes, even a simple refactor, require extensive testing to ensure byte-for-byte compatibility. I also compared the CPU/AMX bf16 path and validated at cosine 0.99994 vs fp32.
For the v2 processing, I’m uploading to Scaleway as the primary and source.coop as the secondary. The first tile showed that source.coop rejects single PUTs over ~100 MB via the Cloudflare proxy, where the same ~118 MB npy file works fine on both Ceph and AWS. I switched the upload to a multipart transfer (64 MB parts), so each destination gets chunks from a single code path.
Pedro requested v2 tiles for the global weather stations: ~80K tiles. This showed a ~2.6% error rate. Sampling the most recent 1000 errors showed the breakdown of failures: HTTP 520 for 782 PUT requests, HTTP 522 for 201 PUT requests and HTTP 502/524/525 taking the slack. These are transient, but they exceed my generous backoff and retry window, so I’ve had to make source.coop an optional target and will deal with the sync later!
Azure and MPC
With Azure credits coming to an end, I’ve substantially reduced my burn rate. However, before I removed the 350 D-series machines, I copied the logs back to my machine. This was perfect timing as I was able to provide feedback on statistics and typical errors to the Microsoft Planetary Computer team, along with my mitigations. Headline numbers for perspective: 1,774,656 v1.1 tiles plus 100k+ v2, ~800 blob reads per tile (~1.5 billion reads), exactly two STAC queries per tile, S2 a median 68 scenes per tile and S1 a median 29, peaking at ~1,750 concurrent jobs across 350 Azure VMs and 4 Vultr boxes at ~3,000 tiles/hour.
ucam-eo/geotessera/issues
The genesis backfill kept closing coverage requests. Brazil is complete 2017–2025, which let a batch of geotessera issues close (#317, #323, #325), the last needing two re-submission passes to fill gaps left by cells that didn’t process. This allowed me to make the PCA for South America (see header image).
Sentinel-2 banding, and a calibrated NBAR correction
The global-basis PCA render of the 2025 Amazon embeddings shows faint diagonal banding, and the question was whether the band angle matched the Sentinel-2 orbit. It does. ~275 km spacing, ~12.5° west-of-south at the equator. This places the artefact in the S2 input, not the mosaic or the PCA, and means it’s a view-angle (BRDF) effect as Clement suggested.
Iterating on this at native 10 m was hopeless (a South America reprocess is months of GPU), so the real enabler was a cheap local reproduction: one pixel per 2 km on the mosaic’s own UTM grid, sampling every raw scene’s bands only at the band’s cell points via COG overviews, collected once on an Azure VM co-located with the data and then reused locally through the real v1.1 encoder on CPU. That gives a ~15-minute edit-and-re-render loop, validated byte-for-byte against the published S3 embeddings at cosine 0.9998–0.9999.
The banding is created by overlapping S2 orbits, which means that on the edges of the orbits there is an overlap giving twice as many observations (which is fine), but these observations are taken from the opposite extreme of the view angle. Some early results look promising. See before/after image below.
Source Cooperative
The v1.1 mirror to Source Cooperative, copying out of the Ceph RGW store in Paris to the source.coop bucket, whose only writable entry point is a Cloudflare proxy, continues to grind along. A bespoke boto3 copier was needed to correctly handle the metadata signing and the mismatch in single-part/multi-part upload requirements. This hit a GIL ceiling at one process (16 workers, ~124 MB/s, CPU 85% idle); running one process per year nearly doubled it to ~230 MB/s, and a third gave only +5%. Interestingly, the limitation is the Cloudflare proxy, which throws random 5xx errors when you try to write too quickly.
Reviewing the dClimate Zarr pipeline
Separately, I reviewed dClimate’s tessera-embeddings pipeline (PR #82). This is a Python/Ray system that ingests Sentinel data into chunked Zarr and assembles a global GeoZarr, which is a different codebase from my worker pipeline. The review surfaced one thing worth flagging to the authors: fill_zone_year_flow wires on_cancellation but not on_crashed, so a Prefect Crashed state (spot reclaim, OOM, SIGKILL) would skip teardown and leak the whole billed GPU fleet — the exact leak the design sets out to prevent.
Ceph
The Scaleway RGW cluster’s three-week buckets.data resize reached structural completion: on 22 July pg_num and pgp_num both hit 512, so all splitting and all placement-release is done, and only the misplaced tail is draining. Then on 25 July, the cluster went to HEALTH_ERR from a failing disk, osd.57, which had racked up “11 reads repaired”. EC 10+2 coding silently reconstructs the data on read. Replacing osd.57 is the pending decision. There is a tremendous load with the source.coop mirror reading at 200MB/s, the workers writing at 100MB/s and recovery running at 50MB/s. 10.344% of blocks are still misplaced.