Git 2.55 broke opam repo add for all the 4.x base images, and ocaml.org was under scraper attack. On the Tessera side, the v1 dataset got converted to Zarr via Fargate, and there was further investigation into the banding correction.
Git 2.55 breaks opam repo add
The Docker base image builder has recently been reporting build failures across several distributions:
[ERROR] Could not update repository "ocaml-patches-overlay": Failed to regenerate
local repository archive: "/usr/bin/tar cfz ..." exited with code 1
"/usr/bin/tar: ocaml-patches-overlay/.git/objects: file changed as we read it"
Pulling each affected image and correlating with its git version showed the split was purely by git version and had nothing to do with the distribution: openSUSE 16.0 (git 2.51) works, openSUSE Tumbleweed, Fedora 44 and Debian unstable (git 2.55) fail, Debian 11 (git 2.30.2) works.
The git release notes for v2.55.0 report:
“git maintenance” that goes background did not use the lockfile to prevent multiple maintenance processes from running at the same time, which has been corrected.
Running git init and git fetch under GIT_TRACE=1 on both versions showed that git maintenance run --auto --quiet --detach was spawned on both. What 2.55 added is a maintenance.lock file, created and removed inside .git/objects. That changes the directory’s mtime while opam’s tar cfz is reading it, and tar names the lock file explicitly in its complaint. Three fetch-and-tar cycles per image: three failures out of three on 2.55, three passes out of three on 2.51.
I added git config --global gc.autoDetach false to Linux.Git.init, which every distribution family’s base image already calls, plus the Cygwin equivalent for Windows. Two deliberate choices: gc.autoDetach rather than maintenance.autoDetach because the latter falls back to the former anyway and only the former is understood by older gits, which I verified on Debian 11’s 2.30.2. And gc.autoDetach false rather than gc.auto 0 because it keeps repacking, so image layers stay packed and small, and only removes the race by making maintenance synchronous. That is ocurrent/ocaml-dockerfile#270, and I tagged the opam issue as well: ocaml/opam#7031.
Distro maintenance, and a release
While I was in ocaml-dockerfile, I did a sweep for new OS releases, querying the registries and then pulling images to read /etc/os-release rather than trusting a tag name. Three tags looked new, and two were pre-release: Fedora 45 identifies itself as a Container Image Prerelease on rawhide, and Ubuntu 26.10 as a development branch. Only Alpine 3.24.1 is a real release. The table values were read out of the image, with the method calibrated against the existing 3.23 row first: two of the three had moved: builtin_ocaml_of_distro to 4.14.3 and bubblewrap_version to 0.11.2.
Separately, an s390x worker reported no matching manifest for linux/s390x on debian:12. Checking the full manifest set showed only debian:12 has dropped s390x support. Debian 13, testing and unstable still publish s390x, as does every Ubuntu from 18.04 to 26.04, and ppc64le on Debian 12 is unaffected. So this is a one-line surgical change, not a general s390x retreat. Both changes went out together as ocurrent/ocaml-dockerfile#271.
Those three fixes, plus samoht’s ocurrent/ocaml-dockerfile#269 where a tmpfs mount was rendering as type=bind, became ocaml-dockerfile 8.4.2. The release went in as ocaml/opam-repository#30451.
Getting that into the base images is ocurrent/docker-base-images#358, which just needs a new opam-repository SHA in the Dockerfile and builds.expected to be regenerated. Checking the diff showed: 52 alpine-3.24 FROM lines replacing 3.23, 15 removed Debian 12 s390x lines, and 108 gc.autoDetach additions. I have deliberately not pushed it to live, because that would rebuild every image; it can ride the natural Saturday rebuild cycle instead.
ocaml.org outage
On Wednesday, www.ocaml.org failed twice in fifteen minutes with Varnish errors, recovering only on reboot. Thankfully, these were pretty short outages.
The traffic was the trigger. Roughly 1,180 distinct source IP addresses per minute at about 20 requests a second, so close to one request per IP. Forty-five per cent claimed to be Chrome on macOS, with YisouSpider and Lightpanda making up another quarter. Eighty-seven per cent of it was hitting the /p/ and /u/ package-documentation routes.
Each of those routes makes ocaml.org fetch per-page JSON from dill, and dill was busy processing an updated Debian distribution. It accepted TCP on port 443 but served nothing, and was often too loaded for SSHD to even complete a banner exchange. Every docs request then sat on the full 60-second first_byte_timeout, pinning a Dream handler and a Varnish thread for a minute at a time. Varnish reached 1,455 threads with 531 queued sessions, both backends failed the 2-second health probe and were marked Sick, and with no healthy backend, Varnish returned 503 for everything, including the homepage. At peak, 74% of requests were failing.
Based on the other problems we’ve seen recently, my guess was file descriptor exhaustion. The containers had the classic 1024 soft NOFILE limit despite the Docker daemon running with LimitNOFILE=infinity. At roughly two descriptors per in-flight docs request and 20 requests a second, 1024 is reached after about 25 seconds of dill being slow, after which accept() starts failing while the kernel is still completing handshakes from the listen backlog. With dill healthy, descriptor usage sits at 12 to 26, so nothing was leaking, but when it’s slow to respond, the file descriptors were backing up. Increasing the soft limit to 65,536 and the hard limit to 524,288 seemed to be enough to allow the site to run consistently.
After the change, both backends went healthy, the homepage served in 0.11 s and a docs page in 0.66 s, and a 30-second sample showed zero 503s with descriptor usage at 24 to 44 out of 65,536.
A login for the health-check logs
Last week, I built an optional GitHub login in front of check.ci.ocaml.org’s /log/ pages, as the proper answer to the scraper traffic that a blunt rate cap had only been holding off. This week it went live.
The one piece that needed to be done by hand was creating the OAuth application, since GitHub has no API for that. I deployed the new image first, because without the config block it simply behaves as before, then added the github-oauth config and restarted the service. Anonymous requests to a log now get a 302 into GitHub’s authorisation flow, carrying the exact return path, while the dashboard, diffs, JSON API and metrics all stay public. Any GitHub account is accepted, not an access list.
I also opened two PRs to get this upstream, ocurrent/opam-health-check#110 for the login and ocurrent/opam-health-check#111 for the docker_hub 0.2.0 adaptation, which fixes the “could not get digest” failure on recent images whose manifest lists carry “unknown” attestation entries. The second one replaces Kate’s draft ocurrent/opam-health-check#39, which I noticed after I’d done mine!
OCluster/day10
Two weeks ago, I added day10 as a job type to OCluster with only the health-check verb implemented. This week I added build, which is the one that matters for porting ocurrent/ocaml-ci, because an ocaml-ci job is a project build rather than a package check.
The interesting difference is that a build job has a source repo. OCluster already checks out the job’s repository and commits into src for custom jobs, so the dispatch only had to pass that as the positional directory along with the dune arguments. Testing shows that OCluster creates the checkout as root, day10 builds as opam with uid 1000, and dune build wants to write _build into that directory, so every build died on a permission error. Making the checkout group-writable fixes it, and it is safe under concurrent access because src comes from a per-job temporary directory that is deleted when the job’s callback returns, so two builds never share the same one.
Validated end-to-end on doris against ocaml/ocaml-re picked randomly. Running @install @check @runtest surfaced genuine project failures, missing memtrace for the benchmarks and js_of_ocaml for the expect tests, which is good evidence the verb reports real results rather than swallowing them; @install alone came back with “day10 build succeeded”.
The ocaml-ci port itself is in progress and uncommitted.
OCluster README
Thomas noticed that the example command line given in the OCluster README doesn’t actually work and dies with an uncaught exception. ocurrent/ocluster#263. --obuilder-store had silently become mandatory even though a store-less worker is a supported configuration that simply rejects OBuilder jobs. ocurrent/ocluster#264 parses the store into an option and turns the invalid combinations into ordinary command-line errors instead of uncaught exceptions.
At the same time, I opened ocurrent/obuilder#218, which fixes the store lists in the error messages, which were missing overlayfs: in one and docker: in the other, despite both being accepted.
logpack redesign
The log archive format I prototyped last week for opam-health-check had three problems: the global dictionary cannot be pruned, the compression only arrives after a compaction cycle, and random access was not actually fast. Last week, I decided not to pursue it, but I did find some time to have another quick look.
Rather than using LZW-style compression, I decided to treat the logs more like a video codec.
A logpack archive is a directory of self-contained epoch files: the first run in an epoch is the “keyframe”. Its files are sorted by path so that similar logs sit next to each other, concatenated, and cut into ~32 MB blocks that are each compressed independently with zstd, letting the compressor exploit the huge amount of boilerplate shared between files in the same run.
Every later run in the epoch stores each file as a tiny patch (from zstd --patch-from) against the same-named file in the keyframe. It’s always the keyframe directly, never the previous run, so reading any log ever only costs decompressing part of one block plus applying one patch. Files identical to their keyframe copy cost nothing, brand-new files are packed into small blocks of their own, and when a run’s patches grow too large, indicating that the logs have genuinely changed, then that run becomes the keyframe of a fresh epoch.
As an example, 5.2/good/irmin.3.11.0 across two runs with different opam-repo commits. Textually, 20 lines changed in a 241 KB file: the commit hash (twice), and a ppx_deriving version bump 6.1.2 to 6.1.3, rippling through the solver output and build lines. The entire patch is 108 bytes of an ordinary zstd frame.
On a selection of fourteen runs spanning October to July and five different opam-repository commits, 73.3 GB compresses to 219.9 MB, which is 333x, in fifteen minutes. Any single log opens in under 70 ms, and verifying all 185,897 members takes 13.5 s.
The 32-bit compiler and autoconf 2.72
The nightly rebase of my 32-bit fork of OCaml failed on the 5th after eight seconds. Upstream had merged the autoconf 2.72 regeneration the day before, which rewrites configure from else $as_nop into else case e in #( e), and that conflicted on the very first commit of my stack, the revert that restores the arm and i386 backends. Keeping upstream’s new syntax and re-applying the i386 entry was all it needed; all seventeen commits then replayed cleanly and the following night was a success.
Sentinel-2 banding: what the encoder actually wants
This is where most of my thinking went. Last week ended with the position that the correction works, but we do not know how much to apply or when.
Measuring the improvement
Every number measured so far has indicated the size of the artefact, either in reflectance or in a rendered image. What actually matters is whether the delivered embeddings are better.
My approach here was to take pairs of forest pixels that are statistically identical, same dense rainforest, same distance apart, but arrange for one set of pairs to straddle a Sentinel-2 swath boundary and the other to sit entirely within one swath. If the embeddings were free of orbital artefact, both sets would be equally far apart in embedding space. Any extra distance across the boundary is the artefact.
The boundary itself is fitted from the uncorrected embeddings alone, and it converges to a line tilted 12.7 degrees from north, which matches Sentinel-2’s descending heading. Running the same design against a fake boundary placed well inside one swath, where every pixel is pure single-orbit, gives a small negative bias about five times smaller than the real signal, so the measurement is not an artefact of the design.
The answer for v1.1 is that crossing a swath boundary adds 17 to 21 per cent to the distance between statistically identical forest pixels, and that the tropical-forest correction (Ploton et al) removes 81 to 91 per cent of that excess. That is the first evidence that correcting improves the deliverable rather than just the reflectance.
A weight field harvested from the imagery
The correction has one adjustment per band. At the low end, there is the generic MODIS anisotropy with the strong correction of Ploton’s tropical forest at the other end.
The right value varies by land cover, so rather than try to predict it, I have measured it everywhere. Essentially, fly every orbit, find pixels which are imaged on close runs from different perspectives and generate a world correction map. Interpolate between the measurements to create a continuous field, as any instant change would create new artefacts.
e.g. Switching from tropical forest correction to MODIS introduces a step 30 per cent as large as the banding it removes, and geographically aligned with it, which is the worst possible kind of new artefact.
The harvest covers 2,288 orbit and latitude-band segments, of which 1,103 have land and 1,009 yielded fits, at about 19 hours and 83 GB of cached granules. The resulting field is 9 MB at half-degree resolution, and the largest weight step anywhere on the globe works out to a per-pixel change of around four ten-thousandths of a per cent, so the new-visible-line hazard is structurally impossible rather than merely unlikely.
v2
Then I ran the same ROI through the v2 encoder, both natively and with the field correction applied, and re-ran the same test.
| run | boundary excess |
|---|---|
| uncorrected v1.1 | 20.6 / 19.0 / 16.8% |
| v2 native | 13.1 / 11.5 / 10.2% |
| v2 with the global field | 25.3 / 25.2 / 23.0% |
Out of the box, v2 has about half the artefact of v1.1, so the new encoder has learned a good deal of view-angle invariance on its own with no input correction at all. And the field correction more than doubles v2’s boundary excess and raises within-swath distances as well, so it is injecting swath-correlated structure rather than removing it.
I think v1.1 passed reflectance artefacts through to its embeddings, so cleaning the input cleaned the output, and the strongest cleaning won. v2 has already learned to ignore the natural view-angle signature. Feeding it reflectance with that signature removed is a double correction: the input no longer resembles anything v2 was trained on.
Apply MODIS
I ran a third set with the gentle MODIS correction on v2. This has previously helped v1.1 but didn’t move it far enough.
| v2 variant | boundary excess | vs native |
|---|---|---|
| native | 13.1 / 11.5 / 10.2% | |
| with MODIS | 8.7 / 7.4 / 6.5% | -33 to -37% |
| with the global field | 25.3 / 25.2 / 23.0% | +103 to +136% |
From the numbers, the gentle MODIS correction reduces the boundary while the global field magnifies it.
Where that leaves the field
The field is a bust, which is a shame because on paper it is the right idea.
Running it against v1.1 over French Guiana, where the field sits at Ploton’s values to within 0.13 in every band, it removes only 10 to 19 per cent of the boundary excess, where exact Ploton removes 81 to 91 per cent. That is despite the two being nearly indistinguishable at reflectance level, 57 per cent against 66 per cent of the cross-swath difference removed.
My theory was that the blue and green weights were to blame, so I rebuilt the field with those tied to red, which put every band within 0.13 of Ploton. The rerun left a boundary excess of 17.5/17.1/13.6 per cent, against 17 to 21 per cent uncorrected, so it had barely moved. The embeddings themselves did change, by the predicted amount and in the predicted direction, but the rotation away from tropical only went from 18 degrees to 16. Blue was not the cause.
So the finding is that under v1.1, removing the artefact from the embeddings requires per-band exactness well below 0.1 in the weight. That is not something refitting can fix, because it is the harvest estimator’s own noise floor meeting the encoder’s sensitivity.
The most interesting part is why the corrected mosaic still looks good, which was what made me suspicious in the first place.
A PCA of the embeddings produces all 128 components, but the mosaic only renders the first three, as red, green and blue. Splitting the boundary excess into those three against the remaining 125:
| dataset | full 128-d | top-3 PCs | other 125 |
|---|---|---|---|
| uncorrected | 1.087 | 1.484 | 0.006 |
| MODIS | 0.509 | 0.655 | 0.079 |
| tropical forest | 0.228 | -0.000 | 0.297 |
| field r2 | 0.972 | 0.241 | 1.040 |
The original artefact lives almost entirely in those first three components, which is exactly why it is so visible in a mosaic. The field removes 84 per cent of it there, so the render honestly does look nearly as clean as Ploton’s, and side by side, I would struggle to pick the working correction out of the two above. But it writes a new artefact into the low-variance dimensions, which were pristine before. Anyone looking at pictures sees a success; anyone using the full vector, which is what the embeddings are actually for, sees something close to uncorrected.
What upsets those sensitive dimensions is not how strong the correction is, but whether it treats the ten bands consistently. MODIS and Ploton are each a single physical model, so all ten bands get adjusted by one coherent recipe. Get the amplitude wrong, as MODIS does by under-correcting everywhere, and what is left over still looks like a weaker version of the same artefact, so it stays in the few dimensions the artefact already occupied, and the clean ones stay clean. The field fits each band separately, so its small per-band errors have no shared shape. That produces a slightly wrong mix of colours rather than a slightly wrong amount of correction, and a wrong colour mix is precisely what those low-variance dimensions are there to notice.
This gives a straightforward answer for now. For v1.1 over dense tropical forest, apply Ploton’s tropical forest coefficients. For v2, apply MODIS. The field’s proper role is as training input for the next encoder, where learned invariance rather than per-band exactness would absorb the residual.
Chad on Slurm
The Chad run, 10,711 tiles, needed two fixes to survive a week on a shared Slurm cluster.
The overnight stall on the first night was Slurm’s MaxArraySize of 1001: the top-up loop was submitting array indices above 1000, which is simply rejected. Now every chunk submits indices from one with an offset passed in the environment.
The bigger problem is that login nodes kill user processes when the session ends. There is no crontab, nohup and setsid do not survive, and the round-robin login nodes mean you cannot even rely on node-local state. The answer is to self-chain inside Slurm: the last array tasks of each chunk run a lock-guarded script that submits the next chunk, and sbatch works fine from a compute node. That makes the run genuinely autonomous, and it kept advancing through node changes, SSH key expiration and long allocation droughts with no manual submissions at all.
Chad is 10,711 tiles for 2025 only, finished in 6 days 23.5 hours using 2,293 GPU hours!
Embedding requests
Six issues closed this week: the Cambridge neotropics request, RTL, Trentino, a Bangladesh soil-pH request, GMV’s Paraguay years, and FSU Jena’s Great Limpopo Transfrontier Park, which came to 661 tiles across South Africa, Mozambique and Zimbabwe. Aneesh’s is complete on v2 and awaiting the close workflow.
Converting v1 from NPY to Zarr
The plan was to use AWS Fargate spot instances to convert the v1 NPY data to Zarr. Anil built a container to do the conversion, and I ran a small test on Fargate. Our expectation was that RAM would be limited so we would overflow to disk. However, the test showed that each worker had I/O wait between 70 and 97 per cent. The overflow ephemeral storage had a high burst rate but a pretty low sustained I/O rate.
Running in RAM, either by overflowing to /dev/shm or simply not configuring an overflow device, was roughly ten times faster.
Since the jobs didn’t scale particularly well (double the number of vCPU didn’t get you double the throughput), I decided to run with many small jobs rather than fewer large ones.
2024 is a bumper year with full coverage, unlike other years, and not all UTM zones have equal amounts of land. So rather than 60 jobs, one per UTM zone processing all years, I went with 540 independent jobs, each processing one year. I submitted all the 2024 jobs first, allowing the other years to start when capacity became available, so the large land area UTM zones from 2024 would run in parallel with the many other, much smaller chunks.
The Fargate Spot quota allowed only four concurrent tasks, so I filed for an increase to 640.
All 540 fills completed, then 60 zone closures, then a single consolidation pass writing 840 nodes. The fleet totals come to 715 jobs, 987 container runs, 108 Spot reclaims and about 18,795 vCPU-hours over 63 hours of wall clock, peaking at 42 concurrent containers.
Some of the conversion jobs failed with short reads, and then with “magic string is not correct”. These were not conversion bugs; they were genuinely corrupt source tiles, in two flavours: truncated files with a valid header but missing data, and files that are nothing but null bytes.
A full inventory across v1 is 130 truncated and 152 zero-filled, 282 objects out of about 8.8 million, which is 0.003 per cent. Of the zero-filled ones checked against the origin machine, 91 were good at source and have been re-copied, 39 are zero at origin too, and 20 are absent. 189 objects have been deleted from all three stores.
Getting three stores back in sync
With Scaleway Ceph as my authoritative store of v1.1 and v2 objects, a reconciliation against the other two showed source.coop 136,682 objects behind, about 10.65 TB.
The main pass copied 135,626 v1.1 objects server-side and 1,029 v2 objects streamed, with no failures. A residual 2,339 turned out not to be failures at all but new objects Ceph had gained during the hour the backfill was running, and a catch-up pass cleared those in 72 seconds. source.coop now holds 100 per cent of Ceph’s v1.1 and v2 data, with zero size mismatches anywhere and everything stamped with the newer checksum algorithm.
One tuning note, driving s3api copy-object under xargs -P 48 maxed out around 17 a second, bottlenecked purely on CLI process startup with the box only 35 per cent loaded. Raising parallelism to 128 doubled it!