Correcting last week’s opam-repo-ci overlay, running day10 alongside OBuilder on every job, and fixing the Windows sparse-file error at its source. Plus Tessera tiles dispatched in Zarr order, and a C11 compiler written by Claude Fable that builds OCaml.
Overlaying a PR is wrong #
Last week, I described passing two --opam-repository sources to day10 for opam-repo-ci, the PR head first and master second, and relying on first-source-wins to get merge semantics. That was wrong. opam-repo-ci analyses a PR to list the changed packages, but a PR commit SHA isn’t just the changed packages; it’s actually a whole point-in-time snapshot of the opam-repository.
Repo.create indexes on name.version and keeps the first source that offers a package. Both sources are complete opam-repository trees, not diffs. So the head does not just contribute its own changes, it shadows master for every package it contains, which is nearly all of them. For a PR branched from an old master, the overlay quietly reverts the whole repository to the branch point.
I caught this on a real job where the PR head carried 32,755 packages, compared to the current master’s 18,886. The overlay had silently undone a prune of nearly fourteen thousand packages.
The replacement is a real three-way merge, done entirely in the object database. Context.merge_tree runs git merge-tree in the worker’s existing mirror and returns the merged tree OID, which day10 reads with git archive. There is no worktree and no 30,000-file checkout; it takes about a second and produces at most a few tiny objects. It is correct for additions, modifications and deletions alike, and conflicts cannot arise because opam-repo-ci rejects unmergeable PRs during analysis. The single-commit case that ocaml-ci and opam-health-check use is unchanged.
I updated day10’s --opam-repository path:rev parameter, which now peels to a tree rather than (only) a commit, so a bare tree OID resolves, as do branches, tags and commits.
Parallel builds #
With that in place, opam-repo-ci started running riscv64 day10 builds in addition to the OBuilder jobs. It’s the same test submitted to both.
| verdict | tasks |
|---|---|
| agree, pass | 133 |
| agree, fail or no solution | 101 |
| day10 pass, OBuilder infrastructure failure or cancelled | 9 |
| day10 pass, OBuilder real test failure | 7 |
| compiler pin no-solutions | 20 |
| hollow success | 1 |
| cancelled while queued | 2 |
On tasks where both produced a verdict, day10’s median time was under a minute against OBuilder’s twenty-five, and it returned answers for binaryen and apron, which OBuilder never finished at all.
The two blocks of disagreement are both real issues. The seven were a day10 bug, below. The twenty are an opam-repo-ci gap: the OBuilder path appends --update-invariant when the package under test is ocaml-compiler, ocaml-base-compiler or ocaml-variants, so the PR’s compiler can replace the one in the switch. The day10 path has no equivalent yet, so every task in a compiler PR comes back as no solution in about ten seconds. There is a second gap of the same shape: OBuilder honours x-ci-accept-failures and turns an expected riscv64 failure into a skip, and day10 has nothing for that either yet.
Reporting caught up during the week too. day10’s no-solutions first showed in the web UI as a plain failure, because the classifier saw an [ERROR] failure marker and stopped there, while the log analysis underneath was correctly calling it “Package not available”. They now read identically in both columns. Thanks to Jan for spotting this issue#3.
–with-test was never passed #
The seven tasks where day10 passed a test that OBuilder failed all had the same cause. config.package is name.version for health-check and a bare name for ci. All three backends compared it against OpamPackage.name_to_string, which is only ever the name, so for health-check the comparison could never be true and --with-test was never passed to opam-build. The run-test commands were silently skipped.
What made it look as though it worked is that test dependencies were still installed, because the solver correctly included the --with-test dependencies. So the switch looked right, the build looked right, and nothing ran the tests. OBuilder ran them and found that cstruct’s suites link against crowbar without declaring it.
FreeBSD and Windows had the mirror-image bug: they passed --with-test without any target check, so a dependency’s tests would run as well.
A week of day10 fixes #
Packages that only exist as avoid-version #
A package whose every available version carries avoid-version was dropped from the candidate list, so the solver reported no known implementations and the job was classed as having no solution. That made eight packages permanently uninstallable, oxcaml among them, along with anything first released as a flagged prerelease. opam does not refuse them: the flag is a soft CUDF property minimised at the second level of the default criteria, so opam prefers a solution without such a package and installs it when there is no other way. Candidate ordering cannot express that, which is opam-0install-cudf issue 5 and why the filter existed. So day10 now solves twice, once without them and, if that fails, again with them.
Every package’s files/ was dropped #
git archive emits tree entries in bytewise name order, and “files” sorts before “opam”, so a package’s extra files were always read before its opam file. The code could only attach extras to an entry that already existed, so it found nothing and discarded them, for every git-backed repository. Building from a plain directory was unaffected. Any package with a patch would have failed there. Extra “files” are now rejected by the maintainers and the lint checks, but the code needs to be right for historic packages.
The dpkg database #
Layers are merged by keeping the first copy of any path, so only one layer’s /var/lib/dpkg/status survived and depexts installed by the others looked uninstalled. 149 of 3,807 cached layer logs carried an opam warning that libgmp-dev could no longer be found. Harmless, since the files themselves merge and nothing in a build asks dpkg, but it appeared in every job. The merge now writes the union of the layers’ status files, which is what the Windows backend already did for cygwin.
Unavailable depexts #
opam-build now checks OpamSwitchState.depexts_unavailable before installing depexts and exits non-zero with opam’s own wording, including the distribution-specific repository hint, rather than starting a build that cannot succeed.
Obuilder HCS backend #
Windows container shim #
ctr task delete --force hangs forever, and I finally know why! An orphaned containerd-shim-runhcs-v1 outlives its container: the task shows as STOPPED against a dead pid while the shim is still running. Kill the shim, and the delete completes instantly.
ctr retries #
Some ctr operations fail with a pipe-busy error under load, which day10 works around by submitting them again. I copied over that retry idea to obuilder. With it in place on ltsc2025-1, orphans went from 37 to 0, already-exists errors from 486 to 0, with 27 retries fired and no divergence.
Fixing the sparse files at source #
Two weeks ago, I traced one of the Windows worker failures to ctr: failed to reimport snapshot: ... unknown stream ID 9, which is BACKUP_SPARSE_BLOCK: a sparse file left in Windows Update’s download cache breaks the layer export.
The windows, windows/server and windows/servercore base images set the Automatic Updates policy to “auto download and notify for install”. So the Update Orchestrator wakes Windows Update inside the container and downloads patches into C:\Windows\SoftwareDistribution, and those downloads are sparse files. ocurrent/ocaml-dockerfile#272 sets NoAutoUpdate and stops wuauserv, UsoSvc and WaaSMedicSvc from starting in every stage. Nothing writes a sparse file there any more.
That went out as ocaml-dockerfile 8.4.3, through ocaml/opam-repository#30635 and a pin bump in ocurrent/docker-base-images#359.
sage redeployed for docs testing #
ocaml-docs-ci has been live on dill for some weeks now. The previous machine sage has been wiped and rebuilt as the new testing instance using Ubuntu’s autoinstallation over PXE.
This gave me an excuse to merge ocurrent/ocurrent-deployer#266, which adds a Docker Compose deployment path. It embeds a current_compose_v2 plugin that runs docker compose up for a deployment, reading docker-compose.yml from the build source and pinning built images to their digests.
sage and dill both use this deployment path from the branches with the same name. This PR allows ocurrent-deployer to actually deploy things; previously, it could only update them. The benefit is that each repository takes ownership of this configuration rather than needing a central Ansible configuration.
tessera-grid #
tessera-tiles is now a stdlib-only tessera-grid library plus a CLI. mtelvers/genesis depends on the library rather than carrying its own copy, which means both tools now enumerate identical cells for the same ROI. mtelvers/genesis had been using a centre-or-corner-or-vertex rule, and the library uses shared area, so they disagreed a tiny amount at the edges.
The primary reason for the change was that Anil asked for the tiles to be processed in the ‘Zarr’ order rather than randomly. geotessera’s Zarr shard fill order is UTM zone, then 40.96 km shard-row bands north to south, then shard-column bands west to east, then grid rows. I use tessera-tiles to quickly add tiles to genesis’ priority queue so both tools needed to use the same ‘Zarr’ ordering.
The only wrinkle was that the random ordering meant that after restarting genesis it was unlikely to issue exactly the same tiles to be dispatched again, but with the deterministic ordering, a collision was guaranteed. Therefore, I added a random per-ROI offset at startup.
geotessera/issues #
I have been working through the geotessera/issues for the v2 requests closing: #334, #342, #354, #365, #366, #373, #376, #392, #398, #399, #400, #401. I’ve been processing in size order, smallest to largest.
One interesting thing to come out of this is that requests for bounding boxes need to be inspected closely, as often these pull in substantial areas of ocean, sometimes as much as 40% of the requested area.
A user reported being able to download 1 of their 400 tiles. The tiles existed on source.coop, but the manifest had not been updated.
OCaml C Compiler, occ #
It was an idle question: what is the minimum C a compiler has to support in order to build OCaml? The answer is that OCaml’s runtime needs C11, with a short list of extensions, so a compiler sized by that one target is an achievable thing to write and a project with a set goal. I challenged Claude Fable to write it. It took around 6 hours, resulting in mtelvers/occ, which is about 6,700 lines of OCaml plus 290 lines of headers, with no dependency outside the standard library.
make world.opt && make tests on OCaml 5.6.0+dev was achieved. The test suite reports 1,679 tests considered, 1 failed, and 57 skipped. The single failure is native-debugger, which compares gdb backtraces against a reference recorded with gcc at -O2: gcc inlines caml_startup_common, and gdb reports the inlined call site as an extra frame. occ emits DWARF, so gdb gets file, line and parameter information, but occ does not inline. Bytecode produced by the occ-built compiler is byte-identical to the gcc-built compiler’s.
The interpreter it produces runs about 2.5 times slower than gcc’s -O2 build, which I thought was a pretty good starting point.
Obviously, as a bootstrap compiler, this introduces the chicken-and-egg problem, but it might have a small use in situations where one might want to audit a smaller subset of code than gcc.
Anil pointed out that there is a large corpus of C compiler code available, so the model was probably well-trained, but I was impressed by how far AI has come. It raises all the usual questions about whether you can trust the code, here perhaps more so, as you’d not only need to question the compiler itself but also every binary that it produces.
I could probably have written occ myself. It would have taken a year, it would have had a lot more bugs, and in that year, I would have been forced to understand a great deal that I now do not. What I have instead is a repository with a compiler in it. I’m glad it exists, and I’m aware of what I didn’t learn getting it.
I’m less sympathetic to the industry’s distrust of AI-generated code, because it draws the line in a place I don’t recognise. There is a separation between ideas and implementation, and the value has always sat with the ideas. Nobody credits gcc with the program it emits: the source is the idea, the binary is mechanics. An AI system does the same thing one level up, where the prompt is the idea and the source is the output.
The analogy is not exact. gcc comes with a contract that the binary does what the source says, and when it doesn’t, we call it a miscompilation and fix it. Nobody offers that contract between a prompt and the code it produces. Here, I didn’t need one because the target supplied it. occ is trustworthy for exactly one claim: that it builds OCaml 5.6.0+dev and passes the test suite. That claim was settled by a machine that had no opinion on who wrote the code. I wouldn’t claim anything further because that was the test coverage, regardless of who or what typed it.