day10 now runs side-by-side with OBuilder on the RISC-V opam-repo-ci workers, and the layer cache shows why it wins. Plus the Lwt bug that stopped Windows workers reconnecting to the scheduler, an LTSC 2022 image that hadn’t built in months, and ocaml.org taken down by a 371 MB search index.
day10 against OBuilder #
Two weeks ago, I benchmarked 36 repositories submitted to two identical workers, one running day10 and the other OBuilder. This showed that in a controlled experiment, day10 was 2.6 times faster. This week, day10 has been running identical jobs for opam-repo-ci on the RISC-V in parallel with the OBuilder workload. day10 has additionally been doing all the OCaml-CI traffic on RISC-V.
Side-by-side comparison #
After the backlog of historic opam-repo-ci jobs had cleared, I opened a PR ocaml/opam-repository#30699 to release ocurrent/ocaml-dockerfile. ocaml-dockerfile consists of 3 packages which would need to be tested on both OCaml 5.5.0 and 4.14.4. Both workers ran head-to-head on the same six builds.
Wall-clock seconds per task, with the layers day10 actually built. 284 seconds against 4,581.
These three packages have similar but not identical dependencies. OBuilder’s own install lists show it at 17, 19 and 26 packages with each a complete subset of the other. However, OBuilder isn’t able to take advantage of this and builds all three sets of dependencies from scratch on each build.
day10 pulled the majority of packages from its cache layers and built the remainder; however, as dockerfile is a subset of the others and was ordered last, the entire run came from the cache, completing in 9 seconds.
Where the speed comes from #
Every day10 log tallies its own closure: [NOTE] Using for a layer taken off disk, [NOTE] Building for one it had to make. Considering 523 jobs on machine carpenter, that is 18,611 hits against 2,397 layers added, and two views of the caches show where it comes from.
Layers in each cache over the first 48 hours. riscv-bm-04 starts 22 hours later from cold.
Each curve is a staircase of cold-cache bursts. A riser is a job meeting a dependency that the machine has not seen before; a tread is a job that finds everything already built.
Carpenter’s long tread is interesting: it built nothing at all for nearly 3 hours while twenty-nine OCaml-CI jobs were completed. All the layers were in the cache.
Jobs plotted at the moment they are received: height is that job’s hit rate, circle area is how many layers it had to resolve. The strip under each panel is the source of the job opam-repo-ci or OCaml-CI.
A job resolves its whole closure the moment it starts, hits are looked up and misses built. Large circles at the top 100% line indicate that jobs were entirely satisfied from cache layers. OCaml-CI jobs build a project on top of its dependencies and may legitimately hit cache on every layer, while an opam-repo-ci job exists to test a package that is new or changed, so by definition, it must create at least one new layer.
How many times did we build dune? #
Dune is the clearest case of layer reuse. Jokingly, I’ve said that our CI systems build dune about a million times a day. Across 320 opam-repo-ci jobs run on both builders, same PR heads and same compiler:
| builder | dune compiled |
|---|---|
| OBuilder | 88 |
| day10 | 1 |
- The other jobs either did not depend upon dune, or were unavialable on RISC-V.
The Lwt bug that cost us a cluster #
Back in week 34, I wrote about four Windows workers showing as disconnected at the scheduler with nothing actually down: guests running, services running, SSH fine, logs still scrolling. I have the cause now, and it is in Lwt.
Windows signals a failed asynchronous connect through the exception file descriptor set, not the write set. Lwt’s connect used wrap_syscall Write, which only ever waits for writability, so a refused or unroutable connect was never signalled at all. The promise remained pending forever, and the caller hung with no error or exception.
Connecting to a closed port on localhost, which is an immediate ECONNREFUSED on Unix, simply never returned:
let fd = Lwt_unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
Lwt.pick [
(Lwt_unix.connect fd (Unix.ADDR_INET (Unix.inet_addr_loopback, 9))
>|= fun () -> `Connected);
(Lwt_unix.sleep 20.0 >|= fun () -> `Timeout);
]
(* -> `Timeout on Windows, ECONNREFUSED on Unix *)
Successful connects were unaffected, because the socket does become writable, which is exactly why this only ever showed up on reconnection after a failure. A Windows worker that lost its scheduler logged Lost connection ... will retry once and then hung in connect forever, staying registered and looking perfectly healthy while doing nothing at all, whereas a Linux worker reconnected normally.
The fix drives completion explicitly, watching the write and exception sets together and reporting the outcome through getsockopt_error. select is reliable for sockets on Windows; it is pipe handles that it cannot handle. I’ve appended this to my Windows fixes PR ocsigen/lwt#1103.
The LTSC 2022 kernel mismatch #
The LTSC 2022 base image had not built successfully for a while. The job hangs at step 20 of 77, inside Cygwin’s setup-x86_64.exe, and then never fails. It waits. There is therefore no error message anywhere in the logs, which is what made it tricky to find as there was no error to read.
The cause is isolation, not Cygwin. The LTSC 2022 userland is build 20348 and the hosts are Server 2025, build 26100. Under process isolation, a Windows container shares the host kernel, and Cygwin’s fork emulation, its native symlinks and its shared memory are the most kernel-sensitive code in the whole image. I note that Microsoft does support running LTSC 2022 images on LTSC 2025 servers.
What made it findable was a differential rather than more log reading: the same Cygwin step succeeds for LTSC 2025, which matches the host kernel, and hangs for LTSC 2022, which does not. From there, a minimal reproduction failed in about five minutes, against ten and a half hours of silence from the real build.
The fix is one line of Docker configuration, defaulting to Hyper-V isolation, which is the recipe LTSC 2019 has always used as differing container versions are not supported without it. uname inside the container then reports CYGWIN_NT-10.0-20348, its own kernel rather than the host’s. The first successful image came out at 12.8 GB, all 77 steps in about five and a quarter hours, having never previously cleared step 20. Nothing changed in obuilder, ocluster or lwt, all three of which I had suspected at various points.
The WebView2 breakage #
In a separate Windows breakage specific to the MSVC images, ever since 17.14, Microsoft.VisualStudio.Workload.VCTools depends on Microsoft.VisualStudio.PackageGroup.CoreEditor, which requires Microsoft.WebView2.
That package runs the bundled Edge WebView2 Runtime offline installer, which first goes to the Internet to update Edge Update itself. The newer Edge Update then cannot find the offline payload it was supposed to install, failing with 0x80070003, and the whole Build Tools install falls over with 1603.
WebView2 is only used by the IDE, and there is no Edge in a Server Core container anyway, so the fix is to pre-seed the package’s detection condition in the registry. The installer considers it already present and skips it.
ocaml.org HTTP 503 #
ocaml.org started returning Varnish 503s, reported as ocaml/infrastructure#192.
The awso package, the AWS bindings, had its documentation published to dill at 03:20 that morning with a search index index.js of 371 MB, and a sidebar.json of 23 MB. For scale comparison, core is 11.5 MB, merlin-lib 3 MB and dune 177 KB.
Every documentation page render calls search_index_digest, which, on a cache miss, downloads the entire index into an OCaml string purely to derive a cache-busting hash. Fetching 371 MB from dill takes over a minute, so requests pile up and overlap. Both server replicas run with no memory limit, on a 15 GiB box with no swap, and both grew to between 10 and 12 GiB resident within about 90 seconds before the OOM killer took them. Twenty-three kills. The “recent start time” visible from outside was the crash loop, not a deploy.
I blocked /p/awso/* in Varnish, which stopped the crashing in a few minutes: memory fell from over 10 GiB to about 1.5, and the OOMs stopped. This makes the documentation for awso unavailable. Issues are now open on docs-ci and ocaml.org to address this longer term.
Both day10 gaps closed #
Last week’s shadow-build comparison left two known disagreements between the day10 and OBuilder paths. Both are now closed.
x-ci-accept-failures. When a package fails on a platform its maintainer has listed as tolerated, OBuilder reports [SKIP] Failure ignored. day10 marks the same thing with a [NOTE] accept_failures line, but leaves the genuine [ERROR] lines in the log as well, and opam-repo-ci’s generic [ERROR] (.+) catch-all matched those and overrode the skip, so accepted failures were still gating. A score-100 rule for the marker lets the skip win; day10 does the platform matching itself, so the marker being present already means the failure is accepted.
Compiler packages. The day10 path now sets --update-invariant for ocaml-variants, ocaml-base-compiler and ocaml-compiler targets, so the solver may change the compiler rather than pinning both it and the target and returning no solution. Lower bounds are threaded through the dispatch too, so those jobs run on day10 via --prefer-oldest instead of falling through to OBuilder.
Bounding the day10 cache #
The worker now has a day10 cache manager, modelled on the obuilder one. With --day10-prune-threshold set, a background loop watches free space on the cache partition, exported as a day10_space_free metric, and runs day10 prune when it drops below the threshold.
root@carpenter:~/day10# day10 cache-info --fork 4 --cache-dir /var/cache/day10/
[NOTE] Measuring 2202 layer(s) with no recorded size; later runs read the recorded ones
debian-13-riscv64 2481 layers 49.7G 63 failed
hours since last used size layers
0- 6 ################## 11.7G 804
6- 12 ########## 6.8G 380
12- 18 ### 2.1G 270
18- 24 ########## 6.8G 226
24- 30 ########### 7.6G 319
30- 36 ########### 7.3G 242
36- 42 ########## 6.9G 217
42- 48 # 501.0M 22
48- 54 # 5.5M 1
would free --days 90 0.0B 0 layers
--days 30 0.0B 0 layers
--percent 90 5.1G 142 layers
--percent 50 24.9G 919 layers
A blank log frame #
OCaml-CI’s step-log stream wrote the page header and the first chunk of log, but only flushed inside the streaming loop, after Current_rpc.Job.log returned more data. For a running job with nothing past the current offset, that call blocks in wait_for_log_data. So a short log, such as a queued job showing only its header, stayed in the buffer and was never sent, leaving a blank frame until the job produced more output. Large logs happened to overflow the socket buffer and flush themselves, which is why it had gone unnoticed.
OBuilder jobs give the reproduction Dockerfile immediately after the job is created causing the buffer to be flushed. day10’s shorter prologue highlighted this problem as the buffer was not flushed and it showed a blank log frame. The fix is in ocurrent/ocaml-ci#1071.
Retiring dl2.geotessera.org #
dl2.geotessera.org now serves a polite 410 pointing readers at geotessera 0.10.
OCaml 5.5.1 and opam 2.6 have been released! #
As with previous releases, OCaml 5.5.1 was added to ocurrent/ocaml-version and opam 2.6 was added to ocurrent/ocaml-dockerfile and allowed them to be combined into an update for ocurrent/docker-base-images. The base images were built over the weekend, as usual, to a fully green dashboard.
OCaml C Compiler, occ #
Last week, this was a C11 compiler in OCaml that could build the OCaml runtime. This week it is the rest of the toolchain as well. This was overwhelmingly an automated effort with little input from me. mtelvers/occ
There is now an assembler, occas, which reads the AT&T assembly that occ, ocamlopt and the runtime’s amd64.S produce. It is checked by demanding identical section bytes and relocations against GNU as, and it gets them on all 239 runtime units, on 281 ocamlopt-compiled compiler modules and on amd64.S. There is an archiver, occar, which rebuilds every static library in the OCaml tree from its members and gets the original file back byte-for-byte, 34 of 34. There is a linker, occld, producing static executables, shared objects and partial links, which the OCaml test suite pushed into implementing symbol versioning properly: 334 tests failed because an unversioned reference to realpath binds to glibc’s older one, which rejects a null second argument.
occ also builds musl 1.2.5 on its own, which needed real GNU inline assembly, variable length arrays, 80-bit long double on the x87 stack and a handful of linkage attributes. OCaml then builds against that musl, giving statically linked executables with no GNU component anywhere in the toolchain.
And there is a shell, occsh, a make, occmake, and the utilities the build calls in a single binary, occutils, including the whole of sed and of awk.
The result is the claim I find genuinely surprising. With a PATH containing only those programs, ./configure && make world.opt builds the OCaml compiler, and there is no program written in C anywhere on it.