day10 reverse dependencies are live and found a dune regression; the aardvark 2017 run finished; and the v2~beta1 Zarr store was rebuilt on a grid that no longer depends on its own data.
Revdeps on day10, turned on #
Last week I wrote that the opam-repo-ci side of the day10 reverse-dependency work was finished but not deployed. The workers had already been updated to support the revdeps command last week, so this week it should have been a simple case of pushing opam-repo-ci commits from origin/day10 and upstream/live.
However, the update in ocurrent.0.8.0 had raised its constraint to prometheus-app >= 2.0, meaning that current_web.dev did not compile, as CollectorRegistry.collect is now synchronous rather than returning a promise.
Prometheus.CollectorRegistry.(collect default) >>= fun data ->
Error: This expression has type ... snapshot ...
but an expression was expected of type 'a Lwt.t
My day10 version of opam-repo-ci has ocurrent pinned to proof-of-work PR ocurrent/ocurrent/pull/475 so that needed to be rebased.
How long a build takes #
In week 35, I ran a controlled experiment comparing build times across 36 builds on a pair of identical RISC-V workers. With revdeps on, I now have a large-scale test, particularly as there is the dune 3.25.0~alpha0 release opam-repository#30850 open. dune has more dependents than anything else in the repository.
| OBuilder | day10 |
|
|---|---|---|
| revdep builds | 47,356 | 49,253 |
| distinct packages | 14,021 | 14,491 |
| hardware | 11 workers, 328 slots | doris alone, 64 slots |
| wall clock | 20h 50m | 10h 31m |
| summed job time | 5,036 job-hours | 400 job-hours |
| median job | 242s | 2s |
The wall clock is the least fair number, as doris was also working through the revdeps for every other open PR at the same time. The summed job time gives the actual time spend much more clearly: 400 job-hours instead of 5,036.
As a second comparison, later in the week, the conf-perl change in #29613 has around 3,000 dependents per compiler, so a single pull request produces about 6,000 day10 jobs. In one hour, doris finished 11,695 of them.
183 differences #
Of course, speed is very much secondary to agreement with obuilder. Of the 47,312 debian-13 x86_64 twins, 43,568 passed on both, and 3,297 failed on both. That leaves 264 where OBuilder failed and day10 did not, and 183 the other way.
The 264 are almost all OBuilder infrastructure: 150 are Failed to get sources, a curl that did not complete, 72 are skips, and 13 were cancelled.
The 183 break down as follows:
| cause | jobs |
|---|---|
| tests run without the package installed | 88 |
| an optional dependency present at build time | 38 |
topiary and a missing share directory |
20 |
OPAMCONFIRMLEVEL in expected output |
6 |
post dependencies absent from the test layer |
6 |
a package reading %{env:PWD} |
4 |
| newer dependencies from the master merge | 5 |
| apt, and a crash | 3 |
| unexplained | 13 |
A dune alpha that stopped exporting OCAMLPATH #
The 88 all have the same issue: a test that steps outside _build and asks the switch for the package by name. For example, bogue runs ocamlfind query bogue; hxd’s cram test writes a dune project with (libraries hxd.string) and builds it; GT and ostap run mdx, whose top level does #require; and flint loads dllflint_stubs.so through lib/stublibs.
They fail under day10 and pass under OBuilder because of an ordering difference. day10-install --reinstall removes the old copy and then builds; opam’s own reinstall builds first, then removes, then installs. OBuilder’s job therefore still has the previous install in the switch while the tests run, and day10’s does not. You can see it in the logs: OBuilder prints compiled, removed, installed, and day10 prints installed, removed, then extracts and builds.
Outside of obuilder/day10 what happens in a plain container with just opam?
docker run --rm -it ocaml/opam:debian-13-ocaml-5.5
opam install dune.3.25.0~alpha0
opam install -t hxd.0.5.0 # Error: Library "hxd.string" not found
opam install hxd.0.5.0
opam reinstall -t hxd.0.5.0 # passes
While creating the reproduction steps, I initially used dune 3.24.2 and was unable to reproduce the problem. It was that insight which made me consider whether this was due to dune release. GT and bogue behave the same way.
Up to 3.24, dune set OCAMLPATH=<root>/_build/install/<context>/lib, from make_root_env in src/dune_rules/super_context.ml. In 3.25.0~alpha0, that is gone, replaced by a per-action layout that is only set up for actions declaring (deps (package ...)). One rule shows it:
(rule (alias runtest) (action (system "echo OCAMLPATH=${OCAMLPATH:-unset}")))
dune 3.24.2: OCAMLPATH=/tmp/e/_build/install/default/lib
dune 3.25.0~alpha0: OCAMLPATH=unset
So a fresh opam install -t breaks for eight packages, on their current releases as well as old ones: bogue on all thirteen versions, hxd on nine, plus flint, GT, ostap, embedded_ocaml_templates, ocaml-index and topup. Reported as ocaml/dune#16617.
opam switch create does not create share #
topiary depends on the Rust compiler, not the OCaml one, and its installation does a bare mkdir "%{share}%/topiary". In a switch with no compiler, there is no share directory, because opam switch create does not make one. The compiler build normally creates it as a side effect of writing share/ocaml/build-id, which is why this has never come up: under obuilder the compiler is part of the base image so is always there.
I opened ocaml/opam#7205 to get opam to create share (and etc, which was the only other one missing) when the switch is created. However, the CI went red, because the tests assert share being created and owned by the first package that needs it and reclaimed on removal. I read that as the test documenting deliberate design and withdrew the pull request. Kate’s reply was that the tests only show the state of things at a particular point, not necessarily the intent, and asked me to reopen it.
The other end is opam-repository#30938, which adds mkdir -p to every topiary version. At the moment, a package that needs the share directory should create it.
Other issues #
I documented these by creating issues on day10 so the history and reasoning is preserved.
mtelvers/day10#5. OPAMCONFIRMLEVEL=unsafe-yes was exported into the container, where nothing reads it: day10-install authorises the depext sudo itself and installs system packages through OpamSolution.install_depexts rather than the opam CLI. What it did reach was package tests. An opam asked for the 2.0 CLI prints a warning that the variable was ignored, and an expect-style test that captures opam’s output fails on the extra line. That accounted for dune-release and satyrographos.
mtelvers/day10#6. day10 health-check ocaml-option-llvm.1 on an OCaml 5.5.1 base reports [NOTE] success with a dependency graph containing one node, having built nothing. opam, given the same tree, refuses the install outright. It only reproduces against a tree with #29582 merged onto master; on plain master the package has no solution at all. This is currently still open.
mtelvers/day10#7. post dependencies are resolved when day10 solves and then never built. merlin’s test suite calls %{bin:ocaml-index}, which is a post dependency, and gets Program ocaml-index not found in the tree or in PATH. catala needs conf-ninja the same way. Shon opened ocurrent/opam-repo-ci#489 about exactly this the day revdeps went live, which is a fair thing to notice first. This is still open.
mtelvers/day10-install#1. imguiml’s src/dune reads %{env:PWD}, and day10-install execs build commands directly, so PWD is whatever it was at startup. I had assumed opam sets PWD, but it does not. It chdirs and passes the environment through, and plain opam install imguiml.v1.90.6 fails identically on 2.1.6 and 2.5.1. OBuilder only looks correct because bubblewrap’s sandbox.sh is a shell, and a shell resets PWD when it starts. The bug is the package’s, and %{project_root} is what it wants. Closed.
One more thing I have not created an issue for is that opam-repo-ci’s is_compiler_package recognises three names, ocaml-variants, ocaml-base-compiler and ocaml-compiler, so the ocaml-option-* and ocaml-options-* packages, which carry flags: compiler, never get --update-invariant and come out “not available” on every compiler variant. The list is duplicated in the OBuilder spec and in the day10 job builder.
Thousands of directories in /tmp #
Builds started failing early on with Unix.Unix_error(Unix.ENOSPC, "mkdir", "/tmp/opam-xxx-258365").
day10 didn’t create them, or did it? day10 binds to the opam library and read_command_output builds a stdout path of its own and passes it in, and opamProcess deliberately excludes a caller-supplied stdout from the files it unlinks afterwards, so the log of every such command stays. opam tries to rmdir the directory on exit, but it is never empty, so the rmdir fails and the directory stays.
day10 polls OpamSysPoll for the core count and the OS version before it does anything else, and because Os.default_jobs is a top-level binding, even day10 --help does it. That is two such commands and one abandoned directory per run, and turning revdeps on multiplied the number of runs by a large factor.
Commit 0c0e340 adds the directory as a cleanup resource so a signalled run drops it as well.
opam pin removed #
day10 had an opam.template whose only purpose was pinning six opam libraries to a fork, carrying one patch: copy_file preserving the source’s mtime, so a configure script does not come out newer than its configure.ac and send make looking for autotools that are not installed. See ocaml/opam#7196.
The pin is no longer needed, as all the work is done in day10-install; this allows day10 to move from 2.4.1 to 2.6.1. Commit f775e2a.
Tessera #
The aardvark run is finished #
mtelvers/ferry, which runs the aardvark tile pipeline, has got to the end of it. All the tiles have been built, encoded, shipped and uploaded, virtually all done on a single NVIDIA L4.
v2~beta1 zarr #
The last NPY store which needs to be regenerated is v2~beta1. I followed the playbook from v1.1: delete the data and refill it with Fargate spot instances. This appeared to work but there were several zone-years which didn’t exit cleanly. I’ve learnt my lesson, and do not allow Fargate to automatically resume failed jobs. The logs showed:
Stats catch-up for shard (189,7) failed: index 16 is out of bounds for axis 1 with size 16
shard_coords_for_tiles and build_shard_index walk range(col//SHARD, (col+w-1)//SHARD + 1) with no clamp to the grid. When a zone’s grid width is an exact multiple of the shard size, and utm33’s v2 grid is 65,536 pixels, which is exactly sixteen shards, a floor-placed tile on the eastern edge emits shard column 16 when the last valid one is 15. The write path already clips pixels, so the shard contains no data and nothing visible goes wrong, but the coordinate still reaches the statistics mask, causing an exception. That kills the fold mid-update, leaving the count at its -1 sentinel, and every later shard then reports that the statistics need rebuilding.
It is now clamped in commit 742d8ee, and v1 escaped only by geometry: its utm33 grid is 69,632 pixels, which is seventeen shards and not a knife edge.
A manifest a month out of date #
I ran an audit on v2~beta1 before I started and again after it finished. Unfortunately, this showed that the manifest.parquet, which is dated 11 September, only lists 305,184 tiles. The bucket holds 373,767. 68,583 tiles uploaded since then had never been converted.
Regenerating the manifest needed geotessera-registry s3scan, which would not run: its endpoint defaults to the read proxy, which cannot serve an anonymous ListObjectsV2. Pointing at the real S3 API, it returned 373,765 tiles.
The grid depended on its own data #
Fortunately, the v2~beta1 cost was less than $100, but it’s still annoying to need to rerun it again so soon.
However, there’s a further issue, the existing store can’t hold the new tiles and init_store sized each zone’s grid to the bounding box of the tiles that happened to exist when it ran, so new tiles that extend the coverage invalidate the grid and force a full regeneration. Plus, the preview pyramid is keyed off zone coordinates, an RGB regenerate is needed too. For example, zone 14’s old grid was northern hemisphere only; the new manifest has tiles at about 67 degrees south, overflowing it by 1,037,833 pixels and zone 15 overflows by 777,040, zone 12 by 572,259, etc.
Given the partial state of the v2~beta1 dataset, rerunning it again now would just mean we’d need to rerun it again later the next time the data changed. I can see why it does it like this. It doesn’t create empty space containing just metadata.
5871e3b makes the store deterministic based upon the UTM zones. _compute_zone_grid_deterministic projects the zone’s full six-degree longitude band across the whole range of latitude into the zone’s CRS and snaps to the shard size, so the grid depends only on the zone number and nothing else. Every zone now comes out at 69,632 by 2,002,944 pixels, which is 17 by 489 shards, and it contains all 373,765 tiles across all 60 zones with zero overflow. Empty shards cost nothing, because a sharded store with skip-existing never writes an all-nodata shard. Adding tiles is now an incremental fill.
Transverse Mercator stays well behaved even at the poles, because the band is only three degrees either side of its central meridian, and the data does go to 89.95 degrees south, beyond UTM’s nominal limit.
Both commits are on geotessera#448, which has grown from three commits to five.
Wipe, rebuild #
Then the whole v2 store was wiped and rebuilt on the new grid, from the regenerated manifest: 540 fill jobs, nine years by sixty zones, on Fargate Spot. About 250 task-hours of a 16 vCPU, 120GB task at roughly $0.37 an hour, so somewhere near $90 of compute. On demand, it would have been three times that.
The audit I ran afterwards reads the store’s own objects rather than any commit marker, so it says what is actually there.
v2~beta1 after the rebuild: 83,887 scales shards over nine years. Colour is real embedding data, bands 0 to 2, sampled once per 41 km shard. Red is a tile position that has data in some other year but not in this one.
83,887 scales shards, and the embeddings array matches exactly: every zone-year has the same count in both, and no zone-year is flagged as holding shards in one array and not the other. Thirteen shards that the NPY says should be present are not, across seven zones, and the audit names them.
What the picture mostly shows is not the conversion but the delivery. 2017 and 2024 hold 32,767 and 27,674 shards between them, and the other seven years have between 2,367 and 4,300 each. So almost everywhere with data in 2017 is red in 2019, and that is the shape of v2~beta1 itself rather than anything the rebuild did or failed to do.
Fargate performance #
I mentioned last week the variable performance I’d been seeing. This week, here are some log snippets from Zone 44 for 2017. After initially progressing well it got spot evicted, and I restarted it:
First restart #
Shards: 1,014 land, 1,014 in store, 0 to write, 54 stats catch-up read(s) (8 workers)
Zone 44 y2017: 3/54 (6%, 0.0/s, 1m47s elapsed, eta 30m22s)
Zone 44 y2017: 4/54 (7%, 0.0/s, 2m37s elapsed, eta 32m54s)
Zone 44 y2017: 5/54 (9%, 0.0/s, 3m56s elapsed, eta 38m33s)
Zone 44 y2017: 6/54 (11%, 0.0/s, 4m24s elapsed, eta 35m18s)
Zone 44 y2017: 7/54 (13%, 0.0/s, 4m49s elapsed, eta 32m23s)
Zone 44 y2017: 8/54 (15%, 0.0/s, 5m57s elapsed, eta 34m12s)
Then nothing further in the log.
Second restart #
Shards: 1,014 land, 1,014 in store, 0 to write, 54 stats catch-up read(s) (8 workers)
Then nothing further in the log.
Third restart #
Shards: 1,014 land, 1,014 in store, 0 to write, 54 stats catch-up read(s) (8 workers)
Zone 44 y2017: 3/54 (6%, 0.0/s, 2m50s elapsed, eta 48m14s)
Zone 44 y2017: 4/54 (7%, 0.0/s, 4m41s elapsed, eta 58m38s)
Then nothing further.
Fourth restart #
Shards: 1,014 land, 1,014 in store, 0 to write, 54 stats catch-up read(s) (8 workers)
Zone 44 y2017: 12/54 (22%, 1.2/s, 10s elapsed, eta 35s)
Zone 44 y2017: 27/54 (50%, 1.3/s, 20s elapsed, eta 20s)
Zone 44 y2017: 40/54 (74%, 1.3/s, 30s elapsed, eta 10s)
Zone 44 y2017: 53/54 (98%, 1.0/s, 55s elapsed, eta 1s)
Zone 44 y2017: 54/54 (100%, 0.9/s, 58s elapsed)
Stretch stats: 91,650,006 pixels from 54 shard(s) folded in
Done in under 1 minute.
Two-colouring the zones #
The last phase is the global RGB preview. The cross-zone stretch is done: 66 billion pixels sampled, plus 39 per-zone stretches for the zones dense enough to deserve their own. Then each zone is reprojected into the global pyramid, and unfortunately that cannot simply be fanned out across all sixty, because adjacent UTM zones share boundary chunks and two neighbours writing the same seam chunk at once leaves colour gaps along the join. Running it unconstrained and sequential is safe and takes about eighteen hours.
The sixty zones form an even cycle, so they two-colour. All thirty odd zones run concurrently, then all thirty even ones, and no two neighbours are ever in the same wave and completed in around an hour. The coarsening pass that builds the rest of the pyramid has finished since, so v2~beta1 is done.
A root key rollover that broke a docker pull #
A deployer job failed to pull an image:
dial tcp: lookup registry-1.docker.io on 127.0.0.53:53: server misbehaving
The deployer was fine. The target host runs systemd-resolved with strict DNSSEC validation, which the other deployer targets do not. On 11 October, the root zone began signing its DNSKEY set with KSK-2024, and systemd 249 ships only the KSK-2017 trust anchor and does no automatic key update. Once the cached root DNSKEY expired, at 02:47 UTC on 12 October, every validation chain failed. Go reports the resulting SERVFAIL as “server misbehaving”, which is unhelpful enough to send you looking at Docker.
Putting the DS records into /etc/dnssec-trust-anchors.d/root.positive and restarting resolved fixed this.
Dashboards on status.ocaml.ci.dev #
Jon’s ocaml-docs-ci monitoring was a standalone docker-compose stack with its own Prometheus and Grafana. It is now incorporated into status.ocaml.ci.dev: the docs.ci scrape job covers both dill and sage, since the two swap roles, with a host label relabelled from the address, and the dashboards are file-provisioned into their own folder with every query scoped to the job and host.
Fetching 9KB of /metrics from dill takes between 3 and 44 seconds, with the occasional 502, while sage answers the same request in 50 milliseconds. That looks like the daemon’s main loop blocking under load rather than anything in the monitoring.
FreeBSD, under a real load #
The ZFS changes in ocurrent/obuilder#220 have now been running for three and a half weeks. The median “package unavailable job” time rose from 74 seconds to 125 on 13 October, then came back down through 122, 107 and 88 to 75 by the 17th.
The change was just reflecting the system load at the time. rosemary had a load average of 44 with seven jails on the 13th and 1.73 million vnodes, and the build store ballooned to 19,755 datasets.
Perhaps we are able to cautious call this fixed?
UID standardisation #
Four more accounts moved onto the 30xxx scheme across the EEG machines, leaving only one UID collision. The recurring obstacle is that users think they have logged out, but lingering tmux or screen sessions combined with VS Code processes.
occ: a function per section #
mtelvers/occ is a C11 compiler in OCaml, written to be read, and sized to be able to build the OCaml runtime and pass OCaml’s test suite. This week -ffunction-sections was added. Each function goes into a section of its own, named .text.<name> exactly as gcc names them, so a linker told --gc-sections can drop the ones nothing refers to. Under -g, the compilation unit no longer has a single contiguous address range, so it records a list of them with DW_AT_ranges in .debug_ranges rather than a low and high address that would span an empty .text.
The more interesting part is what the flag found in occas, occ’s own assembler, since both faults are only reachable by a call that crosses a section boundary.
A call to a function the same file defines locally was being given a PLT32 relocation. Nothing can interpose on a local definition, so gas uses an ordinary pc-relative relocation against the symbol’s section instead. Within a single section, the displacement is calculated at assembly time, and no relocation occurs, which is why this had never shown. 136 of the 239 runtime corpus units differed from gas with the flag on, and none do now. The second fault is on RISC-V, which marks every section holding code with the instruction set it belongs to, whereas occas wrote one mark for the whole file. It writes one per section now, and none at all for a .text left empty.
With the flag on, the test set is identical to gas in all cases on x86-64 and, on RISC-V, in all but one file: the known divergence where gas leaves a function’s size to the linker as an add-and-subtract pair and occas works it out. The whole OCaml tree built with CC="occ -ffunction-sections" gives 1,629 passed, 60 skipped, and nothing failed, which is the same as what it gives without the flag.