Converting icechunk to zarr using Fargate workers; rebuilding the Tessera dispatcher around Zarr shards instead of 0.1 degree tiles, and day10 going live with all amd64 distributions.
Tessera #
Icechunk to zarr conversion #
Keep adding workers #
The icechunk migration runs as containers on Fargate. An individual container ran steadily, but it did not manage to use the 16 vCPU and 128GB of RAM it requested. It was clear that I needed to scale wide to process the entire data set. Allocating smaller instances hit a memory check in the code, which aborted the run. It estimated 198 GB for 32 workers, when the real peak is 5.9 GB. Removing the check appeared safe, and the image was updated.
I made a couple of other tweaks to the image, adjusting the connection pool and moving the shard checksum off the main event loop. Both help but were not transformative.
With small instances, I could process much more widely within my vCPU allowance at AWS. I used a 4 vCPU instance, which completed some UTM zones, but it also hit OOM on others. The process then became one of escalation, ending up with 3 queues of increasingly bigger vCPU allowances. AWS allows this entire process to be automated by catching the OOM exit code and resubmitting the job to the next higher capacity queue.
With those in, the migration has been grinding through the world at around 70 shards a minute and 113 GB/min, past 567,000 shards and roughly two-thirds done, across a d4v and a d8v pool.
Hemisphere seams #
Some UTM zones were completed while others were not. The ones that completed were exclusively those whose equator is ocean. The icechunk store has an overlap between the northern and southern hemispheres of around 15km. The conversion code compared the overlap region and required that they are byte-for-byte identical, otherwise it aborted. Where there is ocean, the store holds NaN on both sides so they are identical, but in all land shards they are not so the process aborted. I added GEOTESSERA_SEAM_CONFLICT=north|south, which lets the transcode resolve conflicts where a shard straddles the equator.
From tiles to shards #
For Tessera v2, we will drop NPY and write Zarr directly, therefore avoiding the time-consuming conversion process. This changes the unit of work from a 0.1 degree tile produced as an .npy file to a 4096 pixel shard, which is a single object in S3 per array, filled from 1024 pixel sub-windows.
That is a different enough dispatch problem that this is a fork of the work dispatch server rather than a command line flag. genesis-shards shares the grid maths and the worker authentication with genesis, and none of the atlas, viewport or SVG machinery.
A zone at shard resolution. The heavy outline is one 4096 pixel shard, and the small squares inside it are the 1024 pixel work units.
As with the NPY server, the current state is read from the S3 store at start up. The difference here is that rather than storing thumbnails locally on disk, these are added to the zarr store per shard.
It’s easier for shards to be atomic elements written as a whole, so the request chooses the shards, but all land chooses the sub-windows which are processed. If a ROI produced only the windows inside the requested region, the rest of that shard would be left as no-data, and because completeness is object existence, a later run covering the neighbouring region would see the shard as finished and never fill it in. This became immediately apparent in early testing: utm30 80/5 and 80/6 hold both the Isle of Man and the Mull of Galloway. The initial test request was for the IoM and then later the UK…
Four shards, each labelled with how many of its live windows are done. Green windows are produced and brown ones still hold land to process. The coastline runs straight across the heavy shard boundaries, so the Isle of Man and the Galloway coast fall inside the same shards.
Making the shard atomic across all land also makes reproducing it idempotent, which is what keeps object existence meaningful as a completion test.
The one pixel offset #
Validating the direct-to-Zarr producer against beta1 turned up an issue in the existing conversion.
A 0.1 degree tile’s UTM footprint is the bounding box of a curved quadrilateral, so its projected origin is an arbitrary real number and never a multiple of the 10 m pixel. Two pieces of code then decide which pixel of the zone grid the origin corresponds to, and they answer differently. The producer snaps outward: dpixel hands stackstac unsnapped bounds with snap_bounds=True, and stackstac floors the easting. The converter rounds, in _tile_pixel_offset. Since the zone origin is itself a multiple of the pixel size, that reduces to round(f) against floor(f) on the same fractional offset. They agree below one-half and differ by exactly one pixel above it, independently in each axis.
Tile grid_-0.55_51.95 in zone 30. Its offset from the zone origin is 49684.9685 pixels: the producer floors to column 49684, the converter rounds to 49685, and everything in that tile is published 10 m too far east.
The rule predicts every measured displacement. For each of the 24 tiles contributing to one shard north of London, searching integer offsets for the best per-pixel cosine against the published store gives the predicted shift in all 24 cases, and every tile reaches 0.99999 once shifted, which is bf16 arithmetic noise and nothing else. The shifts are only ever 0 or +1, never -1. Repeating it over Madagascar, a different zone in the other hemisphere with the canonical-northing handling in play, gave six for six on a rule derived from zone 30 and applied unchanged.
Four more tile requests closed #
day10 on seventeen distributions #
At 21:00 on 16 September, day10 went live beyond riscv64, to tackle the full amd64 distribution matrix: seventeen distribution versions against two compilers, on a single worker: doris. In the first fifteen hours, it ran 6,630 jobs that have an OBuilder counterpart on the same PR. We can’t expect a 100% match, because each job builds against opam-repository’s current master head, and the OBuilder run it is compared with was made whenever that PR was last pushed, potentially months or years ago. However, it agrees with OBuilder on 93.9% of them.
Measured from the moment a worker picks the job up, so queue time is excluded. Across all 6,630 twinned tasks that is 73 hours of machine time for day10 against 576 for OBuilder.
The sharpest case is the one that does no work at all. When a package is not available on a variant, day10 answers in a median of two seconds: it solves, finds no solution, and stops. OBuilder takes 175 seconds, because it has to build a container and run opam before it can discover the same thing. Over 1,375 such tasks that is 105 hours against one.
Every distribution ran exactly the same 390 tasks, so the rows are directly comparable.
The failures cluster by packaging rather than by builder. openSUSE Tumbleweed has the most at 108, because several conf-* packages use depext names that do not exist there, and CentOS 9 follows at 94 for the same reason, plus an older toolchain.
One more number. All 36 variants started with an empty cache at 21:00, and fifteen hours later the closure-keyed caching that took days to warm up on riscv64 was already resolving 88% of everything asked of it, uniformly between 87.6% and 88.5% across every distribution.
The differences, and what t_fibers really was #
I went through the disagreements between day10 and OBuilder. Most of the cases where day10 fails and OBuilder passes are explained by the months between the two runs: ocaml-rocksdb lost libpcre3-dev, dose3 regressed against newer compilers, the LLVM bindings moved. The cases the other way round are almost entirely the or* cluster, packages using dolog, parany or cpm without declaring them, which fail only when the closure does not happen to supply the library. Those are package bugs, whichever builder finds them.
The one that is genuinely interesting is moonpool-lwt which failed on 17 tasks with Error: Library "t_fibers" not found. t_fibers is declared in test/fiber/lib/dune as an (optional) library, and it belongs to moonpool, not to moonpool-lwt. One of its dependencies is qcheck-core, which moonpool declares as a {with-test} dependency. day10 builds moonpool as a dependency of moonpool-lwt, where with-test is correctly false, so qcheck-core is not in moonpool’s layer, so dune silently skips the optional library, so it is never installed. moonpool-lwt’s own tests then cannot find it.
OBuilder passes because it has a single shared switch. opam install --with-test moonpool-lwt puts qcheck-core in the same solution, and if it happens to be installed before moonpool is compiled, t_fibers builds and everything works.
This seems to be a limitation in how you can express some dependencies in an opam file, rather than a bug in either builder. There is no way to express “moonpool’s test dependencies are needed in order to build the thing moonpool-lwt’s tests link against”. OBuilder gets the right answer by accident of sharing a switch; day10 gets a defensible answer by isolating fully. Neither is expressing what the package actually means.
Other fixes #
Having looked at a number of PRs for the day10 work, I could see some unrelated fixes to help out on. On opam-repository#30741 a FreeBSD failure came down to a Makefile calling make directly where it needed $(MAKE), so that it resolves to gmake.
And #30767: conf-sdl2 asks for libSDL2-devel on openSUSE, which does not exist there under that name. It is SDL2-devel.
OCaml 5.5.1 and opam 2.6 #
These releases went out across all the services: ocaml-ci, opam-repo-ci, freebsd-infra and macos-infra.
Homebrew has dropped mainstream support for Intel macOS. macOS 11 and later are still supported until September 2027, but installing on a fresh machine means pinning to a specific Homebrew SHA.
occ gains a RISC-V back end #
The OCaml C Compiler is being ported to produce RISC-V native code. The interesting parts are the ones where the ABI differs from x86-64 rather than the code generation: long double is binary128 with no hardware behind it, so conversions and comparisons go through calls; a two-word argument has to start at an even-numbered register; and a floating-point argument takes an integer register next once the float registers are gone. Plus variable-length arrays, inline assembly, atomics narrower than a word, DWARF, and frames and block moves that reach past twelve bits of offset. It’s not quite finished end-to-end yet.
Two small things #
mtelvers/terminal-plotter now draws the chart before the first reading arrives, so a slow command no longer leaves a blank terminal, and it shows the newest value in the header.
The Tarides website deployer had not been running the dune tests which ensure certain content length requirements. These were added to the deployer as an additional status check. I also took the opportunity to change how images are processed so they do not regenerate when only the markdown changes. This makes a big saving on the build time.