NPY to Zarr has been completed for v1 and v1.1; day10 revdeps; Tessera tools packaged and failures in bench.ci.dev.
The re-convert that cost too much #
Last week’s one-pixel fix was merged, which meant the published Zarr stores needed rebuilding. The zarr/v1 store on source.coop is 540 zone-years, sixty UTM zones across nine years, and it holds both the original NPY and the Zarr. We cannot delete the NPY while the Zarr store has a known issue, but storing both is an additional expense. The original NPY-to-Zarr conversion cost under $500.
On Tuesday afternoon, I launched the full re-convert as an AWS Batch fleet on spot pricing, running the converter with --rewrite-existing-shards.
The problem, which I understood about eighteen hours too late, is that --rewrite-existing-shards is not resumable. A spot eviction does not cost you the remaining work; it costs you the whole zone-year from zero. Light zone-years finished within a single spot session. Dense ones, which is broadly all of 2024 and utm32 to utm37 in all years, never accumulated any progress at all. They were evicted, restarted, evicted, restarted.
So on Wednesday morning, I switched to on-demand at 3.2 times the spot rate, thinking that paying a little more for a few hours would be acceptable, since even the biggest zone-year is only 4 hours. However, there was clearly an additional problem as the jobs didn’t complete on schedule. As the spend increased, and with no end in sight, I killed everything and went to bed.
508 of the 540 were already right #
The next morning, before re-running anything, I took stock of the cost and what had been completed. This showed that I had spent roughly five times the original, and I was still only at 365 of 540 zone-years succeeded.
The converter writes the shard data first and computes the statistics afterwards. Scanning the failed jobs’ logs for those whose write bar had reached 100% with no remaining ETA gave 143. Add the 365 that succeeded and 508 of 540 already held correct, seam-fixed embeddings. Only 32 genuinely needed rewriting.
I read the stretch_stats_count chunks from S3 with boto3 and Blosc-decompressed them. Every write-complete slot held -1, the marker that says “statistics not computed”. Every successful slot held a real pixel total.
Looking further, at how --rewrite-existing-shards works, it reads the NPY data, writes the Zarr data in parallel with N workers, then serially rereads the entire dataset it just wrote to compute the statistics. Conversely, writing NPY into an empty store computes the statistics as it writes; therefore, there is no secondary statistics pass.
The statistics build pass can be reached if there is no existing statistics data or marker at all. Resetting a zone-year to stats-only means deleting seven small objects:
utmNN/stretch_stats_count/c/{t}
utmNN/stretch_stats_shards/c/{t}/0/0
utmNN/stretch_stats_sum/c/{t}/0
utmNN/stretch_stats_prod/c/{t}/0/0
utmNN/stretch_sample_count/c/{t}
utmNN/stretch_sample/c/{t}/0/0
utmNN/stretch_sample_scales/c/{t}/0
A non-zero count merges the old sample back in; the sums and products are additive, so the fold adds to what is stored; and a non-empty shard mask sends the fill back down the rebuild branch.
To make the fill resumable, the easiest option was to delete the existing data only in the partially rewritten zones.
The summary was 365 already done, 32 cold refills and 143 stats-only passes. The 32 cold refills ran between half an hour and 4.3 hours each.
The three commits #
I opened PR ucam-eo/geotessera#448 to update the behaviour.
- The rewrite branch now scans the grid once, gives the writers a non-zero sample cap so they return per-shard statistics, and submits every present shard the rewrite does not touch as a catch-up read in the same pool. The serial backfill at the end of each zone-year is no longer present. The progress line also now ends in a newline rather than a carriage return, which is why hours of stall went unnoticed in a redirected Fargate log.
- Shards read back are folded a batch at a time, default 64, so a kill loses at most one batch. A failed read leaves its own shard unmasked rather than condemning the whole slot. The backfill writes a resume marker beside the store, not in it, one object per zone-year, so parallel tasks never contend, and continues from it on a retry.
- The pool now hands results back in batches as they complete, for writes and catch-up reads alike, and each batch is folded immediately. An eviction loses one batch instead of everything held in memory.
Stuck jobs #
Some jobs that had been running for several hours appeared to stop reporting progress relatively soon after they started. Perhaps a dozen progress lines being produced at a rate of 2-3 updates per minute, then nothing for hours. Fargate reported the job as RUNNING, but why was there no output?
Also, the typical job rate was 0.1 shards per second, but some jobs reported a rate of 0.0 shards per second in the log, with ETAs running into weeks.
In both cases, killing the jobs and restarting fixed the issue, underlining the importance of being able to resume the jobs.
I still do not have a root cause. The working theory is an S3 read or write hanging with no timeout, under roughly 24 jobs times 8 workers against source.coop, but a solo job with no neighbours stalled the same way, which rather spoils that explanation.
I ended up with several bash scripts that reported which jobs were running and could stream the logs. When jobs stalled, I had a script to kill and restart them.
Observability #
The direct consequence of my complete lack of trust in the process is mtelvers/fargate.
The left pane is a live table of every job: zone, job definition revision, uptime, seconds since its last log line, state and progress string. The right pane is the selected job’s CloudWatch stream. Every job’s stream is tailed continuously, so the arrow keys flick between logs instantly instead of waiting for a fetch. A running job that has written nothing for more than sixty seconds is marked STALLED in red, and the header counts them. K or Delete terminates the selected job after a confirmation. Jobs that leave the queue stay listed, dimmed, for fifteen minutes with their final status.
The project was written entirely agentically. My prompt asked the agent to read the code base from a couple of existing notty projects and the shell scripts. This was the biggest success story as the working tool was produced in a few minutes and gave me the visibility and the control I wanted.
Both stores rebuilt #
The TUI gave me the visibility I needed to get the v1 heading to completion. I started v1.1 in parallel, but this time I deleted one year of embedding data at a time and refilled it with spot instances. As v1.1 is a much smaller dataset, processing the entire year took only about 4 hours. It finished on Sunday.
The header image on this post is v1, read key by key from the store’s own objects. It is 561,754 shards, each one 41 km square:
| shards | shards | ||
|---|---|---|---|
| 2017 | 63,681 | 2022 | 61,794 |
| 2018 | 62,308 | 2023 | 61,551 |
| 2019 | 63,284 | 2024 | 97,355 |
| 2020 | 62,759 | 2025 | 26,919 |
| 2021 | 62,103 |
Tessera tools #
I packaged up my Tessera tools into mtelvers/ocaml-tessera-tools, which includes the updated dpixel for generating v1.1 and v2, along with some new tools for working with Zarr data.
These are packaged in a Dockerfile which also runs under Apptainer on the HPCs. Apptainer essentially allows docker run to be invoked as the user with things like $HOME automatically mounted.
tessera-dpixel #
tessera-dpixel is an updated port of my dpixel.py, which downloads a year of Sentinel-2 and Sentinel-1 for a region and writes the seven .npy files the encoder consumes.
tessera-shard #
tessera-shard reads region polygons and lists every shard a selection touches, with the row-major indices of the sub-windows that have land in them. A shard is 4096 pixels square, which is 41 km, and the unit of work; a sub-window is a quarter of that each way and the unit of memory. I tried whole-shard downloads first and measured them: one month of one shard took 4m45s and 9.4 GB of RSS, which extrapolates to 40 to 60 GB for a full year. Fine on a big machine, but useless on an Azure worker, or an HPC machine running multiple instances. 1024 pixels is also about the same unit size as the 0.1 degree tiles produced to date.
tessera-zarr-upload #
tessera-zarr-upload uploads a complete shard to S3 using mtelvers/ocaml-s3 and mtelvers/ocaml-zarr.
The Zarr library needed three things before it could write to the production store.
The sharding encoder encoded every in-bounds inner chunk, whereas zarr-python skips any inner chunk equal to the fill value. Without that, a coastal shard that is mostly sea carries 16,384 compressed chunks of zeros. The test for this is to re-encode a shard that zarr-python wrote and compare the bytes, but it failed in a way I had not asked about: zarr-python lays inner chunks out in Morton order within a shard, not C order. With both fixes, re-encoded shards are identical.
The encoder was pulling each inner chunk out, element by element, via boxed polymorphic variants. An embedding shard is 2.1 billion elements, so that path is many minutes per shard. Ndarray is now built on flat Bigarray views with kind-specialised loops, with row-wise blits and an equals_fill that checks about a nanosecond per element, and inner chunks are encoded across domains.
| 32 MiB shard | 1 domain | 8 domains |
|---|---|---|
| encode | 146 ms | 39 ms |
| decode | 121 ms | 50 ms |
At the same rate, a real 2 GiB shard is about 10 seconds sequentially and a few seconds on 8 domains.
Zarr_s3.Store implements STORE_WITH_ERRORS over S3.Client. get now returns bytes option, so a missing key is distinguishable from a failure, which it previously was not. Integration tests run against a MinIO container.
S3.Client.put_flow uploads whatever an Eio.Flow.source yields, choosing single PUT or multipart the same way put_file does, and reads parts as concurrency slots free up, so memory is bounded by part size times concurrency rather than by the object. A 2 GiB shard becomes 32 parts of 64 MiB with four running concurrently, which also means each part retries independently. And ?delimiter on list_page and fold_pages turns list_dir into one request per thousand children instead of a walk over every object under the prefix.
Mangroves from ten examples #
A question I keep coming back to is what a Tessera parallel to ESA WorldCover would look like, and whether you could ask it something like “find all the power stations, and sort them into coal, solar and nuclear”. The framing I settled on is that “find all X” is not a similarity search, it is classification.
Mangroves make a good test case, because Global Mangrove Watch publishes a worldwide answer key to mark against. The regional experiment took 45 seconds across 7 million pixels, including classifier fits. The global one pulled about 11,000 shards, 66 GB, with no retries.
Scored against GMW on held-out world:
| Setup | Precision | Recall | F1 |
|---|---|---|---|
| 10 positive examples | 0.20 | 0.91 | 0.33 |
| plus one active-learning round | 0.35 | 0.49 | |
| plus a 20 km coastal prior | 0.50 | 0.76 | 0.60 |
Aardvark pipeline #
mtelvers/ferry is running the aardvark tile pipeline. Over the last 8 days it has completed the following:
| stage | count |
|---|---|
| download on pima | 5,874 |
| encoded on monteverde, an L4 | 3,045 |
| encoded on daintree, an A30 | 2,028 |
| shipped and uploaded | 5,073 |
| failures | 105 |
All 105 failures are because pima was moved from one network to another on Tuesday lunchtime. The encode stage failed to copy the NPY files from pima and simply moved on to the next tile. This is easy to recover later by submitting the work queue again, which will pick up the missing ones.
pima finished downloading on Sunday morning, so the GPUs now set the pace alone. 11,024 tiles in the run, 7,577 uploaded, 3,447 to go.
A benchmark that takes the host with it #
bench.ci.dev was down on Monday morning. The service was still running, but ARP failed from a machine on the same segment. I added bench.ci.dev to my Slack alert.
systemd-networkd had failed to configure eno1, so there was no DHCP renewal, and dhcpd deleted the A, TXT and PTR records three minutes later. Only the static Gandi records for bench.ci.dev still resolved. A networkctl reconfigure on the serial console fixed it, and dhcpd put all three records straight back with the identical DHCID.
That evening, my Slack alert reported it as down again: ping fine, TCP connecting, SSH never sending its banner. So I installed a small watcher that samples /proc/stat’s process counter every ten seconds and, during a spike, captures the top process names, the parents with the most children and the newest processes, syncing after each capture so the evidence survives a hard reset.
Overnight, there were three OOM kills, all js_of_ocaml, the largest at 92,874,216 kB, about 97% of the machine’s 92 GiB. Eight minutes after the last of them, dhcpd deleted the DNS records again. That is the chain of events: an OOM storm stops networkd from renewing the lease, the lease expires, and the host drops off the network, which is almost certainly what took it away the first time as well. The journal goes back to 27 September, with no OOM kills at all until 1 October, which aligns with the idea that something had just tipped the balance.
The worker run phase had --cpuset-cpus and no memory limit, and the build phase had neither. These Dockerfiles compile the OCaml compiler. ocurrent/current-bench#503 adds a --docker-memory flag applied to both phases and gives the build the same cpuset the run gets, defaulting to off so it is inert upstream until configured. It also moves CPU slot acquisition earlier, so it is held for the whole job, which limits concurrency at some cost to throughput. Deployed to the host at 32g before the PR was opened, because the deployment was the thing that mattered.
On the same day as those OOM kills, js_of_ocaml also produced 72 rows across nine commits quite happily, so this is a runaway on particular commits rather than a creeping requirement.
Jérôme then fixed it properly upstream the same morning in ocsigen/js_of_ocaml#2495, exponential code growth in inline_recursively. The day after, the host had zero OOM kills and a maximum load of 12.34 against 93.16.
A cached clone without the commit #
All the Windows base image builds started failing at step 41 of 79 with fatal: unable to read tree (dd88d077...), across 2019, 2022 and 2025, for a SHA that is perfectly available on every other platform.
The Docker layer holding the git clone step is keyed on the line containing the master pin, which has not moved since 16 September, so the clone layer built on 24 September has been reused ever since. This week, a brand-new 2.6 stanza was added, pointing at a commit made on 1 October, which the cached clone predates. Normally, a pin bump moves master too and busts the layer cache, but this was not the case this time.
Linux was immune for two reasons. It copies the checkout byte-for-byte, including .git/config, using cp, so origin in the copy still points to GitHub, and it runs git fetch before each checkout, so a stale layer heals itself. On Windows, there is no git fetch, and even if there were, it would not help, as the copy is created using git clone, so it would fetch from the file on disk, not the upstream repository.
ocurrent/ocaml-dockerfile#275 sets the remote URL back to GitHub and fetches before the checkout on Windows. Followed by ocurrent/docker-base-images#361, which raises the lower bound to 8.4.5.
Revdeps through day10, built but not turned on #
Last week, I said I would finish the opam-repo-ci side of the day10 reverse dependencies work. I have, and it is not deployed.
ocluster-worker now brackets the revdeps verb’s output with @@@OUTPUT markers exactly as the OBuilder spec already did, so the existing client-side cutter works unchanged. opam-repo-ci computes the list through that verb, with a three-way split in the day10 detection, a tolerant parse because day10’s own warnings can interleave inside the markers, and the list computed against the merged tree. Revdeps are enabled on the debian-13 x86_64 leaf alone, because OBuilder only runs them on master_distro and x86_64 anyway.
The opam-repo-ci commits are on local branches waiting to be pushed to origin/day10 and upstream/live.
Default ACLs, not setgid #
The UID renumbering across the EEG machines made some more progress. /maps had been left out of the whole renumber because it used to be a congo NFS with every user squashed to 400:400, and there was no real ownership to renumber. It has since moved to CephFS, which stores real UIDs.
At the start of the migration, I created a group maps with gid 400. Under congo every file was already owned by that group, so everyone in maps could read everything without anyone having to arrange it. On CephFS, files arrive as 30037:30037, the user’s own personal group, and nobody else in maps can get at them.
The obvious fix is setgid on the directories, but it is only half a fix: it makes new files inherit the group, and with the usual umask 022, they still come out group-readable and not group-writable. Default POSIX ACLs are umask-proof, need no setgid, and go in one place:
setfacl -R -m g:maps:rwX /maps/
setfacl -R -dm g:maps:rwX /maps/
One thing got in the way immediately. acl was not installed on any of the three machines, though CephFS /maps turned out to be mounted with ACL support once it was.