Testing day10 on the RISC-V path in OCaml-CI, diagnosing failed Windows containerd commits, and measuring the Tessera / Microsoft Planetary Computer download bottleneck.
day10 on RISC-V #
In order to test OCaml-CI with day10 I need some real-world data. My plan is to route the whole RISC-V path in OCaml-CI through day10. I picked this as it’s an underutilised path in OCaml-CI, as the current workers are very slow, so most PRs get merged without waiting for it to complete. The day10 per package layer cache should really benefit these workers, as the CPUs are low speed, so repeated recompilation is a high cost.
I repurposed carpenter, which was one of our original RISC-V machines, leaving the higher-performance Scaleway machine and the QEMU virtual machines to service opam-repo-ci as they have been doing for a while.
You may recall that last week, day10 Dockerfile generation was extended to support multiple distributions and architectures which now covers Debian on RISC-V.
On the 20th August, the new image landed on the live service, and within a minute, the logs read Building with day10. The first live batch was all of ocaml/odoc:
| odoc build | time | cache state |
|---|---|---|
| PR #1454 | 4836 s | cold |
| PR #1472 | 4818 s | cold |
| PR #1469 | 1041 s | partial-warm |
| PR #1468 | 1036 s | partial-warm |
| master | 42 s | fully warm |
Twelve of twelve succeeded, and the whole set drained in about two and a half hours on two slots.
The cold build time isn’t directly comparable with Obuilder, as Obuilder uses pre-built images from the Docker base image builder, whereas day10 builds them from scratch on demand. Secondly, day10 builds each package serially in isolation, whereas opam, invoked in Obuilder, builds concurrently. The advantage comes later when the package cache is warm.
A couple of things came out of the early runs:
- Take
prometheuswhich providesprometheus-lwtandprometheus-eioin the same repo. Whenday10tries to build the repo with OCaml 5.5.0, all packages are fine, but with OCaml 4.14.4 then the Eio package fails to solve as it requires OCaml >= 5.0. OCaml-CI knows this but has no way to tellday10only to build a subset. I’ll add a--only-packagesflag today10. - When OCluster kills a job because it is no longer needed, there is a bit of a mess to clear up in terms of temporary directories,
runcand overlay mounts.day10doesn’t handle this gracefully. This eventually caused ocluster to get stuck at aCancelling day10 jobmessage.
Inside the Windows workers #
All four Windows workers showed as disconnected at the scheduler, with nothing actually down: guests running, services running, SSH fine, logs still scrolling. That last part is the trap. The obuilder housekeeping loop runs in a separate Lwt.async, so a worker that has lost the scheduler keeps writing cheerful log lines forever.
OBuilder has a flag --obuilder-prune-threshold, which defaults to 30. The workers were stuck, unable to prune anything as they thought everything had already been pruned.
Back when I first implemented the HCS backend, I had numerous problems with Lwt on Windows, so I am running my forked version, which includes my fixes from PR#1103. However, in some cases, I worked around some issues by ignoring stderr, which meant that I was missing some error text. With those errors captured, Lwt promptly failed, reinforcing why I’d been discarding them, but it did give me the actual error:
ctr: failed to reimport snapshot:
Files/Windows/SoftwareDistribution/Download/...: unknown stream ID 9
Stream ID 9 is BACKUP_SPARSE_BLOCK. A sparse file left in Windows Update’s download cache inside a build container breaks the HCS layer export when the snapshot is committed. That explains the failure, and when that failure occurs, it isn’t cleaned up as it’s on the success path, not the failure path in the exception handler:
Lwt.try_bind
(fun () -> fn result_tmp)
(fun r ->
begin match r with
| Ok () ->
...
(Ctr.snapshot_commit ~key:snap_key ~committed_key () >>= function
| Ok () -> Lwt.return_unit
| Error (`Msg m) ->
Fmt.failwith "Failed to commit snapshot %s: %s" snap_key m)
>>= fun () ->
Os.mv ~src:result_tmp result
| Error _ ->
(* remove the snapshot, then delete result_tmp *)
...
end >>= fun () ->
Lwt.return r)
(fun ex ->
(* remove the snapshot, then delete result_tmp *)
...
Lwt.reraise ex)
Lwt.try_bind f g h only routes exceptions raised by f to the handler h. The Fmt.failwith above sits in g, so it escapes past h altogether, and neither the Error branch nor the exception handler ever runs.
Ordinary build failures take the Error branch and tidy up perfectly; the success path fails to commit due to the sparse file, and leaks. The two commit failures produced exactly two orphaned snapshots with two matching temporary directories.
Orphaned snapshots are untracked, so the LRU will never collect them, and they accumulate into chains. One worker ended up with a thirteen-deep chain hanging off its base image, at which point containerd refuses to commit the base image at all, and the worker fails three jobs in four.
Looking for other ways a snapshot could leak turned up a second one, nothing to do with sparse files. ctr run --rm only removes the container when the run client sees a clean task exit, so a cancelled or timed-out build leaves its container behind, and that container pins its rootfs snapshot. The delete then fails and the layer leaks. On my hcs-container-cleanup branch, hcs_sandbox.run now issues ctr task delete --force and ctr container rm in the finaliser on every exit path, so the build error handler can free the snapshot.
The same branch makes worker start-up recover leftovers rather than discard them. The old code called purge on result-tmp, which deleted each temporary directory but never removed the snapshot named inside it, so every restart threw away the only record of which snapshot the orphan belonged to. Start-up now removes any leftover obuilder-run-* containers first, to unpin their snapshots, then walks result-tmp and removes the specific snapshot recorded in each entry’s layerinfo before dropping the entry.
That reclaims any orphan that still has its temporary directory, which includes the sparse-file ones. It does nothing for the thirteen-deep chains, whose directories were purged long ago.
The sparse-file leak itself is still open. I am currently testing whether clearing the SoftwareDistribution/Download cache before the commit is sufficient to mitigate it.
rosemary and the prune threshold #
Last week’s mitigation on the FreeBSD worker, raising the OBuilder prune threshold to 90, seems to be OK. The store still holds around 19,200 result datasets with a snapshot each, the vnode count is around 1.19 million, and the fstatat lock contention has returned at roughly half its bad-state intensity. Each result dataset is about 7 MB, so even 10% of a 1.65 TB disk is still 19K datasets. We know that when the number of datasets reaches around 30K, the performance will be unacceptable. Potentially increasing the threshold to 95% would be better, although it’s a waste of disk space.
The dpixel download bottleneck #
We have short-term access to Zenith to run Tessera inference, and the GPUs there are impressively fast. So fast that actually supplying the dpixel data from Microsoft Planetary Computer is the real bottleneck. Most of the Tessera time this week went into the download stage of the embedding pipeline to try to keep the GPUs fed with data.
From my conversation with the MPC team a few weeks back, they had suggested that the STAC queries may be the limiting factor and that I could download an offline copy of the STAC database in GeoParquet format. We only make two STAC queries, and the database organisation isn’t ideal for our use. Furthermore, pinpointing the actual errors shows that they occur at the blob download stage rather than at the query stage. I also note that the throttling has a large time of day variance with morning being a much better time than late evening.
When using the Azure A10 machines, the inference stage was ~5 mins with the download taking approximately 10 minutes. Therefore, running three processes, each of which downloads and then infers, with a mutex ensuring only a single process infers at any time, resulted in a GPU-bound inference worker that was constantly busy.
Running on Zenith, the inference is now <45 seconds, and the download (non-Azure) is approximately 15 minutes. Add to that that each machine has 8 GPUs! Ramping up the number of worker threads results in more tiles being processed up to a point:
| workers | tiles/hour |
|---|---|
| 32 | 106 |
| 48 | 244 to 258 |
| 64 | 233 |
| 128 | 11 |
The best case utilisation is about 75%:
| sample | gpu 0 | gpu 1 | gpu 2 | gpu 3 | gpu 4 | gpu 5 | gpu 6 | gpu 7 | busy (≥50%) |
|---|---|---|---|---|---|---|---|---|---|
| sample 1 | 99 | 99 | 0 | 99 | 0 | 4 | 0 | 0 | 3/8 |
| sample 2 | 99 | 100 | 0 | 99 | 0 | 99 | 0 | 0 | 4/8 |
| sample 3 | 100 | 100 | 100 | 100 | 0 | 100 | 0 | 0 | 5/8 |
| sample 4 | 100 | 0 | 100 | 100 | 0 | 100 | 1 | 0 | 4/8 |
| sample 5 | 99 | 0 | 99 | 100 | 99 | 99 | 98 | 0 | 6/8 |
| sample 6 | 99 | 0 | 99 | 99 | 100 | 100 | 100 | 0 | 6/8 |
| sample 7 | 0 | 99 | 99 | 0 | 100 | 100 | 100 | 99 | 6/8 |
| sample 8 | 0 | 100 | 100 | 0 | 100 | 100 | 99 | 99 | 6/8 |
Arguably, with ~64 workers, any bad coding is largely papered over as any slack in one downloader is taken up by another one.
The rejected list, all measured:
| Idea | Verdict |
|---|---|
| Split CPU-download and GPU-infer across machines | Works end to end, roughly zero throughput gain |
GDAL_INGESTED_BYTES_AT_OPEN |
No-op; COG headers already fit the default 16 KB |
CPL_VSIL_CURL_USE_HEAD=NO |
A regression, replacing a cheap HEAD with a full-file GET that is then aborted |
| HTTP/2 multiplexing | Moot; Azure blob’s ALPN falls back to 1.1 |
| Cloud-cover STAC prefilter | Drops data for nearly nothing: it is a 110 km granule-wide number against an 11 km region |
| Raising the cloud-mask coverage gate | The distribution is bimodal with a near-empty middle, so the existing gate is already optimally placed |
| Per-asset retry | No viable form; serial is far slower as it is per band and parallel magnifies the load by refetching |
| Lower download concurrency | The sweep says the opposite; keep it at four |
The split architecture deserves its own note, because it was the most attractive of them. The idea was that the workload barely touches the GPUs, so having CPU-only nodes download into shared storage and feed a GPU node should be a large win. It was built and proved end-to-end, with a GPU node loading from Lustre in basically zero seconds and never touching MPC. I used 64 CPU-only jobs (4 CPUs per job), which Slurm stacked two per physical machine over 32 machines, which produced 208 tiles an hour, which is 3.3 per worker per hour, exactly the same per-worker rate as the thirty-two-worker run. Throughput scaled purely with worker count. Feeding one eight-GPU node would need about 113 nodes, and only 37 exist. It is concurrency relocated to cheaper hardware, not a throughput lever.
S1 warp #
Two changes were kept. The first is reading each image as a single chunk. The library’s default splits an 1139-pixel region into four spatial chunks, two of them slivers, and each chunk is a separate windowed read. Passing an explicit chunk size larger than any tile makes each band a single merged-range request.
In isolation, this is 17% faster. In production, across two 48-worker batches, it is worth exactly nothing. So it is a correctness fix rather than a speed win.
Sentinel-2 came out byte-for-byte identical across all seven output arrays. Sentinel-1 did not: 0.023% of pixels differ, and they sit only in rows 1024 to 1138, which is exactly the sliver chunk. At a smaller chunk size, every row is perturbed. My first instinct was to keep the S2 change only and preserve byte-consistency with tiles already built.
However, looking into the mechanism, I think the single fetch approach is actually better, considering we are looking at a tile of the base unit. The reason the tile is different depending on the chunk size is that the warp is applied in isolation to each chunk, so with a 1024-pixel chunk size, each tile is made up of 4 chunks, each of which is warped differently. The evidence supports it, as a smaller chunk size is worse, and a single chunk is correct by construction, and a sampled differing pixel sits exactly on a seam on a date with only one contributing granule, so it is the warped value itself changing.
Thread pool #
The second change removes a thread pool that fetched a date’s granules in parallel. Because dask caches its thread pool per calling thread, each granule thread got its own pool, so per-worker concurrency against MPC was the configured value multiplied by the number of granules, spiking to about double on dates that straddle a tile boundary. Removing it costs about 26% on those dates and nothing on single-granule dates, in exchange for per-worker load being exactly what it says on the tin. I think the code simplification is worth it.
Trentino reprocessed #
Using Zenith, I processed the Trentino dataset using Tessera v2 with BRDF correction. The reason being that I wanted to ensure that the MODIS correction generalises beyond the tropics. The ideal would be for the corrected dataset to be (virtually) identical in Trentino to before, showing that the correction doesn’t hurt.
The label-free test found no measurable swath-boundary artefact in northern Italy in either run. The real-edge excess (≤ 2.8%) sits below the noise floor (up to 7.7%) that alpine landscape produces at any arbitrary line, whereas in French Guiana the signal stood 5 times above it. Beta1 vs beta2 differ by ±5% of effectively zero. Thus MODIS delivers its measured −35% where the artefact actually lives, and in mild regimes it’s a no-op within noise.
The Cambridge to Scaleway blackhole #
The 15 node storage cluster in Paris became partially unreachable from Cambridge. It was a weird problem as it seems to randomly affect pairs of machines.
The first observation was that one host could not be reached while another could, and the cluster itself was entirely healthy: sixty OSDs up, monitor quorum intact, including the supposedly-unreachable machine, which was still heartbeating internally. So it was alive, and only its public path was dead.
Establishing where took a proper matrix: ten sources against fifteen destinations, extended on purpose with three non-JANET vantage points, including a UK-but-not-Cambridge control. Every failing pair has a Cambridge source, and all three external controls reach all fifteen hosts, which rules out distance and rules out the UK as a variable. Hosts on the same Cambridge subnet lose disjoint sets of destinations, which is the signature of per-flow hashing across a link group with one bad member. The failures are all-or-nothing per source-destination pair, never partial loss, and the identical set of seventeen pairs was reproduced eighteen hours apart. The number given in the table shows the number (out of 3 pings) which were successful.
| source | IP | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | a | b | c | d | e | f |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| svr-avsm2-red | 128.232.113.158 | 3 | 3 | 0 | 3 | 0 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
| svr-avsm2-watch | 128.232.113.162 | 0 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 0 | 3 | 3 | 3 | 3 |
| svr-sar87-ce | 128.232.113.166 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 0 | 3 | 3 | 0 | 0 | 3 | 3 | 3 |
| svr-sj514-ce | 128.232.113.164 | 3 | 0 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
| monteverde | 128.232.93.254 | 3 | 0 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
| chives | 128.232.124.253 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 0 | 3 | 0 | 0 |
| mint | 128.232.124.158 | 0 | 3 | 3 | 0 | 3 | 3 | 3 | 3 | 0 | 0 | 3 | 0 | 3 | 3 | 3 |
| harvester | 109.74.248.x | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
| s390x-worker-01 | 148.100.86.x | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
| ocaml-1 | 140.211.11.x | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
The knock-on effect of this was that the CI workers and builders randomly failed to connect to GitHub or other remote sources. Anil posted on discuss.ocaml.org.
Tracing in reverse from the far end showed no issues connecting through the JANET backbone up to the first Cambridge router, and then silence, where a working pair from the same source at the same moment took the same path and worked correctly.
By late Sunday evening, this situation appeared resolved, but with very little detail available. CL Status
The Conservation Copilot goes live #
Last week’s sizing exercise for the Conservation Evidence Copilot was deployed on a Xen Orchestra host in Cambridge.
The first attempt shared a host with a live site, but that machine had only 3.9 GB of RAM. On startup, Celery went into a fork-crash loop, its children killed as they loaded models and respawned to be killed again. Moving to a VM with 8GB resolved this. I used Caddy as a reverse proxy as usual.