A big week, with a couple of long-running efforts finally wrapping up: the Source Cooperative mirror is essentially complete now that we have direct write access, and the three-week Ceph resize has converged. Elsewhere, a slow-down in the genesis job server that looked like a garbage-collector problem turned out to be an HTTP handling bug.
io_uring is not available (ENOMEM)
Thomas reported that eio PRs on OCaml-CI were intermittently failing their mdx documentation tests with io_uring is not available (ENOMEM), only on the big 256-core worker. It was not a real out-of-memory: io_uring_setup charges ring pages against RLIMIT_MEMLOCK, and because OBuilder containers use no user namespace, every job runs as uid 1000 and shares one machine-wide 8 MB locked-memory budget, which is exactly 1024 default eio rings. The fix was LimitMEMLOCK=infinity on the worker service, rolled out across the linux-x86_64 and linux-ppc64 pools. I wrote the whole investigation up separately mid-week: io_uring is not available (ENOMEM).
A GitHub login for the health-check logs
Last week, I put a blunt global rate cap on /log/ to survive a scraper farm harvesting the opam-health-check logs. This week, I looked at making serving the pages cheap enough that scrapers wouldn’t matter. I tried to swap Kate’s pixz/.txz archives for zstd to get random access. It was twice as fast, but not night and day, and none of the alternatives (git, sqlite, parquet, serving pre-compressed bytes, trained dictionaries) justified the complexity against a client that reads scattered historic dates.
I considered whether a custom library would hit this niche, and developed mtelvers/logpack. Logs were ingested as words into an LZW-style dictionary. It worked and compression approached .tar.xz, but an unbounded shared dictionary didn’t really scale, and it needed a lot more work, which I didn’t have time to work on.
Thus, commit d70a2bc adds an optional GitHub login in front of /log/. The config now has an opt-in github-oauth block, and when absent, the server behaves byte-for-byte as before. Only /log/* is gated; the index, diffs, JSON API, and metrics remain public. Any GitHub account is accepted, so it is a login wall rather than an access list. It is built and pushed, but not yet deployed.
FreeBSD worker stall
Jan reported the FreeBSD worker rosemary had stalled. The scheduler showed 20 running slots, but the host had zero build jails and a flat idle load. First, the wall clock had stepped backwards, and with no ntpd running, nothing prevented it. Second, the FreeBSD weekly locate.updatedb job had been running a find / for over two days, grinding through the ~23,000-dataset obuilder ZFS store and starving the pool of I/O.
The two host settings (disable the weekly locate, run ntpd in slew-only mode so it never steps mid-build) are captured in the ocluster role in ocurrent/freebsd-infra#25.
Tessera job server slow down
As the number of workers has increased, the CPU load on mtelvers/genesis has been increasing. I assumed that this was largely to be expected, more clients, larger maps, etc, but I decided to see exactly where the time was being spent. Logs showed that most of the 250% CPU was being expended in the multicore garbage collector, which is very odd and largely a symptom of something else.
The workers fetch the 175 MB model file with curl -C -, which first sends a HEAD to size it and then a ranged GET to resume. genesis had only a single plain GET handler, so it treated everything, HEAD and ranged GET alike, as a request for the whole file. cohttp-eio emits a body regardless of method, so the HEAD read the whole 175 MB into the heap, and a ranged request got a 200 with the full entity instead of a 206, however curl -C - didn’t download the file as the size was the same and closed the connection. The bytes didn’t leave the machine, but the file was loaded into memory and then dropped. The net effect was genesis reading the model into memory about 2.5 times a second, roughly 430 MB/s of large-object allocation straight into the OCaml major heap, which the garbage collection then freed. A new /debug/gc endpoint made the diagnosis obvious: the heap was about 3 GB, but the live set was only ~179 MB, so this was allocation churn, not retained state. The fix was to give genesis proper HTTP semantics, parsing the range and replying 206, and answering a HEAD with the size but no body. The allocation and the CPU load both dropped away, and /status went from 1.7 s back to under a millisecond.
Vultr and MPC
As the Azure credits have ended, I’ve been pushing the Vultr machines harder. The Microsoft Planetary Computer download time dominates, and I struggle to keep the 8 GPUs per machine at 100% busy. Taking the machines to 176 concurrent jobs appeared to work, but there was an issue. MPC can occasionally fail to retrieve blobs, so the code needs to allow for a blob that is genuinely unavailable, but when pushed very hard, the MPC rate limit looks exactly like an unavailable blob. Artefacts began to appear when too many blobs were dropped, degrading embedding quality. Fortunately, the inference logs allowed me to identify all tiles with more than a single blob failure, and those were deleted and resubmitted. I’ve added guard logic to prevent this from happening and also reduced the concurrency to a more reasonable 64 jobs per worker.
Sentinel-2 banding, continued
Last week, I traced the diagonal banding in the Amazon PCA mosaics to a Sentinel-2 view-angle (BRDF) effect and fitted a correction from the data.
A section of the Amazon 2025 PCA mosaic. The broad diagonal bands are the Sentinel-2 orbit swaths; the finer stripes within them are the twelve detector modules, close enough to count by eye.
The swath boundary is solved
The coarse stripe is the swath boundary, where two adjacent orbits view the same ground from opposite ends of the view-angle range. I tested NASA’s real MCD43 MODIS BRDF product and the peer-reviewed Ploton 2026 method against last week’s hand-fitted c-factor, measuring the residual inter-orbit step on the banding axis (a same-cell measurement, so geography cancels exactly):
| correction | axis step | 128-d coherent step |
|---|---|---|
| baseline (uncorrected) | −1.24 | 6.14 |
| MCD43 local coefficients, no tuning | −0.03 | 5.03 |
| last week’s fitted k≈2.5 (sidelap) | −0.02 | 4.37 |
| Ploton 2026 (published table, no tuning) | +0.08 | 4.87 |
MCD43 and Ploton both null roughly 98% of the swath step with no tuning, independently confirming last week’s result that rainforest needs about twice the global-average correction (MCD43’s Amazon coefficients come out ~2× the global mean; Ploton needs only a k≈0.9 residual). Two fixes of my own were needed to get there. Raw per-scene MCD43 is so noisy over the tropics that it made the banding 52% worse and introduced hard 11 km tile seams; a coarse ~24 km annual coefficient field, averaging ~77,000 retrievals per cell, cut that noise ~280× and matched the archive’s banding amplitude to within 2%. And parse_angles had been assigning the twelve MSI detector view-angle grids with a nanmean, averaging away the very detector steps that matter; keeping them per-detector puts five sharp c-factor steps back at 24.9 km spacing (290 km swath over twelve detectors). I also committed --pca-save/--pca-load to tessera-mosaic, so corrected and uncorrected renders share one PCA basis; otherwise, the colour differences are basis artefacts rather than real ones.
The detector striping is not a BRDF effect
Counting bands and running an FFT across a swath shows twelve bands at ~23.6 km spacing plus a dominant 47 km odd/even detector-pair stagger, matching the Sentinel-2 detector pattern. But adjacent detectors differ by only ~0.3 degrees of view angle, a 0.3–0.5% reflectance step through the BRDF kernels, so a per-detector BRDF correction removes only ~16% of the detector-band power. The rest is inter-detector radiometric calibration and odd/even parallax, which no reflectance-domain correction can reach; the Ploton authors noted the same issue.
A section of the same strip after the Ploton correction. The swath-boundary step is gone, but the fine vertical detector striping is still there across the forest, because it is a radiometric effect rather than a view-angle one.
Knowing how is not knowing when
The hard problem is knowing when to apply a correction, because the coefficients are surface-specific. Measuring the fraction of the cross-swath difference each method removes, across biomes:
| biome | MODIS coeff. | forest coeff. | NDVI-gated |
|---|---|---|---|
| Amazon forest | 26% | 66% | 64% |
| Congo forest | 24% | 47% | 42% |
| Iowa cropland | 22% | 33% | 37% |
| Kansas grassland | 42% | 66% | 65% |
| France cropland | 29% | 53% | 42% |
| Sahara sand | 34% | −202% | 34% |
The tropical-forest coefficients that work so well in the Amazon are catastrophic on the Sahara (−202%) and poor on Iowa maize, so a generic strong correction is dangerous, and MODIS coefficients are the only safe global default. Gating by land cover is the obvious idea, but NDVI can’t separate Iowa maize (0.87) from Amazon (0.86), the S1 SAR cross-pol gate reads the Sahara as “more woody” than maize, and a self-estimated anisotropy from same-day orbit overlaps (the one method that does separate Amazon from Iowa) does not transfer across time: a single-pair fit decays to no better than the generic model by about 50 days apart. Running French Guiana end-to-end through the pipeline with a per-detector MODIS correction did cut the banding RMS by 35% (swath scale −40%, detector −26%) while keeping the embeddings in distribution (per-channel correlation 0.97–0.99, means shifting under 4 DN), so a physical correction is at least safe to feed a future retraining as an input.
This is an ongoing effort with no firm conclusions as yet.
geotessera backfill
The genesis priority queue continued processing coverage requests. #339 (western DRC) went import to submit to a verified 61/61 on S3 across three years, and closed. #293 (MONAFOR and MARAS, Latin America) was reopened after a missed comment; v1.1 covers 77% of its 8,887 site tiles, so I imported the region and processed the ~2,100 missing 2024 tiles, with Mexico and Argentina identified as the big lever (1,850 of the 2,600 missing plots between them). On the African run, Cameroon finished and Chad (10,709 tiles) is currently running.
Source Cooperative: direct write, and done
The multi-week effort to mirror the Tessera embeddings to Source Cooperative finished this week, and the thing that unblocked it was getting direct write access. source.coop’s newly documented “bring your own IAM role” path meant I could email them our role ARN and have it granted. Direct writes to the real us-west-2 bucket now work with no Cloudflare proxy, no per-object egress and no rate-limiting. That retired the entire proxy, credential-scraping and refresh apparatus in one go!
With direct access, and because both the authoritative tessera-embeddings bucket and source.coop are AWS buckets in the same region, everything is copied as free server-side S3-to-S3 transfers where the bytes never touch a client. v1 (873,000 missing objects) finished on 29 July; v1.1 finished the next day; and the landmask tiffs went too. A checksum audit found 796,000 v1 objects, almost all from 2017, still on CRC32, so I re-stamped them to CRC64NVME with the server-side self-copy trick from a few weeks ago, and the whole published dataset is now uniform. The direct path cost essentially nothing in transfer, against the roughly $3,000 the proxy route would have cost for the v1 gap alone and ~$42,000 for the full dataset.
Ceph
The three-week buckets.data resize on the Scaleway RGW cluster finally converged. At one point, the recovery rate had dropped to ~20 MB/s while the disks sat ~99% idle, so it was neither a concurrency nor an mclock-profile limit. The real bind was destination concentration; one underfull OSD was the backfill target for 41 of the 48 waiting PGs, and an OSD only accepts osd_max_backfills at a time. A per-OSD cap on that one OSD took recovery to 124 MB/s. By 2 August, the misplaced count reached zero from a ~50% peak, so I reset all the temporary tuning to defaults and repaired the two inconsistent PGs. The cluster went from HEALTH_ERR to HEALTH_WARN, with only the benign scrub backlog and the marginal osd.57 (still to be replaced) remaining.
Separately, I gave the cluster a native HTTPS endpoint, s3.geotessera.org. The certificate came via an ACME DNS-01 alias delegation, a CNAME from the challenge record into the caelum.ci.dev zone I control. It now serves HTTPS across all fifteen gateways.