The week 26 summary worked well, so I’m doing it again. This is really a nine-day week as the last weekly post was on 26 June.
A file-descriptor leak in the ocaml-ci web UI
Back in May, I was force-restarting the OCaml-CI container every three or four days to clear a slow build-up of CLOSE_WAIT sockets that ended in HTTP 502s (OCaml-CI HTTP 502 errors). The leak accelerated this week to roughly 180-230 file descriptors per hour, bringing the service down several times.
The quick mitigation was to bump nofile (soft 102400, hard 524288) in stack.yml.in from the default 1024. This bought some breathing room for a more detailed investigation.
The previous PR ocurrent/ocurrent#473 (Scraping opam-repo-ci to a standstill) was working and opam-repo-ci was no longer leaking, but OCaml-CI doesn’t use the same stack; it uses Dream/dream-httpaf/gluten.
The first suspect was vendored gluten: IO_loop.start only closes the socket once both loop promises resolve, which never happens if a loop step raises (an ECONNRESET re-raised out of faraday_lwt_unix). I fixed that in a fork and deployed, but the descriptors kept climbing, so that wasn’t the dominant path.
strace on the production process found the real culprit. Io.close was doing shutdown SHUTDOWN_ALL and then close inside a single Lwt.catch. On an already-reset idle keep-alive socket, shutdown raises ENOTCONN; that aborted the block before close ever ran, and the exception was swallowed. So every RST idle connection leaked its descriptor. The fix is tiny: make the shutdown best-effort (try shutdown ... with _ -> ()) and always close. After the deployment, ocaml-ci-web has been flat at ~10 descriptors over 13+ hours, where the old build reached 2,271 in about the same time. These fixes are on my forks mtelvers/dream and mtelvers/gluten, which are pinned into ocaml-ci.
The opam-repo-ci scraper attack
On Saturday, I was in Nottingham for RWNYC, but opam-repo-ci was being scraped to a halt again by 7,482 unique IP addresses all spoofing real browser user-agents with Accept: text/html and no Referer. The bots were walking the deep, dynamic pages of the dashboard: /job/* on the engine (76% of the traffic) and .../variant/* on the web UI. The request rate was only ~2.6/s, comfortably under my existing 10 events/second Caddy cap, but those URLs are Cap’n Proto log-streaming requests over RPC. Even this relatively low rate is enough to saturate the engine.
Firstly, I bought some instant relief using Caddy rate-limiting (via mholt/caddy-ratelimit) by tightened right down for the two expensive path classes, from 10 events/1s to 1 event/2s. Secondly, I reinstated the proof-of-work challenge which I had dropped from the live branch during last week’s ocaml-version rollout. The PoW commit didn’t rebase cleanly on top of the OCaml 5.5.0 work, and since the previous attack had passed, I dropped it. I cherry-picked the real PoW commit back on top of live, which re-pins the current_* family to the current-web-challenge branch and adds Current_web.Challenge.v ~difficulty:12 back into the site. The engine CPU usage went from ~101% to ~19%.
However, looking at the logs, there was still obvious crawled activity. Current_web.Challenge only challenges GET requests whose Accept header contains text/html. A crawler that simply sends Accept: */* walks straight past it. So I added an optional ?protect:(string -> bool) predicate to Challenge.v, and the engine now challenges anything under /job/ regardless of the Accept header. The web frontend uses a simple Cohttp server site, so I allowed the challenge code to be called directly, not just via current_web, so that the server could be protected without recreating everything. Crawl RPCs to the engine dropped to zero.
With opam-repo-ci protected, I rolled out the same protection to OCaml-CI. With the fd leak fixed, it was no longer crashing, but the traffic, which had been causing the leak, was actually bot scraping traffic, so that might as well be blocked as well.
OpenBSD, continued
I wrote up the corrupt copies on the OpenBSD worker separately. The summary is that obuilder’s QEMU backend was issuing a hard quit before the guest’s FFS soft-updates had flushed fixed in (ocurrent/obuilder#212), and the follow-on gtar utime failure was fixed by extracting the copy under doas like every other backend (ocurrent/obuilder#213).
The postscript is that OpenBSD 7.9 was actually the latest version so I rebuilt the two base images on 7.9 which is a Makefile bump (ocurrent/obuilder#214), and ocaml-ci switched over to 7.9 in ocurrent/ocaml-ci#1066. An overnight validation run was clean: no parse errors, no gtar exits, ~290 successes against 304 legitimate build failures.
FreeBSD 15.1
Hannes opened ocurrent/ocaml-ci#1063 after mirage-crypto builds intermittently failed on the FreeBSD runner with opam lstat/unlink ... No such file or directory errors. A file opam has just written vanished out from under it. Digging back through a year of logs, it had struck on only two occasions, a handful of jobs each time, with no clear cause, and the same jobs passing on a manual resubmit.
It’s the same signature as the year-old ocurrent/opam-repo-ci#453, which felt like a race condition in ZFS. Last week’s base-image rebuild (ocurrent/freebsd-infra#22) had left rosemary on FreeBSD 15.0, so this week I upgraded to 15.1-RELEASE with OpenZFS 2.4.2 in the hope of fixing the bug. There are some genuine ZFS fixes in that update, such as (openzfs/zfs#18140) which are worth having but there’s only a slim chance that it’ll actually fix the actual issue.
The update to 15.1 is in ocurrent/freebsd-infra#24, with the CI updates in ocurrent/ocaml-ci#1065 and ocurrent/opam-repo-ci#479.
The RISC-V emulated workers
The emulated riscv64 fleet on navajo (QEMU TCG) needed attention as the /local/scratch had climbed to 93%. The docker system prune wasn’t actually reshrinking the QCOW2 files as this needs discard=unmap.
I also added cache=unsafe, which makes the workers run ~20% faster (3.06 vs 2.54 cores/worker during builds). It’s safe as these are disposable CI workers.
Genesis and the Tessera pipeline
Work continues running the Tessera inference pipeline through mtelvers/genesis, and a lot of this week’s plumbing was about getting the embeddings off the workers efficiently.
The first bottleneck was that the server was CPU-bound on SigV4. Under load, it nailed at ~115% on one core with an upload backlog. A utime/stime split showed ~83% user time spent computing SigV4 payload hashes, not waiting on I/O. Rather than switch to UNSIGNED-PAYLOAD (Ceph genuinely validates the hash and rejects a wrong one), I pushed the hashing onto an Eio.Executor_pool, so it spreads across domains. CPU spread to 178% across 21 threads, and the backlog cleared. That needed a small payload_hash hook in my S3 client, described below.
Parallelising the hashing helped, but funnelling every result through the genesis server was still the real limit. Workers had been POSTing their ~155 MB result back through genesis on okavango, which then re-uploaded to Ceph. The worker logs showed the delay up_wait=2050s (34 minutes queued on the upload lock) plus upload=507s. Workers now write straight to Ceph under a versioned incoming/ prefix and just POST the SHA-256s; which genesis uses to verify the staged object, and then promotes it to the live while preserving the x-amz-meta-sha256 metadata. The same upload went from ~508s to 9.8s with zero lock wait. This surfaced an interesting Ceph wrinkle where the worker’s incoming-scoped key owns the object, so genesis (a different principal) got 403 on the HEAD until the worker PUT with a bucket-owner-full-control ACL.
Since Ceph-to-AWS server-side copy isn’t possible, I had workers fan out to all destinations at once. Genesis exposes GET /buckets and GET /bucket/<name> (returning a sourceable env block with scoped, optionally AssumeRole credentials), and the worker uploads to Ceph and the AWS archive in one pass.
Moving uploads to direct-from-worker had an unintended consequence as they now went up through Python botocore rather than my OCaml s3 library or s5cmd. botocore defaults to CRC32 checksums. The ucam-eo/geotessera client specifically checks for the CRC64NVME checksum and aborts if it is not present. botocore is used to validate the objects and would have correctly used the CRC32 checksum. Roughly 245,000 objects, about 122,000 tiles, each two .npy files had been uploaded between 29 June and 3 July. The pipeline now specifies ChecksumAlgorithm=CRC64NVME on every PUT. The backlog was re-stamped in place server-side, with no download and no egress, by copying each object onto itself with the correct checksum:
aws s3api copy-object \
--bucket tessera-embeddings \
--key v1.1/.../grid_-79.85_76.15/grid_-79.85_76.15.npy \
--copy-source tessera-embeddings/v1.1/.../grid_-79.85_76.15/grid_-79.85_76.15.npy \
--checksum-algorithm CRC64NVME \
--metadata-directive REPLACE \
--profile tessera
The trick is --metadata-directive REPLACE: S3 refuses a copy of an object onto itself unless the metadata changes, and because REPLACE otherwise wipes it, I had to read each object’s sha256 metadata, content-type and storage class back with head-object first and re-supply them so nothing was lost. A threaded version of this (about 48 workers, ~36 objects/second) worked through the ~245,000 objects in a little under two hours with zero errors, the bytes untouched throughout. Forcing CRC64NVME on the workers briefly backfired, as it needs awscrt (botocore[crt]), which wasn’t on the fleet venvs, so uploads crashed fleet-wide with MissingDependency until I installed it across 224 boxes.
Alongside all this, the usual fleet churn: expanding the D32ds_v6 spot fleet into Poland, Norway and Switzerland; flipping Vultr GPU boxes back from the pre-release v2 model back to v1.1; releasing and re-provisioning around spot evictions and too many Planetary Computer outages.
Polarisation gap at the poles
An interesting technical observation surfaced in ucam-eo/geotessera#308, a Hudson Bay / Greenland sea-ice request from Waterloo.
Sentinel-1 images most of the world in VV/VH polarisation, and the entire Tessera pipeline and the model’s normalisation statistics assume VV/VH. But over the poles and sea ice, ESA switches to HH/HV. The VV/VH filter, therefore, drops every scene. This doesn’t matter to the Tessera pipeline as it is happy to continue without Sentinel-1 data, but it does mean there are scenes with valid data which we are discarding.
ocaml-s3
The mtelvers/ocaml-s3 client I wrote last week got a thorough hardening pass. A code review kicked it off, then a run of fixes: an anonymous --no-sign-request mode for public buckets; splitting the s3cli CLI into its own opam package so the library no longer pulls in cmdliner; unifying two duplicate INI parsers (and fixing empty env-vars being signed into malformed requests); retry with exponential backoff and jitter for 503 SlowDown etc; and the payload_hash config hook genesis needed to offload SigV4 hashing.
The 120-second per-attempt timeout I’d added was wrapping the entire body transfer, so a large or slow download could be killed spuriously. This is now scoped to connect plus headers only, with the body left to stream untimed. Furthermore, my Ceph RGW publishes 15 IPs for load balancing, but the client was pinning to the first one (which had previously saturated a single Ceph box’s NIC while the others idled), so dial now round-robins across all resolved addresses. There was a subtle concurrency bug when the wrong AWS region is specified. The request gets a 301 redirect, which updates a shared mutable region field. However, once the first fibre retargeted it, all the others concluded “not a redirect” and surfaced the 301 as an error. Exactly one request in N survived. It now always follows 301/307 and discards redirected connections rather than pooling them.
tessera-mosaic: 250 times faster
mtelvers/tessera-mosaic renders geo-correct PCA and false-colour mosaics from the 0.1 degree embedding tiles. A UK & Ireland PCA render was crawling at 0.7 tiles/second with a one-hour stall before the first tile even appeared. Four changes were applied:
- The render looked CPU-bound, but the CPU was really do garbage collection. For every output pixel, the PCA projection allocated two ~128-element float arrays: a scaled copy of the embedding vector and a mean-subtracted one, which, across billions of pixels, turned into a massive churn through the minor heap. Folding the mean-subtraction into the arithmetic instead (precomputing
mean·pconce per tile) lets the inner loop run as a single allocation-free pass over the channels. compute_pcawas opening every present tile (~810 GB of reads) to keep a handful of sample pixels each; it now samples a random 256-tile subset, turning the one-hour stall into seconds.- Parallel rendering with one stdlib
Domainper core, mutex-guarding only the canvas blit. - The key insight is that PCA is position-independent, so there’s no need to read tiles at 10m and warp down to 2km. A strided read seeks past the rows it skips, so those bytes are never read.
On pima, the new binary hit 175 tiles/second against local disk compared with 0.7, allowing a full UK & Ireland 2km mosaic in under a minute versus an hour-plus, with a long ETA. I then added an S3 streaming backend using my ocaml-s3 library, using ranged GETs to pull ~2 MB per tile instead of downloading 565 GB. A full British Isles mosaic now renders in about 10 minutes from S3, with no local data. There’s a new British Isles PCA showcase in the README.
Ceph
The Scaleway CephFS cluster, which I had repurposed as an RGW object store (~108 TB, 1.4M objects), had a busy week. The buckets.data PG resize from 266 to 512 is disk-I/O-bound at ~245 MiB/s, so it’s a multi-day job regardless of which settings get adjusted.
There was a scary moment when the custer went to HEALTH_ERR on 3 July with four OSDs on one host marked down. It wasn’t a hardware failure. All the RocksDB decided to compact at the same time and as these databases are on the HDD along with the data the compaction saturated the HDD device, so the OSDs missed their heartbeats and got marked down while still running! noout/nodown plus throttling background I/O to the floor brought the load from 9.5 to 2.75 and it healed with no re-backfill (EC 10+2, all PGs stayed active throughout).
Separately, on the Cambridge Ceph cluster, I added the tessera/v2 subvolume and mounts, and copied ~14TB of provisional v2 data from pima. Later, Sadiq noticed that tessera/change-detection was showing as d?????????? on daintree that turned out to be an MDS failover blocklisting one kernel client (Ceph fences off a client whose session goes stale, to stop it writing inconsistent data), which, unlike a FUSE client, can’t self-recover, so it just needed a remount.
Trying to publish to Source Cooperative
A long saga, which is still unfinished. The goal is to publish the ~750 TB of embeddings to Source Cooperative as open data. Their CLI creates 1-hour STS credentials via an OIDC flow, but the /.sts endpoint was 404ing in production (live only on staging), so I built a headless Playwright scraper on a locked-down host to pull credentials from the “View Credentials” dialogue and refresh them over SSH to okavango mid-sync. Despite the hacky nature, this actually worked perfectly for more than 12 hours.
For the copy itself, I built an in-region EC2 auto-scaling fleet. Both buckets are in us-west-2, so streaming tessera-embeddings to source.coop would be free, and a c7gn spot instance would manage ~2.1 GiB/s each way. I estimate it would cost $6 total.
Late one evening, the ‘View credentials’ links stopped working, so I gave up and went to bed. In the morning, everything had changed! source.coop had migrated its credential system to issue proxy credentials that route writes through Cloudflare, outside of AWS. That turned my $6 into roughly $42,000 of AWS egress. I pivoted my approach to uploading from okavango (which holds ~95% of v1 locally). The good bit is that you can now run source-coop login --duration 12h to get credentials. It’s untidy over SSH as it wants to open a browser. You can do that locally. However, it ends with a redirect to http://127.0.0.1:xxxx/callback?code=xxxx which doesn’t work, but running curl -L on that URL over the SSH session works.
aws s3 sync doesn’t stop when the credentials expire. It just keeps trying all the remaining files and failing on every one. timeout 43000 aws s3 sync ... kills the process just before the 12 hours expire.
A 32-bit OCaml compiler
My OCaml compiler fork needed some attention this week as upstream added an Iatomic_fetch_add operation with implementations for Power/AMD64/ARM64.
My fork revived and extended the 32-bit backends (PPC32, PPC64BE, i386, 32-bit ARM) to OCaml 5. It rebases nightly onto upstream and runs the full test suite on all the platforms.
i386 was a straightforward fix with (lock xaddl), but 32-bit ARM needed extra effort as the inline LL/SC loop (ldrex/strex) kept losing to register allocation. With only nine integer registers, there simply wasn’t enough scratch, and reserving one broke the more complex stdlib functions. I added a tiny C runtime helper and had ARM’s selection.ml lower the operation to a C call at the CMM level, letting GCC handle the register pressure. All four architectures passed the full run by Thursday.
Monitoring, and a few odds and ends
I added a --max-download flag to mtelvers/mon to prevent it from downloading massive pages like the health check sites.
mirage/prometheus merged some PRs so I needed to rebase my Eio extension mirage/prometheus#60. There was some discussion on the PR, but its not merged yet.
The Conservation Evidence portal got a feature update through (CEConservationEvidence/CE-Synopsis-Portal#113) which was deploy to portal.conservationevidence.com.