We maintain a mirror (archive) of all opam packages. To take advantage of this, add the archive mirror to opam by setting the global option.
opam option --global 'archive-mirrors+="https://opam.ocaml.org/cache"'
How is the mirror generated and maintained?
opam has a command that generates the mirror, which defaults to reading packages from the current directory.
opam admin cache --link=archives ./cache
The base image builder pulls ocaml/opam:archive, runs opam admin cache to update the cache, and then pushes it back ocaml/opam:archive.
opam2web, which publishes opam.ocaml.org, pulls ocaml/opam:archive and then runs opam admin cache to populate any new items in the cache and then makes the cache available at https://opam.ocaml.org/cache.
Until today, the step indicated by the dotted line was missing. Kate had pointed this out as long ago as 2023 with issue #249 and PR #248, but, for whatever reason, this was never actioned.
With the current unavailability of camlcity.org, this has become a problem. On Monday, I patched opam2web’s Dockerfile to include access to the mirror/cache, which allowed opam2web to build. However, subsequent builds failed because the updated opam.ocaml.org used the latest version of ocaml/opam:archive. This was built on Sunday when camlcity.org was down; therefore, the source for ocamlfind had been dropped from the mirror.
How to do we get out of this problem?
Updating the base image builder does not fix the problem, as camlcity.org is still down and the current ocaml/opam:archive does not contain the missing packages. We only tag the latest version on Dockerhub, but looking through the base image builder logs allowed me to find the SHA256 for last week’s build. ocaml/opam:archive@sha256:a0e2cd50e1185fd9a17a193f52d17981a6f9ccf0b56285cbc07f396d5e3f7882
Taking PR #248, and pointing it to the older image, I used the base image builder locally to push an updated ocaml/opam:archive. This is ocaml/opam:archive@sha256:fb7b62ee305b0b9fff82748803e57a655ca92130ab8624476cd7af428101a643.
- from ~alias:"opam-archive" "ocaml/opam:archive" @@
+ from ~alias:"opam-archive" "ocaml/opam:archive@sha256:a0e2cd50e1185fd9a17a193f52d17981a6f9ccf0b56285cbc07f396d5e3f7882" @@
Now I need to update opam.ocaml.org, but opam2web doesn’t build due to the missing ocamlfind. Checking the opam file showed two source files are needed. One is on GitHub so that’ll be ok.
...
url {
src: "http://download.camlcity.org/download/findlib-1.9.6.tar.gz"
checksum: [
"md5=96c6ee50a32cca9ca277321262dbec57"
"sha512=cfaf1872d6ccda548f07d32cc6b90c3aafe136d2aa6539e03143702171ee0199add55269bba894c77115535dc46a5835901a5d7c75768999e72db503bfd83027"
]
}
available: os != "win32"
extra-source "0001-Harden-test-for-OCaml-5.patch" {
src:
"https://raw.githubusercontent.com/ocaml/opam-source-archives/main/patches/ocamlfind/0001-Harden-test-for-OCaml-5.patch"
checksum: [
"sha256=6fcca5f2f7abf8d6304da6c385348584013ffb8602722a87fb0bacbab5867fe8"
"md5=3cddbf72164c29d4e50e077a92a37c6c"
]
}
Luck was on my side, as find ~/.opam/download-cache/ -name 96c6ee50a32cca9ca277321262dbec57 showed that I had the source in my local opam download cache. I checked out opam2web, copied in the file 96c6ee50a32cca9ca277321262dbec57 and patched the Dockerfile to inject it into the cache:
diff --git i/Dockerfile w/Dockerfile
index eaf0567..84c9db8 100644
--- i/Dockerfile
+++ w/Dockerfile
@@ -34,6 +34,7 @@ RUN sudo mkdir -p /usr/local/bin \
&& sudo chmod a+x /usr/local/bin/man2html
RUN sudo mv /usr/bin/opam-2.3 /usr/bin/opam && opam update
RUN opam option --global 'archive-mirrors+="https://opam.ocaml.org/cache"'
+COPY 96c6ee50a32cca9ca277321262dbec57 /home/opam/.opam/download-cache/md5/96/96c6ee50a32cca9ca277321262dbec57
RUN opam install odoc
RUN git clone https://github.com/ocaml/opam --single-branch --depth 1 --branch master /home/opam/opam
WORKDIR /home/opam/opam
The final step is to build and deploy an updated opam2web incorporating the updated mirror cache. In conjunction with the updated base image builder, this will be self-sustaining. I wrapped the necessary steps into a Makefile.
OPAM_REPO_GIT_SHA := $(shell git -C ~/opam-repository fetch upstream && git -C ~/opam-repository rev-parse upstream/master)
BLOG_GIT_SHA := bdef1bbf939db6797dcd51faef2ea9ac1826f4a5
OPAM_GIT_SHA := 46234090daf4f9c5f446af56a50f78809c04a20a
all: opam2web
cd opam2web && docker --context registry.ci.dev build --pull \
--build-arg OPAM_REPO_GIT_SHA=$(OPAM_REPO_GIT_SHA) \
--build-arg BLOG_GIT_SHA=$(BLOG_GIT_SHA) \
--build-arg OPAM_GIT_SHA=$(OPAM_GIT_SHA) \
-f Dockerfile --iidfile ../docker-iid -- .
@SHA256=$$(cat docker-iid)
docker --context registry.ci.dev tag $$SHA256 registry.ci.dev/opam.ocaml.org:live
docker --context registry.ci.dev login -u $(USERNAME) -p $(PASSWORD) registry.ci.dev
docker --context registry.ci.dev push registry.ci.dev/opam.ocaml.org:live
docker --context opam-4.ocaml.org pull registry.ci.dev/opam.ocaml.org:live
docker --context opam-4.ocaml.org service update infra_opam_live --image $$SHA256
docker --context opam-5.ocaml.org pull registry.ci.dev/opam.ocaml.org:live
docker --context opam-5.ocaml.org service update infra_opam_live --image $$SHA256
opam2web:
git clone --recursive "https://github.com/ocaml-opam/opam2web.git" -b "live"
Check that ocamlfind is included in the new cache
wget https://opam-4.ocaml.org/cache/md5/96/96c6ee50a32cca9ca277321262dbec57
wget https://opam-5.ocaml.org/cache/md5/96/96c6ee50a32cca9ca277321262dbec57