As noted on Thursday, the various OCaml services will need to be moved away from Equinix. Below are my notes on moving OCaml-CI.
Generate an SSH key on the new server chives using ssh-keygen -t ed25519. Copy the public key to c2-3.equinix.ci.dev and save it under ~/.ssh/authorized_keys.
Use rsync to mirror the Docker volumes. -z did improve performance as there appears to be a rate limiter somewhere in the path.
rsync -azvh --progress c2-3.equinix.ci.dev:/var/lib/docker/volumes/ /var/lib/docker/volumes/
After completing the copy, I waited for a quiet moment, and then scaled all of the Docker services to 0. I prefer to scale the services rather than remove them, as the recovery is much easier.
docker service scale infra_grafana=0
docker service scale infra_prometheus=0
docker service scale ocaml-ci_ci=0
docker service scale ocaml-ci_gitlab=0
docker service scale ocaml-ci_web=0
For the final copy, I used --checksum and also added --delete, as the Prometheus database creates segment files that are periodically merged into the main database.
rsync -azvh --checksum --delete --progress c2-3.equinix.ci.dev:/var/lib/docker/volumes/ /var/lib/docker/volumes/
The machine configuration is held in an Ansible Playbook, which includes the Docker stack for Grafana and Prometheus. It can be easily applied to the new machine:
ansible-playbook -e @secrets/ocaml.ci.dev.yml --vault-password-file secrets/vault-password ocaml.ci.dev.yml
OCaml-CI’s Docker stack is held on GitHub ocurrent/ocaml-ci and can be deployed with:
make deploy-stack