No macOS runners available

[aduh95]

No jobs are being picked up in https://ci.nodejs.org/job/node-test-commit-osx/ for the past 18 hours.

[ryanaslett]

Macstadium did maintenance on the Firewalls, which ended up changing the sha hash that the VPN script relies upon to stay connected to the cluster.

I updated the SHA's on both ci.nodejs.org and ci-release.nodejs.org

The procedure to do this, should it happen again, is as follows:

connect to the jenkins server,

journalctl -u orka-vpn.service -n 50 --no-pager

Apr 08 12:14:37 infra-digitalocean-ubuntu14-x64-1 systemd[1]: orka-vpn.service: Failed with result 'exit-code'.
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 systemd[1]: orka-vpn.service: Scheduled restart job, restart counter is at 8871.
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 systemd[1]: Stopped OpenConnect VPN for Orka.
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 systemd[1]: Started OpenConnect VPN for Orka.
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 sudo[657282]:     root : PWD=/ ; USER=root ; COMMAND=/usr/sbin/openconnect --user=ORKA10000020 --passwd-on-stdin --servercert=pin-sha256:<REDACTED_OLD_PIN> 203.0.113.50
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 sudo[657282]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 orka-vpn-connect.sh[657284]: POST https://203.0.113.50/
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 orka-vpn-connect.sh[657284]: Attempting to connect to server 203.0.113.50:443
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 orka-vpn-connect.sh[657284]: Connected to 203.0.113.50:443
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 orka-vpn-connect.sh[657284]: SSL negotiation with 203.0.113.50
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 orka-vpn-connect.sh[657284]: Server certificate verify failed: signer not found
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 orka-vpn-connect.sh[657284]: None of the 1 fingerprint(s) specified via --servercert match server's certificate: pin-sha256:<REDACTED_NEW_PIN>
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 orka-vpn-connect.sh[657284]: SSL connection failure: Error in the certificate.
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 orka-vpn-connect.sh[657284]: Failed to open HTTPS connection to 203.0.113.50
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 orka-vpn-connect.sh[657284]: Failed to complete authentication
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 sudo[657282]: pam_unix(sudo:session): session closed for user root
Apr 08 12:14:47 infra-digitalocean-ubuntu14-x64-1 systemd[1]: orka-vpn.service: Main process exited, code=exited, status=1/FAILURE

Then change the /root/orka-vpn-connect.sh script, and change the SERVER_CERT_PIN="pin-sha256:<REDACTED_OLD_PIN>" to the new value. The systemd service should immediately reconnect.

[panva]

No jobs are being picked up again.

Edit: https://openjs-foundation.slack.com/archives/C03BJP63CH0/p1775774308095039?thread_ts=1775627662.678259&cid=C03BJP63CH0