force cmu-onsite runner due to firewall

.gitea/workflows/deploy.yml

@@ -17,7 +17,12 @@ on:
 
 jobs:
   build-and-deploy:
-    runs-on: ubuntu-latest
+    # Pinned to the onsite CMU runner. Only a runner registered with the
+    # "cmu-onsite" label can reach linux.andrew.cmu.edu, so this guarantees the
+    # deploy runs from inside CMU's network. A cloud/public runner will never
+    # pick up this job. The runner must be registered in Docker mode so this
+    # label maps to a Node-capable Debian image (see README).
+    runs-on: cmu-onsite
 
     env:
       # Absolute base URL where the site is served. Used to build the absolute
@@ -49,6 +54,28 @@ jobs:
           sudo apt-get update
           sudo apt-get install -y lftp
 
+      - name: Preflight — check SFTP host reachability
+        env:
+          SFTP_HOST: ${{ secrets.SFTP_HOST }}
+          SFTP_PORT: ${{ secrets.SFTP_PORT }}
+        run: |
+          set -uo pipefail
+          PORT="${SFTP_PORT:-22}"
+          if [ -z "${SFTP_HOST}" ]; then
+            echo "::error::SFTP_HOST secret is empty. Set it in Gitea → Settings → Actions → Secrets."
+            exit 1
+          fi
+          echo "Testing TCP reachability of ${SFTP_HOST}:${PORT} ..."
+          if ! timeout 15 bash -c "cat < /dev/null > /dev/tcp/${SFTP_HOST}/${PORT}" 2>/dev/null; then
+            echo "::error::Cannot open a TCP connection to ${SFTP_HOST}:${PORT}. This is why lftp reports 'max-retries exceeded'."
+            echo "Likely causes: wrong SFTP_HOST, wrong port, or the runner cannot reach CMU (needs VPN / on-campus network / firewall allowance)."
+            exit 1
+          fi
+          echo "Host is reachable on ${PORT}. Showing the SSH banner:"
+          # Non-fatal: just prints what the server advertises (SSH version, etc.)
+          timeout 10 bash -c "exec 3<>/dev/tcp/${SFTP_HOST}/${PORT}; head -c 200 <&3" 2>/dev/null || true
+          echo
+
       - name: Upload dist/ via SFTP
         env:
           SFTP_HOST: ${{ secrets.SFTP_HOST }}
@@ -60,17 +87,33 @@ jobs:
           set -euo pipefail
           PORT="${SFTP_PORT:-22}"
 
-          # Mirror the local dist/ into the remote path.
-          #   -R              : reverse mirror (local -> remote)
-          #   --delete        : remove remote files no longer present locally
-          #   --verbose       : log transferred files
-          # Host key checking is disabled because CMU's key is not pre-seeded on
-          # the runner; the connection is still encrypted.
+          # lftp drives SFTP by shelling out to the system `ssh`. We pass ssh
+          # options explicitly so this works on a fresh runner against CMU:
+          #   - StrictHostKeyChecking/UserKnownHostsFile: don't fail on an
+          #     unknown host key (the runner has never seen CMU's key).
+          #   - PubkeyAuthentication=no / PreferredAuthentications=password:
+          #     force password auth so ssh doesn't stall trying keys/agent.
+          #   - the KexAlgorithms/HostKeyAlgorithms "+..." lines re-enable
+          #     algorithms some CMU hosts still use that modern OpenSSH disables
+          #     by default (a common cause of a silent connection failure).
+          #
+          # `debug 3` prints the negotiation so any remaining failure is visible
+          # in the job log instead of just "max-retries exceeded".
           lftp -c "
             set sftp:auto-confirm yes;
-            set net:max-retries 3;
-            set net:timeout 20;
+            set sftp:connect-program 'ssh -a -x \
+              -o StrictHostKeyChecking=no \
+              -o UserKnownHostsFile=/dev/null \
+              -o PubkeyAuthentication=no \
+              -o PreferredAuthentications=password \
+              -o KexAlgorithms=+diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1 \
+              -o HostKeyAlgorithms=+ssh-rsa,ssh-dss';
+            set net:max-retries 2;
+            set net:timeout 30;
+            set net:reconnect-interval-base 5;
+            debug 3;
             open -u '${SFTP_USERNAME}','${SFTP_PASSWORD}' sftp://${SFTP_HOST}:${PORT};
+            mkdir -f -p '${SFTP_REMOTE_PATH}';
             mirror -R --delete --verbose ./dist/ '${SFTP_REMOTE_PATH}';
             bye
           "

README.md

@@ -164,6 +164,45 @@ consequences worth knowing:
 CMU's publish CGI to make the files live. It also runs on pushes to `main` and can be triggered
 manually from the Gitea Actions UI.
 
+### Why an on-campus runner is required
+
+`linux.andrew.cmu.edu` (the SFTP target) is only reachable from **inside CMU's network**. A
+cloud/hosted Gitea runner sits on the public internet and cannot reach it, which is what causes
+`lftp` to fail with `max-retries exceeded`. The workflow is therefore pinned to a self-hosted
+runner via `runs-on: cmu-onsite`; only a runner registered with that label — running on a machine
+inside CMU's network — will pick up the job.
+
+### Setting up the `cmu-onsite` runner
+
+Run [`act_runner`](https://gitea.com/gitea/act_runner) on a machine that stays on CMU's network.
+Register it against this repo (or your Gitea instance) with the `cmu-onsite` label mapped to a
+Node-capable Docker image, so the workflow's `apt-get`/Node steps work unchanged:
+
+```bash
+# On the on-campus machine (Docker installed and running):
+act_runner register \
+  --instance https://your-gitea-host \
+  --token <REGISTRATION_TOKEN_FROM_GITEA> \
+  --name cmu-onsite-01 \
+  --labels 'cmu-onsite:docker://node:20-bookworm' \
+  --no-interactive
+
+# Then run it (or install as a service):
+act_runner daemon
+```
+
+Notes:
+
+- The registration token comes from Gitea under **Settings → Actions → Runners → Create new
+  runner** (repo-level) or the equivalent org/site-level page.
+- The `:docker://node:20-bookworm` suffix is what makes the label run jobs in a Debian container
+  with Node preinstalled. Without an image mapping the label would run directly on the host, and
+  the `sudo apt-get install lftp` step would need the tools present on the machine instead.
+- Confirm the runner host can actually reach the target before relying on it:
+  `nc -vz linux.andrew.cmu.edu 22`. Some CMU subnets are firewalled off from the timeshare hosts.
+- Keep the runner process up (systemd service, container restart policy, etc.) so the nightly
+  schedule fires.
+
 ### Required secrets
 
 Set these in your Gitea repo under **Settings → Actions → Secrets**:
@@ -197,10 +236,13 @@ the clocks change.
 
 - The SFTP password is passed to `lftp` via an environment variable interpolated into the command;
   keep it only in Gitea secrets, never in the repo.
-- Host-key checking is disabled on the runner (`sftp:auto-confirm yes`) because CMU's host key is
-  not pre-seeded there. The transfer is still encrypted, but this does not verify the server's
-  identity. If you want strict host-key verification, add the server's key to the runner and
-  remove that setting.
+- Host-key checking is disabled on the runner (via `StrictHostKeyChecking=no` in the SSH
+  connect-program) because CMU's host key is not pre-seeded there. The transfer is still encrypted,
+  but this does not verify the server's identity. If you want strict host-key verification, add the
+  server's key to the runner's `known_hosts` and remove that option.
+- The upload step runs `lftp` with `debug 3` and forces password auth over an explicit `ssh`
+  connect-program. This makes connection failures visible in the job log and re-enables a few older
+  key-exchange / host-key algorithms that some CMU hosts still require.
 
 ## Data persistence