fix(deploy): trust the nginx upstream when picking the live slot
Gitea Actions Runner Test / test-job (push) Successful in 1s
CI / check (push) Successful in 30s
CI / tests-integration (push) Successful in 1m48s
CI / tests-unit (push) Failing after 1m54s
CI / tests-ui (push) Successful in 2m46s
CI / preflight (push) Skipped
CI / deploy (push) Skipped

The deploy failed with "Expected release never became healthy" after 30
attempts. Root cause: read_active_port() counted the slots answering
/api/health and only consulted the nginx upstream when the count was not
exactly one. On this host both slots were healthy, so it fell back to the
upstream file, but a leftover epicnext-cms:local replica was holding slot A
(3002). The candidate was assigned that occupied port, docker run died with
EADDRINUSE, and the health probe then answered from the pre-existing
container on that port. That container reports release "unknown" because it
was built without NEXT_DEPLOYMENT_ID, so the release comparison could never
match and the deploy timed out blaming a release that was never serving.

read_active_port() now orders its sources by how well they describe reality:

1. The nginx upstream file. It is the only source that says where public
   traffic actually enters; everything below it is a consequence.
2. A healthy slot matching that pointer.
3. The other slot when the pointer names a dead port.
4. The pointer itself when nothing answers, so rollback still has a target.
5. Slot A when no upstream file exists at all.

answers_health() was added as a retry-free sibling of healthy(); port
detection should not spend 90 seconds per slot on a process that is either
running now or never will.

start_candidate() now calls assert_port_free() before docker run, so an
occupied port fails immediately and names the listener and the containers
involved, instead of surfacing later as a misleading health-check timeout.

Added scripts/ci-deploy-ports.test.sh, which extracts the two functions from
the real script rather than copying them, and covers the regression: with
both slots healthy and nginx serving slot B, the result must not be slot A.
Verified the test fails against the old logic and passes against the new.
Wired into the check job so this is caught before an image is built.
This commit is contained in:
openhands committed 2026-10-03 18:22:40 +02:00
1 parent 704e33638f
commit 64ad9baf39
3 files changed
+205 -20

No files matched your search

+6
View File
@@ -33,6 +33,12 @@ jobs:
- name: Toolchain check
run: node scripts/check-node-toolchain.mjs
# Port selection decides which blue/green slot stays live. Getting it
# wrong starts the candidate on an occupied port, so the regression that
# caused a failed deploy is covered here, before any image is built.
- name: Deploy port-selection tests
run: bash scripts/ci-deploy-ports.test.sh
- name: Install dependencies
run: pnpm install --frozen-lockfile