fix(ops): stop a compose replica from blocking the blue/green release
Gitea Actions Runner Test / test-job (push) Successful in 1s
CI / check (push) Successful in 30s
CI / tests-integration (push) Successful in 1m57s
CI / tests-unit (push) Successful in 2m3s
CI / tests-ui (push) Successful in 2m49s
CI / preflight (push) Skipped
CI / deploy (push) Successful in 2m48s
Gitea Actions Runner Test / test-job (push) Successful in 1s
CI / check (push) Successful in 30s
CI / tests-integration (push) Successful in 1m57s
CI / tests-unit (push) Successful in 2m3s
CI / tests-ui (push) Successful in 2m49s
CI / preflight (push) Skipped
CI / deploy (push) Successful in 2m48s
The deploy failed after the build, the migrations and the browser gate:
"Port 3002 is already in use". The holder was `epicnext-cms`, a compose
replica of release 6bffc537 that the daily scripts/docker-update.sh cron
had recreated at 03:30 with restart=unless-stopped. nginx serves the green
slot on 3003, so that replica was squatting the blue slot the next
candidate needed, and live traffic never noticed.
It got there because the updater's CI-ownership guard only tested
epicnext-cms-app. After a cutover to the green slot that container is
stopped, renamed and deleted, so the guard stopped firing while the host
stayed CI-managed.
- scripts/docker-update.sh: refuse a compose deployment on a CI host by
checking both slot containers and the nginx upstream, which is the only
thing that still marks the host as blue/green while a slot is idle.
- scripts/ci-deploy.sh: retire a compose replica of this checkout from
the candidate port before starting the candidate, so a stray replica
can never block a release again. Never a slot container, never the port
nginx serves; anything else still fails loudly in assert_port_free.
- Tests cover both directions: a squatting replica is removed and the
release lands, a replica on the live port is left alone.
This commit is contained in:
1 parent
11ad6d4376
commit
5b2eb91c5c
7 files changed
+186
-9
No files matched your search
@@ -58,10 +58,27 @@ trap 'exit 130' INT
|
||||
trap 'exit 143' TERM
|
||||
trap 'log "Update failed; inspect $LOG_FILE. No volumes or local files were deleted."' ERR
|
||||
|
||||
# An existing CI deployment is a different owner of the same host port.
|
||||
if [ "$(docker inspect --format '{{.State.Running}}' epicnext-cms-app 2>/dev/null || true)" = true ]; then
|
||||
die "This host is managed by CI (epicnext-cms-app). Update through CI, not a second Compose deployment."
|
||||
# This host belongs to CI: the blue/green deploy owns both host ports (3002 and
|
||||
# 3003) and one of the two slot containers is always the live release. Compose
|
||||
# may only run where CI does not.
|
||||
#
|
||||
# Checking epicnext-cms-app alone was not enough. After a cutover to the green
|
||||
# slot the blue container is stopped, renamed and deleted, so the guard stopped
|
||||
# firing while the host stayed CI-managed. `docker compose up` then recreated a
|
||||
# replica named epicnext-cms on port 3002 — the blue slot, exactly where the next
|
||||
# candidate has to start — and every later release failed on a busy port until
|
||||
# someone removed that container by hand (see logs/docker-update.cron.log).
|
||||
# Therefore: both slot containers count, and so does the nginx upstream, which is
|
||||
# the only thing that still marks the host as blue/green when a slot is idle.
|
||||
ci_upstream_file="${CMS_UPSTREAM_FILE:-/etc/nginx/snippets/cms_upstream_servers.conf}"
|
||||
if [ -r "$ci_upstream_file" ] && grep -qsE '127\.0\.0\.1:(3002|3003)' "$ci_upstream_file"; then
|
||||
die "This host is managed by CI ($ci_upstream_file points at a blue/green slot). Update through CI, not a second Compose deployment."
|
||||
fi
|
||||
for slot_container in epicnext-cms-app epicnext-cms-green; do
|
||||
if [ "$(docker inspect --format '{{.State.Running}}' "$slot_container" 2>/dev/null || true)" = true ]; then
|
||||
die "This host is managed by CI ($slot_container). Update through CI, not a second Compose deployment."
|
||||
fi
|
||||
done
|
||||
[[ -z "$(git status --porcelain --untracked-files=normal)" ]] || die "Working tree is not clean. Commit or stash local work first."
|
||||
if [[ "$UPDATE_SKIP_PULL" = 0 ]]; then
|
||||
git rev-parse --abbrev-ref --symbolic-full-name '@{upstream}' >/dev/null || die "Configure this branch's Git upstream before updating."
|
||||
|
||||
Reference in new issue
Block a user