fix(ops): stop a compose replica from blocking the blue/green release

The deploy failed after the build, the migrations and the browser gate:
"Port 3002 is already in use". The holder was `epicnext-cms`, a compose
replica of release 6bffc537 that the daily scripts/docker-update.sh cron
had recreated at 03:30 with restart=unless-stopped. nginx serves the green
slot on 3003, so that replica was squatting the blue slot the next
candidate needed, and live traffic never noticed.

It got there because the updater's CI-ownership guard only tested
epicnext-cms-app. After a cutover to the green slot that container is
stopped, renamed and deleted, so the guard stopped firing while the host
stayed CI-managed.

- scripts/docker-update.sh: refuse a compose deployment on a CI host by
  checking both slot containers and the nginx upstream, which is the only
  thing that still marks the host as blue/green while a slot is idle.
- scripts/ci-deploy.sh: retire a compose replica of this checkout from
  the candidate port before starting the candidate, so a stray replica
  can never block a release again. Never a slot container, never the port
  nginx serves; anything else still fails loudly in assert_port_free.
- Tests cover both directions: a squatting replica is removed and the
  release lands, a replica on the live port is left alone.

README.md

@@ -875,6 +875,24 @@ that the CI deploy path deliberately uses `docker run` rather than compose, so
 the resource limits are declared in **both** places — limits that only existed
 in compose would never apply to a real release.
 
+### Compose on a CI host
+
+Compose and CI both want port 3002, so only one of them can own a host. A stray
+compose replica (`docker compose up`, or the daily `scripts/docker-update.sh`
+cron) parked an `epicnext-cms` container on the blue slot while nginx served the
+green slot, and every later release stopped on "Port 3002 is already in use" —
+after the build, the migrations and the browser gate. Two guards now prevent
+that:
+
+- `scripts/docker-update.sh` refuses to run on a CI host. It used to test only
+  `epicnext-cms-app`, but after a cutover to the green slot that container is
+  stopped and deleted, so the guard stopped firing while the host stayed
+  CI-managed. It now checks both slot containers and the nginx upstream.
+- `ci-deploy.sh` retires a compose replica of *this* checkout
+  (`com.docker.compose.project.config_files`) from the candidate port before
+  starting the candidate — but never a slot container, and never the port nginx
+  currently serves. Anything else still fails loudly in `assert_port_free`.
+
 ### The nginx upstream is the switch
 
 nginx does not know about container names; it reads a plain list of backends from

scripts/ci-deploy.sh

@@ -211,6 +211,39 @@ EOF
   return 1
 }
 
+# Ruim een compose-replica op die een blauwe/groene slot bezet.
+#
+# Een `docker compose up` — of de dagelijkse `scripts/docker-update.sh`, waarvan
+# de CI-eigendomscontrole per slot wankelde — laat een replica met container_name
+# `epicnext-cms` achter op poort 3002. Die draait nooit live: nginx wijst naar de
+# poort van een slot-container die dit script zelf heeft gestart, en die staat per
+# definitie aan de andere kant dan de kandidaat. Zonder deze opruimstap loopt elke
+# release vast op een bezette poort totdat iemand de container met de hand
+# verwijdert.
+#
+# Bewust smal, want een container van een ander deployment is niet van ons:
+#   - alleen een replica die uit precies deze checkout komt
+#     (com.docker.compose.project.config_files), niet een losse compose-project;
+#   - nooit een slot-container, want die beheert dit script zelf;
+#   - nooit de poort waar nginx naar wijst.
+# Wat daarnaast nog op de doel-poort zit, laat assert_port_free() met zijn eigen
+# foutmelding staan in plaats van stilzwijgend verdwijnen.
+retire_compose_replicas() {
+  local live_port="$1" cid name="" config_files="" port=""
+  while read -r cid; do
+    [ -n "$cid" ] || continue
+    name="$(docker inspect --format '{{.Name}}' "$cid" 2>/dev/null | sed -n 's#^/##p' || true)"
+    case "$name" in ''|"$slot_a_container"|"$slot_b_container") continue ;; esac
+    config_files="$(docker inspect --format '{{index .Config.Labels "com.docker.compose.project.config_files"}}' "$cid" 2>/dev/null || true)"
+    [ "$config_files" = "$deploy_dir/docker-compose.yml" ] || continue
+    port="$(docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' "$cid" 2>/dev/null | sed -n 's#^PORT=##p' | head -1 || true)"
+    [ -n "$port" ] && [ "$port" != "$live_port" ] || continue
+    echo "Removing compose replica $name on port $port: it squats a blue/green slot and is not the live release (nginx serves $live_port)"
+    docker rm -f "$name" || return 1
+  done < <(docker ps --filter "label=com.docker.compose.project.config_files=$deploy_dir/docker-compose.yml" --format '{{.ID}}')
+  return 0
+}
+
 # Zet de nginx-upstream op de nieuwe poort en herlaadt graceful.
 #
 # De nieuwe inhoud wordt eerst echt weggeschreven en dán getest: `nginx -t` leest
@@ -471,23 +504,28 @@ if [ "$blue_green" -eq 1 ]; then
     docker rm -f "$new_container"
   fi
 
-  # 2. Start de kandidaat ernaast. De live release draait ononderbroken door.
+  # 2. Een compose-replica die ooit is achtergebleven zit hier nog op de
+  #    doel-poort. Hij draait niet live en wordt dus opgeruimd, zodat de release
+  #    niet op een bezette poort stukloopt.
+  retire_compose_replicas "$old_port"
+
+  # 3. Start de kandidaat ernaast. De live release draait ononderbroken door.
   candidate_attempted=1
   start_candidate "$new_port" "$new_container"
 
-  # 3. Gezond? Release-hash klopt? Browsersmoke-test? Pas dan hoeft het oude
+  # 4. Gezond? Release-hash klopt? Browsersmoke-test? Pas dan hoeft het oude
   #    release het veld te ruimen — anders zou een mislukte e2e-test pas ná de
   #    cutover de productie breken in plaats van ervoor.
   healthy "$new_port"
   node scripts/verify-deployed-release.mjs "http://127.0.0.1:$new_port/api/health" "$sha"
   PLAYWRIGHT_BASE_URL="http://127.0.0.1:$new_port" pnpm test:e2e
 
-  # 4. Het enige onomkeerbare moment: vanaf hier wijst nginx naar de kandidaat.
+  # 5. Het enige onomkeerbare moment: vanaf hier wijst nginx naar de kandidaat.
   cutover_started=1
   switch_upstream "$new_port"
   echo "Cut over to port $new_port; retiring port $old_port"
 
-  # 5. Nu mag de oude release weg. Pas ná de swap, zodat er nooit een moment is
+  # 6. Nu mag de oude release weg. Pas ná de swap, zodat er nooit een moment is
   #    waarop er geen enkele container draait.
   if [ -n "$old_container" ] && docker inspect "$old_container" >/dev/null 2>&1; then
     docker stop "$old_container"

scripts/docker-update.sh

@@ -58,10 +58,27 @@ trap 'exit 130' INT
 trap 'exit 143' TERM
 trap 'log "Update failed; inspect $LOG_FILE. No volumes or local files were deleted."' ERR
 
-# An existing CI deployment is a different owner of the same host port.
-if [ "$(docker inspect --format '{{.State.Running}}' epicnext-cms-app 2>/dev/null || true)" = true ]; then
-  die "This host is managed by CI (epicnext-cms-app). Update through CI, not a second Compose deployment."
+# This host belongs to CI: the blue/green deploy owns both host ports (3002 and
+# 3003) and one of the two slot containers is always the live release. Compose
+# may only run where CI does not.
+#
+# Checking epicnext-cms-app alone was not enough. After a cutover to the green
+# slot the blue container is stopped, renamed and deleted, so the guard stopped
+# firing while the host stayed CI-managed. `docker compose up` then recreated a
+# replica named epicnext-cms on port 3002 — the blue slot, exactly where the next
+# candidate has to start — and every later release failed on a busy port until
+# someone removed that container by hand (see logs/docker-update.cron.log).
+# Therefore: both slot containers count, and so does the nginx upstream, which is
+# the only thing that still marks the host as blue/green when a slot is idle.
+ci_upstream_file="${CMS_UPSTREAM_FILE:-/etc/nginx/snippets/cms_upstream_servers.conf}"
+if [ -r "$ci_upstream_file" ] && grep -qsE '127\.0\.0\.1:(3002|3003)' "$ci_upstream_file"; then
+  die "This host is managed by CI ($ci_upstream_file points at a blue/green slot). Update through CI, not a second Compose deployment."
 fi
+for slot_container in epicnext-cms-app epicnext-cms-green; do
+  if [ "$(docker inspect --format '{{.State.Running}}' "$slot_container" 2>/dev/null || true)" = true ]; then
+    die "This host is managed by CI ($slot_container). Update through CI, not a second Compose deployment."
+  fi
+done
 [[ -z "$(git status --porcelain --untracked-files=normal)" ]] || die "Working tree is not clean. Commit or stash local work first."
 if [[ "$UPDATE_SKIP_PULL" = 0 ]]; then
   git rev-parse --abbrev-ref --symbolic-full-name '@{upstream}' >/dev/null || die "Configure this branch's Git upstream before updating."

src/lib/ci-deploy.test.ts

@@ -218,6 +218,11 @@ function simulateBlueGreen(scenario: string, livePort = 3002) {
 		writeFileSync(join(dir, "nginx-site.conf"), "upstream cms_app { }\n");
 		// Het live container-bestand, zodat `docker inspect` hem als draaiend ziet.
 		writeFileSync(join(dir, "epicnext-cms-app"), "old\n");
+		// Een compose-replica uit deze checkout (scenario 'port-taken-compose' of
+		// 'compose-live'); retire_compose_replicas() in het script zoekt hem op bij
+		// zijn container-id en beslist aan zijn PORT of hij live draait.
+		if (scenario === "port-taken-compose" || scenario === "compose-live")
+			writeFileSync(join(dir, "epicnext-cms"), "compose\n");
 		const result = spawnSync(bash, [resolve(root, "scripts/ci-deploy.sh")], {
 			cwd: dir,
 			encoding: "utf8",
@@ -316,6 +321,33 @@ describe("blue/green cutover", () => {
 		expect(r.upstream).not.toContain("127.0.0.1:3003");
 	});
 
+	it("removes a compose replica that squats the candidate port and then deploys", () => {
+		// Live op 3003, dus de kandidaat moet op 3002. Een compose-replica uit
+		// deze checkout zit daar al: precies de situatie die de release tegenhield
+		// totdat iemand de container met de hand verwijderde.
+		const r = simulateBlueGreen("port-taken-compose", 3003);
+		expect(r.status, r.output).toBe(0);
+		expect(r.output).toContain(
+			"Removing compose replica epicnext-cms on port 3002",
+		);
+		// Vóór het starten van de kandidaat, anders blijft 3002 bezet.
+		expect(r.calls.indexOf("docker rm -f epicnext-cms\n")).toBeGreaterThan(-1);
+		expect(r.calls.indexOf("docker rm -f epicnext-cms\n")).toBeLessThan(
+			r.calls.indexOf("docker run -d --name epicnext-cms-app"),
+		);
+		expect(r.upstream).toContain("127.0.0.1:3002");
+	});
+
+	it("leaves a compose replica alone when it is the live release", () => {
+		// De replica draait hier op 3003, de poort waar nginx naar wijst. Hoe
+		// onheilijk een compose-container ook is, hij serveert het verkeer en
+		// wordt dus niet weggenomen.
+		const r = simulateBlueGreen("compose-live", 3003);
+		expect(r.status, r.output).toBe(0);
+		expect(r.calls).not.toContain("docker rm -f epicnext-cms\n");
+		expect(r.upstream).toContain("127.0.0.1:3002");
+	});
+
 	it.each([
 		"run-failure",
 		"health-failure",

src/lib/docker-update.test.ts

@@ -56,6 +56,22 @@ function simulate(scenario: string, args: string[] = []) {
 			join(dir, "scripts/docker-prune.sh"),
 		);
 		writeFileSync(join(dir, ".env"), "HOTEL_NAME=Test\n");
+		// De CI-eigendomscontrole leest het nginx-upstream-bestand, net als
+		// scripts/ci-deploy.sh. Tests draaien op de productiehost, dus het
+		// scenario 'ci-upstream' levert zelf een bestand met een slot erin en alle
+		// andere scenario's een leeg bestand — anders zou elke test hier op een
+		// echte blue/green-host onterecht stoppen.
+		writeFileSync(
+			join(
+				dir,
+				scenario === "ci-upstream"
+					? "cms_upstream_servers.conf"
+					: "no-such-upstream.conf",
+			),
+			scenario === "ci-upstream"
+				? "server 127.0.0.1:3003 max_fails=2 fail_timeout=10s;\n"
+				: "",
+		);
 		if (scenario.startsWith("saved-"))
 			writeFileSync(
 				join(dir, ".docker-install"),
@@ -82,6 +98,12 @@ function simulate(scenario: string, args: string[] = []) {
 						: scenario.startsWith("registry")
 							? "registry.test/team/cms"
 							: "",
+					CMS_UPSTREAM_FILE: join(
+						dir,
+						scenario === "ci-upstream"
+							? "cms_upstream_servers.conf"
+							: "no-such-upstream.conf",
+					).replaceAll("\\", "/"),
 				},
 			},
 		);
@@ -169,6 +191,8 @@ describe("Docker clone updates", () => {
 	it.each([
 		"dirty",
 		"ci-active",
+		"ci-green",
+		"ci-upstream",
 		"pull-failure",
 		"build-failure",
 		"migration-failure",
@@ -177,6 +201,20 @@ describe("Docker clone updates", () => {
 		expect(r.status, r.output).not.toBe(0);
 		expect(r.calls).not.toContain("compose up");
 	});
+	// Regressie: na een cutover naar het groene slot bestaat epicnext-cms-app
+	// niet meer en zag deze controle alleen een vrijgekomen blauwe container. De
+	// dagelijkse `docker-update.sh` startte toen een compose-replica op poort
+	// 3002 en blokkeerde elke volgende release. De blauwe container alleen
+	// controleren is dus geen bewijs dat de host niet door CI beheerd wordt.
+	it.each(["ci-green", "ci-upstream"])(
+		"refuses a compose deployment while CI owns the host (%s)",
+		(scenario) => {
+			const r = simulate(scenario);
+			expect(r.status, r.output).not.toBe(0);
+			expect(r.output).toContain("This host is managed by CI");
+			expect(r.calls).not.toContain("compose up");
+		},
+	);
 	it.each(["wrong-image", "wrong-release", "wrong-public", "recreate-failure"])(
 		"never reports success for %s",
 		(scenario) => {

src/test/ci-deploy-harness.sh

@@ -37,6 +37,17 @@ curl() {
   if [ "$SCENARIO" = health-failure ] && candidate_is_new; then return 1; fi
   echo '{"database":true}'
 }
+# Een compose-replica uit precies deze checkout (`docker compose up`, of de
+# dagelijkse docker-update.sh) kan een blauwe/groene slot bezetten en de release
+# blokkeren. De scenario's hieronder leveren zo'n replica op:
+#   port-taken-compose  draait op 3002, terwijl live op 3003 draait → mag weg;
+#   compose-live        draait op 3003, de poort waar nginx naar wijst → blijft.
+compose_replica_port() {
+  case "$SCENARIO" in
+    port-taken-compose) printf '3002\n' ;;
+    compose-live) printf '3003\n' ;;
+  esac
+}
 sleep() { :; }
 # De poortconflict-check (assert_port_free in scripts/ci-deploy.sh) leest de
 # echte luisterende sockets. Zonder deze stub zou de simulatie de containers van
@@ -45,7 +56,7 @@ sleep() { :; }
 # virtueel heeft toegewezen. Standaard is geen enkele poort bezet; het scenario
 # 'port-taken' reserveert expliciet de kandidaatpoort.
 ss() {
-  local requested="" arg
+  local requested="" arg replica_port
   for arg in "$@"; do
     case "$arg" in
       sport=*) requested="${arg#sport=}" ;;
@@ -58,13 +69,35 @@ ss() {
     printf 'LISTEN 0      511          0.0.0.0:%s          0.0.0.0:*\n' "$requested"
     return 0
   fi
+  # De replica houdt zijn poort bezet zolang hij bestaat: zo ziet de simulatie
+  # dat de kandidaat op 3002 pas kan starten nadat retire_compose_replicas() hem
+  # heeft verwijderd.
+  replica_port="$(compose_replica_port)"
+  if [ -n "$replica_port" ] && [ "$requested" = "$replica_port" ] && [ -f "$TEST_DIR/epicnext-cms" ]; then
+    printf 'State  Recv-Q Send-Q Local Address:Port Peer Address:Port\n'
+    printf 'LISTEN 0      511          0.0.0.0:%s          0.0.0.0:*\n' "$replica_port"
+    return 0
+  fi
   printf 'State  Recv-Q Send-Q Local Address:Port Peer Address:Port\n'
 }
 docker() {
   echo "docker $*" >> "$TEST_DIR/calls"
   local name="${@: -1}"
   case "$1" in
+    ps)
+      [ -z "$(compose_replica_port)" ] || echo compose123 ;;
     inspect)
+      # De replica wordt in de simulatie bij zijn container-id opgezocht, net als
+      # in het echte script.
+      if [ "$name" = compose123 ]; then
+        case "${3:-}" in
+          '{{.Name}}') echo /epicnext-cms ;;
+          '{{.Image}}') echo sha256:compose ;;
+          '{{index .Config.Labels "com.docker.compose.project.config_files"}}') echo "$TEST_DIR/production/docker-compose.yml" ;;
+          '{{range .Config.Env}}{{println .}}{{end}}') printf 'PORT=%s\nHOSTNAME=0.0.0.0\n' "$(compose_replica_port)" ;;
+        esac
+        return 0
+      fi
       [ -f "$TEST_DIR/$name" ] || return 1
       if [ "${3:-}" = '{{.State.Running}}' ]; then echo true
       elif [ "${3:-}" = '{{.Image}}' ]; then echo "sha256:$(cat "$TEST_DIR/$name")"
@@ -92,7 +125,7 @@ docker() {
     *) return 0 ;;
   esac
 }
-export -f git flock pnpm curl sleep ss docker nginx candidate_is_new
+export -f git flock pnpm curl sleep ss docker nginx candidate_is_new compose_replica_port
 
 node() {
   echo "node $*" >> "$TEST_DIR/calls"

src/test/docker-update-harness.sh

@@ -16,6 +16,7 @@ docker() {
  case "$1 ${2:-}" in
   'inspect --format')
    if [ "${@: -1}" = epicnext-cms-app ]; then [ "$SCENARIO" = ci-active ] && echo true; return 0; fi
+   if [ "${@: -1}" = epicnext-cms-green ]; then [ "$SCENARIO" = ci-green ] && echo true; return 0; fi
    if [ "${@: -1}" = previous123 ] || [ -f "$TEST_DIR/restored" ]; then echo sha256:old; return 0; fi
    if [ "$SCENARIO" = wrong-image ]; then echo sha256:old; else echo sha256:new; fi ;;
   'image inspect')