diff --git a/.dockerignore b/.dockerignore index 73032f50..0220e222 100644 --- a/.dockerignore +++ b/.dockerignore @@ -26,6 +26,18 @@ bun.lockb public/nitro-assets public/swf +# Databasebackups (~232MB, dagelijkse rotatie): horen niet in de buildcontext. +# Ze maakten elke context groot en invalideerden bij elke rotatie de +# `COPY . .`-laag, waardoor de Next-build volledig opnieuw draaide. +backups/ + +# Wordt tijdens de build opnieuw gegenereerd door `pnpm assets:editor` +# (bron: node_modules/tinymce) en staat niet in git. +public/vendor/ + +# Lokale buildcache voor `docker compose build` (cache_from/cache_to). +.docker-cache/ + .deploy.lock logs diff --git a/.gitignore b/.gitignore index f81169dd..cf69c15e 100644 --- a/.gitignore +++ b/.gitignore @@ -57,6 +57,9 @@ blob-report/ build-reports/ !/docs/performance-budgets.md +# Local buildcache (docker compose cache_from/cache_to) +.docker-cache/ + # One-off local snapshots (o.a. catalog-integrity/pre-fix.json): runtime-werk, # geen bron. Per map opgeslagen om een incident terug te kunnen lezen. backups/ diff --git a/Dockerfile b/Dockerfile index 9bff7de1..bdaa443b 100644 --- a/Dockerfile +++ b/Dockerfile @@ -3,21 +3,43 @@ FROM node:26.10.0-alpine AS migrations WORKDIR /app ENV NEXT_TELEMETRY_DISABLED=1 -# Installeer git, bash en pnpm v12. bash is nodig voor +# Installeer git, bash en corepack. bash is nodig voor # scripts/with-memory-cap.sh (gebruikt bashisme zoals arrays en BASH_REMATCH); # Alpine levert geen bash mee. +# +# pnpm komt niet meer uit een losgepind npm-pakket, maar uit corepack: die +# leest de versie én de sha512-hash uit de `packageManager`-pin van +# package.json. Daarmee kan de image nooit uit sync raken met het lockfile +# (vroeger stond hier pnpm@12.10.1 terwijl package.json 12.11.2 voorschreef) +# en wordt de gedownloade pnpm-binary ook in de runtime-image bewaard. RUN --mount=type=cache,target=/var/cache/apk \ + --mount=type=cache,target=/root/.npm \ apk add --no-cache git bash \ - && npm install -g pnpm@12.10.1 + && npm install -g corepack@0.36.0 \ + && corepack enable pnpm + +# corepack bewaart de gedownloade pnpm buiten de cache-mount, zodat de runtime +# (`docker run --entrypoint pnpm db:migrate`) pnpm ook +# zonder netwerk kan gebruiken. +ENV COREPACK_HOME="/usr/local/share/corepack" +ENV COREPACK_ENABLE_DOWNLOAD_PROMPT=0 # Stel het PATH zo in dat Alpine pnpm gegarandeerd overal herkent ENV PNPM_HOME="/usr/local/share/pnpm" ENV PATH="$PNPM_HOME:/usr/local/bin:$PATH" +# Alleen package.json: de `packageManager`-pin bepaalt welke pnpm corepack +# installeert. Deze laag blijft gecachet wanneer alleen de lockfile verandert. +COPY package.json ./ +RUN corepack install + COPY package.json pnpm-lock.yaml* pnpm-workspace.yaml* .npmrc* ./ -# Voer de installatie uit met de pnpm v12 store cache-mount -RUN --mount=type=cache,target=/root/.local/share/pnpm/store \ +# Voer de installatie uit met de pnpm v12 store cache-mount. De mount moet op +# de store staan die pnpm ook echt gebruikt ($PNPM_HOME/store/v11); met het +# oude pad (/root/.local/share/pnpm/store) cache'te de mount niets en +# downloadde elke build alle packages opnieuw. +RUN --mount=type=cache,target=/usr/local/share/pnpm/store \ pnpm install --frozen-lockfile --ignore-scripts COPY . . diff --git a/README.md b/README.md index 01e76f44..0d788cb4 100644 --- a/README.md +++ b/README.md @@ -307,6 +307,28 @@ responsibility. Docker uses the committed pnpm lockfile and pinned Node version. The build cache can stay enabled: copying changed source invalidates the application build layer. Deleting all Docker cache is not an update mechanism. +#### Build toolchain and cache + +The image never installs a pinned pnpm of its own. The Dockerfile installs +[corepack](https://nodejs.org/api/corepack.html) and lets it resolve the pnpm +version *and* its sha256/sha512 hash from the `packageManager` field in +`package.json`. One pin therefore drives the Docker build, the CI runner and +the host — the image can no longer drift away from the lockfile, and a tampered +package tarball is rejected instead of silently installed. Bumping pnpm is a +single edit in `package.json` (plus the README version table, checked by +`pnpm readme:versions:check`). + +Two caches make repeated builds cheap: + +- **pnpm store** — a BuildKit cache mount on `$PNPM_HOME/store` keeps the + downloaded packages between builds. A lockfile change then re-uses the store + (`reused 493, downloaded 0`) instead of re-downloading every package. +- **Local build cache** — `docker compose build` exports the build cache to + `.docker-cache/cms` and imports it again on the next build. Unlike the + daemon's own cache, this survives `scripts/docker-prune.sh`, which trims the + daemon cache to 4 GB after every deploy. The directory is ignored by both + Git and the build context; delete it to force a cold build. + ### Volumes Only the CMS (and the optional avatar imager container, see [Avatar Imaging](#avatar-imaging)) run in Docker. The heavy client assets and gamedata stay on the host and are shared into the container so imports and uploads persist: @@ -346,7 +368,13 @@ docker compose build --build-arg RUN_USER=1000 The CMS container runs with `network_mode: host`. This lets the existing `127.0.0.1` references in `.env` keep working against host services — MariaDB (`3306`), Valkey/Redis (`6379`), the emulator RCON/API (`3003`/`3001`) and the avatar imager — without re-writing any environment variables. The container consequently listens **directly on the host's port 3002**, so stop any previous host-side CMS (e.g. `pm2 stop next`) that occupies that port before starting it. -The **build** also runs on the host network (`build.network: host`) because this host disables Docker iptables (`/etc/docker/daemon.json`: `"iptables": false`), which would otherwise leave build containers without outbound NAT/DNS when fetching packages. +The **build** deliberately does *not* use host networking. BuildKit (v0.26, Docker 29) +refuses to grant host networking unless every caller passes +`--allow=network.host`, and `docker compose build` has no such flag — asking for +`build.network: host` therefore turned every rebuild into an immediate +"additional privileges requested" failure. The build only needs outbound +internet (apk, corepack/pnpm, next/font/google), which the default bridge +provides. ### Health Check @@ -595,7 +623,7 @@ URLs look like `https:///imaging/avatarimage?figure=hd-180-1.ch-210-66&im ### Requirements (host) -- Docker (daemon with `build.network: host`, same as the CMS build — this host disables Docker iptables). +- Docker (daemon with BuildKit; the CMS build runs on the default bridge, not on host networking — see [Networking](#networking)). - An nginx that serves `/gamedata/` (FigureData/FigureMap/EffectMap…) and `/gamedata/bundled` (furniture bundle assets — `.hab` by default, the extension imports write and this deployment's client requests, plus any legacy `.nitro` still on disk; choose the written extension with `FURNI_BUNDLE_EXT`) over HTTP so the renderer can fetch them. The compose file points at `host.docker.internal:8081`. ### Configuration — `/docker/Polaris-imager/.env` diff --git a/docker-compose.yml b/docker-compose.yml index 6d121155..b8bab787 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -29,6 +29,18 @@ x-cms: &cms dockerfile: Dockerfile args: NEXT_DEPLOYMENT_ID: ${CMS_RELEASE:-unknown} + # Lokale buildcache op schijf. `scripts/docker-prune.sh` brengt de + # daemon-cache na elke deploy terug tot 4GB, waardoor de lagen van de + # vorige release verdwijnen. Deze export overleeft die prune: een rebuild + # na een lockfile-wijziging hergebruikt dan de al gedownloade packages + # (pnpm store) in plaats van ze opnieuw te downloaden. + # De map ligt binnen de buildcontext, zodat buildx een ontbrekende + # cachemap stilzwijgend negeert i.p.v. de build te laten falen; zowel + # .dockerignore als .gitignore sluiten de map uit. + cache_from: + - type=local,src=.docker-cache/cms + cache_to: + - type=local,dest=.docker-cache/cms,mode=max # Runtime host networking IS required: nginx reaches the CMS over 127.0.0.1 # on the single port that is actually served (3002). network_mode: host @@ -67,6 +79,10 @@ services: byparr: image: ghcr.io/thephaseless/byparr:latest + # `latest` is een drijvende tag. Zonder pull_policy haalt compose de + # bestaande image nooit opnieuw op en blijft byparr hangen op de eerste + # versie die ooit werd binnengehaald. + pull_policy: always container_name: byparr network_mode: host restart: unless-stopped diff --git a/renovate.json b/renovate.json index 62760092..958a6344 100644 --- a/renovate.json +++ b/renovate.json @@ -61,7 +61,7 @@ "enabled": false }, { - "description": "De pnpm packageManager-pin is afgestemd op Docker en de runner-toolchain; niet automatisch bumpen", + "description": "De pnpm packageManager-pin blijft handmatig: Docker (corepack) en de runner volgen de pin automatisch, maar een bump raakt ook de lockfile en de README-versietabel (scripts/sync-readme-versions.mjs)", "matchPackageNames": ["pnpm"], "enabled": false } diff --git a/scripts/docker-update.sh b/scripts/docker-update.sh index 28ab3d34..5debf689 100755 --- a/scripts/docker-update.sh +++ b/scripts/docker-update.sh @@ -126,7 +126,9 @@ if [[ -n "${CMS_IMAGE_REPOSITORY:-}" ]]; then docker tag "$app_reference" "epicnext-cms:$CMS_RELEASE" docker tag "$remote_migration_image" "$migration_image" else - docker build --target migrations --build-arg NEXT_DEPLOYMENT_ID="$CMS_RELEASE" -t "$migration_image" . >>"$LOG_FILE" 2>&1 + docker build --target migrations --build-arg NEXT_DEPLOYMENT_ID="$CMS_RELEASE" \ + --cache-from type=local,src=.docker-cache/cms \ + -t "$migration_image" . >>"$LOG_FILE" 2>&1 docker compose build --build-arg NEXT_DEPLOYMENT_ID="$CMS_RELEASE" cms >>"$LOG_FILE" 2>&1 fi expected_image="$(docker image inspect --format '{{.Id}}' "epicnext-cms:$CMS_RELEASE")" diff --git a/src/lib/docker-build-contract.test.ts b/src/lib/docker-build-contract.test.ts index ac90de31..c5800bf5 100644 --- a/src/lib/docker-build-contract.test.ts +++ b/src/lib/docker-build-contract.test.ts @@ -87,3 +87,53 @@ it("does not prerender installation metadata into a shared image", () => { ); } }); + +it("resolves pnpm from the packageManager pin instead of a hardcoded version", () => { + // Een vastgepind pnpm-pakket in de Dockerfile loopt altijd uit sync met de + // packageManager-pin in package.json (12.10.1 vs 12.11.2). + expect(dockerfile).not.toMatch(/npm install -g pnpm@/); + expect(dockerfile).toContain("corepack enable pnpm"); + expect(dockerfile).toContain("corepack install"); + + // De gedownloade pnpm moet in de image staan, anders kan de runtime + // (`docker run --entrypoint pnpm db:migrate`) pnpm niet + // gebruiken. Een cache-mount verdwijnt na de build. + const corepackHome = dockerfile.match(/ENV COREPACK_HOME="([^"]+)"/)?.[1]; + expect(corepackHome).toBeTruthy(); + expect(dockerfile).not.toContain(`--mount=type=cache,target=${corepackHome}`); +}); + +it("caches the pnpm store on the path pnpm actually uses", () => { + const pnpmHome = dockerfile.match(/ENV PNPM_HOME="([^"]+)"/)?.[1]; + expect(pnpmHome).toBeTruthy(); + const cacheMounts = [ + ...dockerfile.matchAll(/--mount=type=cache,target=([^\s\\]+)/g), + ].map((match) => match[1]); + const storeMounts = cacheMounts.filter((target) => target.includes("pnpm")); + expect(storeMounts.length).toBeGreaterThan(0); + // Met een ander pad (bv. /root/.local/share/pnpm/store) cache't de mount + // niets en downloadt elke build alle packages opnieuw. + for (const target of storeMounts) { + expect(target).toBe(`${pnpmHome}/store`); + } +}); + +it("keeps volatile and regenerated files out of the build context", () => { + const ignored = readFileSync(".dockerignore", "utf8"); + // `backups/` (dagelijkse rotatie, ~232MB) maakte elke buildcontext groot + // en invalideerde bij elke rotatie de `COPY . .`-laag. + expect(ignored).toMatch(/^backups\/?$/m); + // public/vendor/tinymce wordt tijdens de build opnieuw gegenereerd door + // `pnpm assets:editor` en staat niet in git. + expect(ignored).toMatch(/^public\/vendor\/?$/m); +}); + +it("keeps a local build cache that survives the daemon cache prune", () => { + const compose = readFileSync("docker-compose.yml", "utf8"); + expect(compose).toContain("cache_from:"); + expect(compose).toContain("cache_to:"); + expect(compose).toContain("type=local"); + // De cache-map mag niet in de buildcontext of in git terechtkomen. + expect(readFileSync(".dockerignore", "utf8")).toMatch(/^\.docker-cache\/?$/m); + expect(readFileSync(".gitignore", "utf8")).toMatch(/^\.docker-cache\/?$/m); +});