perf(build): resolve pnpm from the pin and reuse the real pnpm store
CI / check (push) Successful in 29s
CI / tests-integration (push) Successful in 2m13s
CI / tests-unit (push) Successful in 2m28s
CI / tests-imaging-browsers (push) Successful in 2m29s
CI / tests-quality (push) Successful in 55s
CI / tests-ui (push) Successful in 5m10s
CI / preflight (push) Skipped
CI / deploy (push) Successful in 3m47s
CI / check (push) Successful in 29s
CI / tests-integration (push) Successful in 2m13s
CI / tests-unit (push) Successful in 2m28s
CI / tests-imaging-browsers (push) Successful in 2m29s
CI / tests-quality (push) Successful in 55s
CI / tests-ui (push) Successful in 5m10s
CI / preflight (push) Skipped
CI / deploy (push) Successful in 3m47s
The Docker build was slower than it needed to be and could drift away from the lockfile. Three separate causes, fixed together. The image installed a pinned [email protected].1 while package.json pins 12.11.2 through `packageManager`. pnpm silently self-managed the difference at build time: it downloaded the right version into an uncached location and never verified the sha512 hash. Install corepack instead and let it resolve version and hash from the pin, so the image can no longer diverge and a tampered tarball is rejected. The resolved pnpm lives in COREPACK_HOME inside the image, so the migrations image still runs pnpm offline (`docker run --entrypoint pnpm ... db:migrate`). The pnpm store cache-mount pointed at /root/.local/share/pnpm/store, but pnpm keeps its store in $PNPM_HOME/store/v11. The mount was a no-op: every build re-downloaded all 493 packages ("reused 0, downloaded 493"). Mounting the store where pnpm actually keeps it turns that into "reused 493, downloaded 0". The build context carried backups/ (232MB of rotating database dumps) and public/vendor/ (regenerated by `pnpm assets:editor` from node_modules). Both grew the context and invalidated the `COPY . .` layer on every daily backup rotation, forcing a full Next.js rebuild. Excluding them takes the context from 276MB to 23MB. Compose now also exports the build cache to .docker-cache/cms and imports it again, so the cache survives `scripts/docker-prune.sh`, which trims the daemon cache to 4GB after every deploy. Verified by pruning the entire daemon cache: the next build restored every layer from the local export and downloaded nothing. Also: byparr's floating `:latest` tag is now pulled on every start, the migrations build in scripts/docker-update.sh imports the same local cache, the README no longer claims the build needs host networking or an iptables-disabled daemon, the pnpm rationale in renovate.json reflects that Docker follows the pin automatically, and contract tests pin the new invariants so the store path cannot drift back.
This commit is contained in:
1 parent
1e81cac504
commit
8ca538f99f
8 files changed
+141
-8
No files matched your search
@@ -26,6 +26,18 @@ bun.lockb
|
||||
public/nitro-assets
|
||||
public/swf
|
||||
|
||||
# Databasebackups (~232MB, dagelijkse rotatie): horen niet in de buildcontext.
|
||||
# Ze maakten elke context groot en invalideerden bij elke rotatie de
|
||||
# `COPY . .`-laag, waardoor de Next-build volledig opnieuw draaide.
|
||||
backups/
|
||||
|
||||
# Wordt tijdens de build opnieuw gegenereerd door `pnpm assets:editor`
|
||||
# (bron: node_modules/tinymce) en staat niet in git.
|
||||
public/vendor/
|
||||
|
||||
# Lokale buildcache voor `docker compose build` (cache_from/cache_to).
|
||||
.docker-cache/
|
||||
|
||||
.deploy.lock
|
||||
logs
|
||||
|
||||
|
||||
@@ -57,6 +57,9 @@ blob-report/
|
||||
build-reports/
|
||||
!/docs/performance-budgets.md
|
||||
|
||||
# Local buildcache (docker compose cache_from/cache_to)
|
||||
.docker-cache/
|
||||
|
||||
# One-off local snapshots (o.a. catalog-integrity/pre-fix.json): runtime-werk,
|
||||
# geen bron. Per map opgeslagen om een incident terug te kunnen lezen.
|
||||
backups/
|
||||
+26
-4
@@ -3,21 +3,43 @@ FROM node:26.10.0-alpine AS migrations
|
||||
WORKDIR /app
|
||||
ENV NEXT_TELEMETRY_DISABLED=1
|
||||
|
||||
# Installeer git, bash en pnpm v12. bash is nodig voor
|
||||
# Installeer git, bash en corepack. bash is nodig voor
|
||||
# scripts/with-memory-cap.sh (gebruikt bashisme zoals arrays en BASH_REMATCH);
|
||||
# Alpine levert geen bash mee.
|
||||
#
|
||||
# pnpm komt niet meer uit een losgepind npm-pakket, maar uit corepack: die
|
||||
# leest de versie én de sha512-hash uit de `packageManager`-pin van
|
||||
# package.json. Daarmee kan de image nooit uit sync raken met het lockfile
|
||||
# (vroeger stond hier [email protected] terwijl package.json 12.11.2 voorschreef)
|
||||
# en wordt de gedownloade pnpm-binary ook in de runtime-image bewaard.
|
||||
RUN --mount=type=cache,target=/var/cache/apk \
|
||||
--mount=type=cache,target=/root/.npm \
|
||||
apk add --no-cache git bash \
|
||||
&& npm install -g [email protected]
|
||||
&& npm install -g [email protected] \
|
||||
&& corepack enable pnpm
|
||||
|
||||
# corepack bewaart de gedownloade pnpm buiten de cache-mount, zodat de runtime
|
||||
# (`docker run --entrypoint pnpm <migrations-image> db:migrate`) pnpm ook
|
||||
# zonder netwerk kan gebruiken.
|
||||
ENV COREPACK_HOME="/usr/local/share/corepack"
|
||||
ENV COREPACK_ENABLE_DOWNLOAD_PROMPT=0
|
||||
|
||||
# Stel het PATH zo in dat Alpine pnpm gegarandeerd overal herkent
|
||||
ENV PNPM_HOME="/usr/local/share/pnpm"
|
||||
ENV PATH="$PNPM_HOME:/usr/local/bin:$PATH"
|
||||
|
||||
# Alleen package.json: de `packageManager`-pin bepaalt welke pnpm corepack
|
||||
# installeert. Deze laag blijft gecachet wanneer alleen de lockfile verandert.
|
||||
COPY package.json ./
|
||||
RUN corepack install
|
||||
|
||||
COPY package.json pnpm-lock.yaml* pnpm-workspace.yaml* .npmrc* ./
|
||||
|
||||
# Voer de installatie uit met de pnpm v12 store cache-mount
|
||||
RUN --mount=type=cache,target=/root/.local/share/pnpm/store \
|
||||
# Voer de installatie uit met de pnpm v12 store cache-mount. De mount moet op
|
||||
# de store staan die pnpm ook echt gebruikt ($PNPM_HOME/store/v11); met het
|
||||
# oude pad (/root/.local/share/pnpm/store) cache'te de mount niets en
|
||||
# downloadde elke build alle packages opnieuw.
|
||||
RUN --mount=type=cache,target=/usr/local/share/pnpm/store \
|
||||
pnpm install --frozen-lockfile --ignore-scripts
|
||||
|
||||
COPY . .
|
||||
|
||||
@@ -307,6 +307,28 @@ responsibility. Docker uses the committed pnpm lockfile and pinned Node version.
|
||||
The build cache can stay enabled: copying changed source invalidates the
|
||||
application build layer. Deleting all Docker cache is not an update mechanism.
|
||||
|
||||
#### Build toolchain and cache
|
||||
|
||||
The image never installs a pinned pnpm of its own. The Dockerfile installs
|
||||
[corepack](https://nodejs.org/api/corepack.html) and lets it resolve the pnpm
|
||||
version *and* its sha256/sha512 hash from the `packageManager` field in
|
||||
`package.json`. One pin therefore drives the Docker build, the CI runner and
|
||||
the host — the image can no longer drift away from the lockfile, and a tampered
|
||||
package tarball is rejected instead of silently installed. Bumping pnpm is a
|
||||
single edit in `package.json` (plus the README version table, checked by
|
||||
`pnpm readme:versions:check`).
|
||||
|
||||
Two caches make repeated builds cheap:
|
||||
|
||||
- **pnpm store** — a BuildKit cache mount on `$PNPM_HOME/store` keeps the
|
||||
downloaded packages between builds. A lockfile change then re-uses the store
|
||||
(`reused 493, downloaded 0`) instead of re-downloading every package.
|
||||
- **Local build cache** — `docker compose build` exports the build cache to
|
||||
`.docker-cache/cms` and imports it again on the next build. Unlike the
|
||||
daemon's own cache, this survives `scripts/docker-prune.sh`, which trims the
|
||||
daemon cache to 4 GB after every deploy. The directory is ignored by both
|
||||
Git and the build context; delete it to force a cold build.
|
||||
|
||||
### Volumes
|
||||
|
||||
Only the CMS (and the optional avatar imager container, see [Avatar Imaging](#avatar-imaging)) run in Docker. The heavy client assets and gamedata stay on the host and are shared into the container so imports and uploads persist:
|
||||
@@ -346,7 +368,13 @@ docker compose build --build-arg RUN_USER=1000
|
||||
|
||||
The CMS container runs with `network_mode: host`. This lets the existing `127.0.0.1` references in `.env` keep working against host services — MariaDB (`3306`), Valkey/Redis (`6379`), the emulator RCON/API (`3003`/`3001`) and the avatar imager — without re-writing any environment variables. The container consequently listens **directly on the host's port 3002**, so stop any previous host-side CMS (e.g. `pm2 stop next`) that occupies that port before starting it.
|
||||
|
||||
The **build** also runs on the host network (`build.network: host`) because this host disables Docker iptables (`/etc/docker/daemon.json`: `"iptables": false`), which would otherwise leave build containers without outbound NAT/DNS when fetching packages.
|
||||
The **build** deliberately does *not* use host networking. BuildKit (v0.26, Docker 29)
|
||||
refuses to grant host networking unless every caller passes
|
||||
`--allow=network.host`, and `docker compose build` has no such flag — asking for
|
||||
`build.network: host` therefore turned every rebuild into an immediate
|
||||
"additional privileges requested" failure. The build only needs outbound
|
||||
internet (apk, corepack/pnpm, next/font/google), which the default bridge
|
||||
provides.
|
||||
|
||||
### Health Check
|
||||
|
||||
@@ -595,7 +623,7 @@ URLs look like `https://<hotel>/imaging/avatarimage?figure=hd-180-1.ch-210-66&im
|
||||
|
||||
### Requirements (host)
|
||||
|
||||
- Docker (daemon with `build.network: host`, same as the CMS build — this host disables Docker iptables).
|
||||
- Docker (daemon with BuildKit; the CMS build runs on the default bridge, not on host networking — see [Networking](#networking)).
|
||||
- An nginx that serves `/gamedata/` (FigureData/FigureMap/EffectMap…) and `/gamedata/bundled` (furniture bundle assets — `.hab` by default, the extension imports write and this deployment's client requests, plus any legacy `.nitro` still on disk; choose the written extension with `FURNI_BUNDLE_EXT`) over HTTP so the renderer can fetch them. The compose file points at `host.docker.internal:8081`.
|
||||
|
||||
### Configuration — `/docker/Polaris-imager/.env`
|
||||
|
||||
@@ -29,6 +29,18 @@ x-cms: &cms
|
||||
dockerfile: Dockerfile
|
||||
args:
|
||||
NEXT_DEPLOYMENT_ID: ${CMS_RELEASE:-unknown}
|
||||
# Lokale buildcache op schijf. `scripts/docker-prune.sh` brengt de
|
||||
# daemon-cache na elke deploy terug tot 4GB, waardoor de lagen van de
|
||||
# vorige release verdwijnen. Deze export overleeft die prune: een rebuild
|
||||
# na een lockfile-wijziging hergebruikt dan de al gedownloade packages
|
||||
# (pnpm store) in plaats van ze opnieuw te downloaden.
|
||||
# De map ligt binnen de buildcontext, zodat buildx een ontbrekende
|
||||
# cachemap stilzwijgend negeert i.p.v. de build te laten falen; zowel
|
||||
# .dockerignore als .gitignore sluiten de map uit.
|
||||
cache_from:
|
||||
- type=local,src=.docker-cache/cms
|
||||
cache_to:
|
||||
- type=local,dest=.docker-cache/cms,mode=max
|
||||
# Runtime host networking IS required: nginx reaches the CMS over 127.0.0.1
|
||||
# on the single port that is actually served (3002).
|
||||
network_mode: host
|
||||
@@ -67,6 +79,10 @@ services:
|
||||
|
||||
byparr:
|
||||
image: ghcr.io/thephaseless/byparr:latest
|
||||
# `latest` is een drijvende tag. Zonder pull_policy haalt compose de
|
||||
# bestaande image nooit opnieuw op en blijft byparr hangen op de eerste
|
||||
# versie die ooit werd binnengehaald.
|
||||
pull_policy: always
|
||||
container_name: byparr
|
||||
network_mode: host
|
||||
restart: unless-stopped
|
||||
|
||||
+1
-1
@@ -61,7 +61,7 @@
|
||||
"enabled": false
|
||||
},
|
||||
{
|
||||
"description": "De pnpm packageManager-pin is afgestemd op Docker en de runner-toolchain; niet automatisch bumpen",
|
||||
"description": "De pnpm packageManager-pin blijft handmatig: Docker (corepack) en de runner volgen de pin automatisch, maar een bump raakt ook de lockfile en de README-versietabel (scripts/sync-readme-versions.mjs)",
|
||||
"matchPackageNames": ["pnpm"],
|
||||
"enabled": false
|
||||
}
|
||||
|
||||
@@ -126,7 +126,9 @@ if [[ -n "${CMS_IMAGE_REPOSITORY:-}" ]]; then
|
||||
docker tag "$app_reference" "epicnext-cms:$CMS_RELEASE"
|
||||
docker tag "$remote_migration_image" "$migration_image"
|
||||
else
|
||||
docker build --target migrations --build-arg NEXT_DEPLOYMENT_ID="$CMS_RELEASE" -t "$migration_image" . >>"$LOG_FILE" 2>&1
|
||||
docker build --target migrations --build-arg NEXT_DEPLOYMENT_ID="$CMS_RELEASE" \
|
||||
--cache-from type=local,src=.docker-cache/cms \
|
||||
-t "$migration_image" . >>"$LOG_FILE" 2>&1
|
||||
docker compose build --build-arg NEXT_DEPLOYMENT_ID="$CMS_RELEASE" cms >>"$LOG_FILE" 2>&1
|
||||
fi
|
||||
expected_image="$(docker image inspect --format '{{.Id}}' "epicnext-cms:$CMS_RELEASE")"
|
||||
|
||||
@@ -87,3 +87,53 @@ it("does not prerender installation metadata into a shared image", () => {
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("resolves pnpm from the packageManager pin instead of a hardcoded version", () => {
|
||||
// Een vastgepind pnpm-pakket in de Dockerfile loopt altijd uit sync met de
|
||||
// packageManager-pin in package.json (12.10.1 vs 12.11.2).
|
||||
expect(dockerfile).not.toMatch(/npm install -g pnpm@/);
|
||||
expect(dockerfile).toContain("corepack enable pnpm");
|
||||
expect(dockerfile).toContain("corepack install");
|
||||
|
||||
// De gedownloade pnpm moet in de image staan, anders kan de runtime
|
||||
// (`docker run --entrypoint pnpm <migrations-image> db:migrate`) pnpm niet
|
||||
// gebruiken. Een cache-mount verdwijnt na de build.
|
||||
const corepackHome = dockerfile.match(/ENV COREPACK_HOME="([^"]+)"/)?.[1];
|
||||
expect(corepackHome).toBeTruthy();
|
||||
expect(dockerfile).not.toContain(`--mount=type=cache,target=${corepackHome}`);
|
||||
});
|
||||
|
||||
it("caches the pnpm store on the path pnpm actually uses", () => {
|
||||
const pnpmHome = dockerfile.match(/ENV PNPM_HOME="([^"]+)"/)?.[1];
|
||||
expect(pnpmHome).toBeTruthy();
|
||||
const cacheMounts = [
|
||||
...dockerfile.matchAll(/--mount=type=cache,target=([^\s\\]+)/g),
|
||||
].map((match) => match[1]);
|
||||
const storeMounts = cacheMounts.filter((target) => target.includes("pnpm"));
|
||||
expect(storeMounts.length).toBeGreaterThan(0);
|
||||
// Met een ander pad (bv. /root/.local/share/pnpm/store) cache't de mount
|
||||
// niets en downloadt elke build alle packages opnieuw.
|
||||
for (const target of storeMounts) {
|
||||
expect(target).toBe(`${pnpmHome}/store`);
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps volatile and regenerated files out of the build context", () => {
|
||||
const ignored = readFileSync(".dockerignore", "utf8");
|
||||
// `backups/` (dagelijkse rotatie, ~232MB) maakte elke buildcontext groot
|
||||
// en invalideerde bij elke rotatie de `COPY . .`-laag.
|
||||
expect(ignored).toMatch(/^backups\/?$/m);
|
||||
// public/vendor/tinymce wordt tijdens de build opnieuw gegenereerd door
|
||||
// `pnpm assets:editor` en staat niet in git.
|
||||
expect(ignored).toMatch(/^public\/vendor\/?$/m);
|
||||
});
|
||||
|
||||
it("keeps a local build cache that survives the daemon cache prune", () => {
|
||||
const compose = readFileSync("docker-compose.yml", "utf8");
|
||||
expect(compose).toContain("cache_from:");
|
||||
expect(compose).toContain("cache_to:");
|
||||
expect(compose).toContain("type=local");
|
||||
// De cache-map mag niet in de buildcontext of in git terechtkomen.
|
||||
expect(readFileSync(".dockerignore", "utf8")).toMatch(/^\.docker-cache\/?$/m);
|
||||
expect(readFileSync(".gitignore", "utf8")).toMatch(/^\.docker-cache\/?$/m);
|
||||
});
|
||||
Reference in new issue
Block a user