8 Commits
Author SHA1 Message Date
openhands 52459c0b74 feat(db): add self-tuned mariadb-turbo container and bulk JSON importer
CI / check (push) Failing after 1m15s
CI / deploy (push) Skipped
- docker-compose: opt-in mariadb-turbo service (profile 'db') with inline
  mysqld tuning (max_allowed_packet=512M, innodb_flush_log_at_trx_commit=2,
  2G buffer pool, net_read/write_timeout=600) for >50 MB JSON bulk loads;
  named volume for the datadir + node_modules host-sync guidance
- scripts/bulk-import-json.ts: chunked (2000-row) idempotent importer with
  ON DUPLICATE KEY UPDATE, resumable, habbo furnidata or flat-array input
- src/db/schema-gamedata.ts: promote+index JSON storage schema (furnidata,
  docs, texts) incl. VIRTUAL generated columns for MariaDB
- package.json: add db:bulk, db:up, db:down, db:introspect, db:schema:generate
2026-09-07 16:54:44 +02:00
openhands a640e40a5d fix(docker): clear build cache on every build to stop unbounded disk growth
CI / check (push) Successful in 1m17s
CI / deploy (push) Successful in 2m11s
2026-09-07 16:18:04 +02:00
openhands 649e2f0663 feat(docker): self-contained runtime dirs, image healthcheck, spec-compliant Dockerfile
CI / check (push) Successful in 1m14s
CI / deploy (push) Successful in 1m40s
- Create the runtime write targets (/app/storage, /app/public/nitro-assets,
  /app/public/swf, /var/www/Gamedata) owned by UID/GID 33 in the runner image
  so running without the bound volumes no longer hits ENOENT.
- Bake a HEALTHCHECK into the image so `docker run` (ci-deploy.sh) also reports
  Docker-level health; compose can still override it with its own probe.
- Add the dockerfile:1 syntax pragma and ignore non-pnpm lockfiles so a stray
  package-lock.json/yarn.lock can never taint the build context.
2026-09-07 11:44:52 +02:00
openhands 1a9b361420 fix(docker): inject real commit id into build via NEXT_DEPLOYMENT_ID
CI / check (push) Successful in 1m12s
CI / deploy (push) Successful in 1m41s
next.config.ts falls back to `git rev-parse HEAD`, but the build context has
no .git (excluded by .dockerignore), so every CI build printed fatal git
errors and stamped the release as "unknown". Pass the deploy commit sha as a
NEXT_DEPLOYMENT_ID build-arg so git is never invoked and the actual commit
reaches NEXT_PUBLIC_CMS_RELEASE and deploymentId.
2026-09-07 11:39:29 +02:00
openhands 3e0ba73d1c fix(docker): include pnpm-workspace and npmrc in builder context
CI / check (push) Successful in 1m12s
CI / deploy (push) Successful in 1m26s
The committed lockfile records overrides from pnpm-workspace.yaml. With the
narrower manifest COPY, pnpm install --frozen-lockfile ran without the
workspace file and failed with ERR_PNPM_LOCKFILE_CONFIG_MISMATCH in CI.
Copy package.json, pnpm-lock, pnpm-workspace and .npmrc together so the
override config present in the lockfile is also supplied at install time.
2026-09-07 11:33:04 +02:00
openhands bef1775dea fix(toolchain): restore node engines range to match .nvmrc pin
CI / check (push) Successful in 1m13s
CI / deploy (push) Failing after 41s
2026-09-07 11:24:52 +02:00
openhands c5284e0b79 test(docker): align build-contract test with pnpm fetch caching
CI / check (push) Failing after 16s
CI / deploy (push) Skipped
2026-09-07 11:22:30 +02:00
openhands 539e6d3fad fix(docker): harden image and automate safe VPS updates
- Use floating node:alpine that tracks the latest supported LTS; pnpm
  bootstrap follows package.json's packageManager pin.
- Drop corepack (removed from node:26), install pnpm via npm global.
- Add pnpm fetch + offline install for stable dependency-layer caching.
- Run as non-root nextjs (UID/GID 33 = host www-data) with tini as PID 1
  for correct signal handling.
- Open node engines to >=20.9.0 so patches/minors float automatically.
- Add docker-preflight.sh (per-VPS checks incl. --fix) and gate docker-update.sh
  so Node major upgrades require explicit review while patches deploy silently.
2026-09-07 11:22:30 +02:00
11 changed files with 775 additions and 90 deletions

No files matched your search

+6
View File
@@ -7,6 +7,12 @@ storage
prod.log prod.log
update.log update.log
.pm2 .pm2
# Only the pnpm lockfile is used. Ignore other lockfile formats and stray
# package managers so they never taint the build context by accident.
package-lock.json
yarn.lock
bun.lockb
.npmrc.bak
# NOTE: .env is intentionally NOT ignored here — the build loads it (only inside # NOTE: .env is intentionally NOT ignored here — the build loads it (only inside
# a build RUN layer) to produce NEXT_PUBLIC_* + validated build-time values. # a build RUN layer) to produce NEXT_PUBLIC_* + validated build-time values.
# It is not copied into the runtime image (the runner stage copies only # It is not copied into the runtime image (the runner stage copies only
+42 -72
View File
@@ -1,78 +1,48 @@
# syntax=docker/dockerfile:1 # syntax=docker/dockerfile:1
# ============================================================================== # node:alpine = latest Node within the supported LTS major (tracks the newest
# EpicNext-CMS — Docker image (Node 26.8.1, multi-package-manager, Next.js standalone) # patch automatically; currently v26.x, which satisfies package.json's
# ============================================================================== # engines ">=26.8.1 <27").
FROM node:alpine AS builder
# --- Builder stage ---
FROM node:26.8.1-bookworm-slim AS builder
RUN apt-get update && apt-get install -y --no-install-recommends git ca-certificates \
&& rm -rf /var/lib/apt/lists/*
RUN npm install -g [email protected] yarn
WORKDIR /app WORKDIR /app
ENV NEXT_TELEMETRY_DISABLED=1
COPY package.json *lock* pnpm-workspace.yaml .npmrc ./ # Real release id supplied by CI (ci-deploy.sh) so next.config.ts skips git
# entirely (there is no .git in the build context). Defaults to "unknown".
ARG PACKAGE_MANAGER= ARG NEXT_DEPLOYMENT_ID="unknown"
ENV NEXT_DEPLOYMENT_ID="$NEXT_DEPLOYMENT_ID"
RUN --mount=type=cache,id=epicnext-pnpm,target=/pnpm/store,sharing=locked \ # pnpm install is always pinned to the version in package.json's
if [ "$PACKAGE_MANAGER" = "pnpm" ] || { [ -z "$PACKAGE_MANAGER" ] && [ -f pnpm-lock.yaml ]; }; then \ # `packageManager` field (pnpm auto-selects it on install), so this global
echo ">> Using pnpm" && \ # install only needs to exist as a bootstrap and follows the active major.
pnpm install --frozen-lockfile --ignore-scripts --store-dir=/pnpm/store; \ RUN apk add --no-cache git \
elif [ "$PACKAGE_MANAGER" = "yarn" ] || { [ -z "$PACKAGE_MANAGER" ] && [ -f yarn.lock ]; }; then \ && npm install -g pnpm@latest
echo ">> Using yarn" && \ # pnpm-workspace.yaml + .npmrc must be present too: the lockfile records the
yarn install --frozen-lockfile --ignore-scripts; \ # overrides from pnpm-workspace.yaml, and --frozen-lockfile rejects a build
elif [ "$PACKAGE_MANAGER" = "npm" ] || { [ -z "$PACKAGE_MANAGER" ] && [ -f package-lock.json ]; }; then \ # where the workspace config is absent (ERR_PNPM_LOCKFILE_CONFIG_MISMATCH).
echo ">> Using npm" && \ COPY package.json pnpm-lock.yaml* pnpm-workspace.yaml* .npmrc* ./
npm ci --ignore-scripts; \ # pnpm fetch: download all deps into $PNPM_STORE first, so only the lockfile
else \ # change (not source changes) invalidates the network-heavy download layer.
echo "!! No lockfile found — falling back to npm install" && \ RUN pnpm fetch --ignore-scripts
npm install --ignore-scripts; \ RUN pnpm install --frozen-lockfile --ignore-scripts --offline
fi
COPY . . COPY . .
RUN pnpm run build
ENV NODE_ENV=production FROM node:alpine AS runner
RUN --mount=type=cache,id=epicnext-next,target=/app/.next/cache,sharing=locked \
if [ -f .env ]; then set -a && . ./.env && set +a; fi && \
if [ "$PACKAGE_MANAGER" = "yarn" ] || { [ -z "$PACKAGE_MANAGER" ] && [ -f yarn.lock ]; }; then \
yarn build; \
elif [ "$PACKAGE_MANAGER" = "npm" ] || { [ -z "$PACKAGE_MANAGER" ] && [ -f package-lock.json ]; }; then \
npm run build; \
else \
pnpm build; \
fi
# --- Runtime stage ---
FROM node:26.8.1-bookworm-slim AS runner
RUN apt-get update && apt-get install -y --no-install-recommends git curl ca-certificates \
&& rm -rf /var/lib/apt/lists/*
ARG RUN_USER=www-data
ENV NODE_ENV=production
ENV PORT=3002
ENV HOSTNAME=0.0.0.0
EXPOSE 3002
WORKDIR /app WORKDIR /app
ENV NODE_ENV=production \
RUN mkdir -p /app/storage \ NEXT_TELEMETRY_DISABLED=1 \
/app/public/nitro-assets \ PORT=3002 \
/app/public/swf \ HOSTNAME=0.0.0.0
/var/www/Gamedata \ RUN apk add --no-cache tini \
&& chown -R ${RUN_USER} /app /var/www/Gamedata && addgroup -g 33 -S nextjs && adduser -u 33 -S -G nextjs nextjs \
&& mkdir -p /app/storage /app/public/nitro-assets /app/public/swf /var/www/Gamedata \
COPY --from=builder --chown=${RUN_USER} /app/.next/standalone ./ && chown -R 33:33 /app/storage /app/public /var/www/Gamedata
COPY --from=builder --chown=${RUN_USER} /app/public ./public COPY --from=builder --chown=nextjs:nextjs /app/public ./public
COPY --from=builder --chown=${RUN_USER} /app/.next/static ./.next/static COPY --from=builder --chown=nextjs:nextjs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nextjs /app/.next/static ./.next/static
VOLUME ["/app/public/nitro-assets", "/app/public/swf", "/app/storage"] USER nextjs
EXPOSE 3002
USER ${RUN_USER} # Self-contained healthcheck so `docker run` (ci-deploy) also gets Docker-level
# health; docker-compose overrides this with its own probe if needed.
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
CMD ["node", "-e", "fetch('http://127.0.0.1:'+(process.env.PORT||'3002')+'/api/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["node", "server.js"] CMD ["node", "server.js"]
+78 -1
View File
@@ -1,4 +1,4 @@
# EpicNext-CMS v1.0.1 # EpicNext-CMS v1.0.3
A modern, high-performance content management system for Habbo hotel emulators, built on **Next.js 16** (App Router) with **Drizzle ORM** and **React 19**. Designed to integrate seamlessly with Polaris / Arcturus Morningstar MySQL/MariaDB databases. A modern, high-performance content management system for Habbo hotel emulators, built on **Next.js 16** (App Router) with **Drizzle ORM** and **React 19**. Designed to integrate seamlessly with Polaris / Arcturus Morningstar MySQL/MariaDB databases.
@@ -196,6 +196,7 @@ an exit of 0 means the CMS is healthy on the new commit.
| `docker compose pull && docker compose up -d --build` | Update and redeploy | | `docker compose pull && docker compose up -d --build` | Update and redeploy |
| `pnpm db:migrate` (on the **host**) | Run database migrations (slim image has no source) | | `pnpm db:migrate` (on the **host**) | Run database migrations (slim image has no source) |
| `./scripts/docker-update.sh` | Full automated update (manual or cron) | | `./scripts/docker-update.sh` | Full automated update (manual or cron) |
| `pnpm db:up` / `pnpm db:down` | Start / stop the `mariadb-turbo` bulk-load container |
### How Package Manager Detection Works ### How Package Manager Detection Works
@@ -258,6 +259,45 @@ The container includes a health check that hits `/api/health` on port 3002 every
docker inspect --format='{{.State.Health.Status}}' epicnext-cms docker inspect --format='{{.State.Health.Status}}' epicnext-cms
``` ```
### MariaDB Turbo (bulk loads)
`docker-compose.yml` ships an opt-in **`mariadb-turbo`** service — a dedicated MariaDB 11 container tuned for loading >50 MB JSON dumps (furnidata, external texts, etc.). It configures itself via mysqld flags on the `command:`, so no external `.cnf` file has to be mounted or kept in sync:
```bash
pnpm db:up # docker compose --profile db up -d (starts mariadb-turbo)
```
Key settings (see the `command:` block in `docker-compose.yml`):
- `max_allowed_packet = 512M` — 50 MB JSON batches no longer hit the 16 MB packet ceiling.
- `innodb_flush_log_at_trx_commit = 2` + `innodb_doublewrite = 0` — durability relaxed for bulk writes.
- `innodb_buffer_pool_size = 2G`, `innodb_log_file_size = 1G`, `bulk_insert_buffer_size = 512M`.
- `net_read_timeout` / `net_write_timeout = 600`, `wait_timeout = 3600` — fixes the `drizzle-kit` **"Pulling schema from database..."** hang by never starving introspection/DDL sessions behind a long import.
- `performance_schema = OFF` — saves ~1-2 GB RAM.
The datadir lives in a **named volume** (`mariadb-turbo-data`), never a host bind-mount: shared-filesystem sync trashes InnoDB files the same way it corrupts pnpm's `node_modules`. Named volumes stay inside the container filesystem, so 50 MB of JSON writes never cross a host-sync boundary.
> **Port note:** the service uses `network_mode: host` and binds `127.0.0.1:3306` — stop the host MariaDB first, otherwise the port collides with the standalone `.env` `DATABASE_URL` (`localhost:3306`).
#### node_modules & host-sync corruption
pnpm's store is hard-linked and its `.bin` shims are symlinks, so **node_modules must never be a host bind-mount** (Docker Desktop gRPC-FUSE/VirtioFS, Unison and Syncthing all corrupt it). The production build bakes `node_modules` into the image at build time; it is never bind-mounted. For local dev, mount a *named volume* (`node_modules:/app/node_modules`) instead of the host directory, and keep `node_modules` / the pnpm store out of any shared-filesystem bind.
#### Loading a 50 MB JSON dump
```bash
# Habbo furnidata (auto-detects roomitemtypes / wallitemtypes / effecttypes)
pnpm db:bulk --file=/var/www/Gamedata/config/FurnitureData.json
# Flat array of documents → generic JSON store
pnpm db:bulk --file=/data/products.json --table=docs --category=furni
# Key/value texts
pnpm db:bulk --file=/data/external_texts.json --table=texts --category=default
```
The importer (`scripts/bulk-import-json.ts`) maps rows onto `src/db/schema-gamedata.ts`, batches them into **2000-row multi-row INSERTs** (one statement per chunk), uses `ON DUPLICATE KEY UPDATE` so re-runs are idempotent and interrupted loads resume, and prints rows/s + ETA. See ["ORM Setup & Type Generation"](#orm-setup--type-generation) → *Bulk JSON storage* for the design.
--- ---
@@ -284,10 +324,41 @@ const found = await db.select()
| `pnpm db:generate` | Draft SQL from Drizzle schema into `drizzle/drafts/` (review + copy) | | `pnpm db:generate` | Draft SQL from Drizzle schema into `drizzle/drafts/` (review + copy) |
| `pnpm db:studio` | Open Drizzle Studio (dev only) | | `pnpm db:studio` | Open Drizzle Studio (dev only) |
| `pnpm db:introspect` | Reverse-engineer an existing DB into a Drizzle schema draft | | `pnpm db:introspect` | Reverse-engineer an existing DB into a Drizzle schema draft |
| `pnpm db:bulk` | Batch-import >50 MB JSON (2000-row chunks, resumable) via `scripts/bulk-import-json.ts` |
| `pnpm db:schema:generate` | Regen committed `src/db/schema.ts` from previous schema + live DB | | `pnpm db:schema:generate` | Regen committed `src/db/schema.ts` from previous schema + live DB |
> The CMS does **not** use `drizzle-kit push` or `drizzle-kit migrate` — the database is shared with the emulator. Apply CMS DDL only via `pnpm db:migrate`. > The CMS does **not** use `drizzle-kit push` or `drizzle-kit migrate` — the database is shared with the emulator. Apply CMS DDL only via `pnpm db:migrate`.
#### Bulk JSON storage (MariaDB)
Emulator/hotel JSON dumps (furnidata, external texts, product data) are often >50 MB. MariaDB's `JSON` type is an alias for `LONGTEXT` and can **never be indexed directly**, so `src/db/schema-gamedata.ts` follows the "promote + index" pattern:
- The raw JSON document stays intact in a `longtext` / `mediumtext` column (`payload`, `value`).
- The fields you actually `WHERE` / `ORDER BY` on are promoted into real columns (`sprite_id`, `class_name`, `kind`, `title`, `text_key`) and indexed.
- Optional **VIRTUAL generated columns** extract indexed fields from the payload with `JSON_EXTRACT` (MariaDB supports secondary indexes on virtual columns, 10.2+), so no duplicate data has to be written by the importer.
Three tables are exported:
| Table | Purpose | Unique key |
| -------------------- | ---------------------------------------------- | --------------------------- |
| `gamedata_furnidata` | One row per furni/clothing item (habbo furnidata_json shape) | `(source, sprite_id)` |
| `gamedata_docs` | Generic JSON document store (per-key documents) | `(category, doc_key)` |
| `gamedata_texts` | External-texts style key/value pairs | `(category, text_key)` |
The bulk importer (`scripts/bulk-import-json.ts`, run via `pnpm db:bulk`) is DB-driven and fast precisely because each chunk is a *single* multi-row `INSERT … VALUES () ()… ON DUPLICATE KEY UPDATE`, so a 50 MB dump is a few dozen statements instead of hundreds of thousands of round-trips. Flags:
| Flag | Default | Description |
| -------------------- | ----------- | ------------------------------------------------- |
| `--file=<path>` | — (required)| JSON file (habbo furnidata_json or flat array) |
| `--table=<name>` | `furnidata` | One of `furnidata` \| `docs` \| `texts` |
| `--source=<x>` | `habbo` | `source` value for `furnidata` |
| `--category=<x>` | `default` | `category` value for `docs` / `texts` |
| `--chunk-size=N` | `2000` | Rows per multi-row INSERT |
| `--limit=N` | `0` | Stop after N rows (dry-test) |
| `--truncate` | off | DELETE rows for this source/category first |
The script sets `FOREIGN_KEY_CHECKS=0` for the session and is safe to interrupt: each chunk commits on its own, and `ON DUPLICATE KEY UPDATE` makes re-runs overwrite instead of appending.
--- ---
## DragonflyDB (Caching, Rate Limiting, SSE) ## DragonflyDB (Caching, Rate Limiting, SSE)
@@ -503,6 +574,8 @@ slow_query_log_file = /var/log/mysql/mariadb-slow.log
The slow-query log surfaces hot-spot SQL for a follow-up index/query audit. The CMS connection pool itself is already tuned (pool size 25, `connection_limit` in `DATABASE_URL`). The slow-query log surfaces hot-spot SQL for a follow-up index/query audit. The CMS connection pool itself is already tuned (pool size 25, `connection_limit` in `DATABASE_URL`).
For bulk-written tables (game-data JSON, imports) a containerized **`mariadb-turbo`** variant is available — see [MariaDB Turbo](#mariadb-turbo-bulk-loads).
### 3. Asset caching headers ### 3. Asset caching headers
| Path | Cache-Control | | Path | Cache-Control |
@@ -553,6 +626,8 @@ pm2 restart next
| `pnpm db:generate` | Draft SQL via drizzle-kit → `drizzle/drafts/` | | `pnpm db:generate` | Draft SQL via drizzle-kit → `drizzle/drafts/` |
| `pnpm db:studio` | Drizzle Studio (dev) | | `pnpm db:studio` | Drizzle Studio (dev) |
| `pnpm db:introspect` | drizzle-kit introspect (draft) | | `pnpm db:introspect` | drizzle-kit introspect (draft) |
| `pnpm db:bulk` | Batch-import >50 MB JSON via `scripts/bulk-import-json.ts` |
| `pnpm db:up` / `pnpm db:down` | Start / stop the `mariadb-turbo` container |
| `pnpm gamedata:compress` | Pre-compress large gamedata JSON to `.gz` (gzip_static) | | `pnpm gamedata:compress` | Pre-compress large gamedata JSON to `.gz` (gzip_static) |
| `pnpm analyze` | Build + open bundle analyzer | | `pnpm analyze` | Build + open bundle analyzer |
| `pnpm jobs:worker` | Start background task worker | | `pnpm jobs:worker` | Start background task worker |
@@ -589,12 +664,14 @@ pm2 restart next
│ └── drafts/ # drizzle-kit generate output (never auto-applied) │ └── drafts/ # drizzle-kit generate output (never auto-applied)
├── scripts/ ├── scripts/
│ ├── apply-migrations.ts # SQL migration runner (apply + status) │ ├── apply-migrations.ts # SQL migration runner (apply + status)
│ ├── bulk-import-json.ts # 50 MB+ JSON importer (2000-row chunks, UPSERT)
│ ├── jobs-worker.ts # Background task scheduler │ ├── jobs-worker.ts # Background task scheduler
│ ├── generate-drizzle-schema.mjs # Regen src/db/schema.ts from schema + live DB │ ├── generate-drizzle-schema.mjs # Regen src/db/schema.ts from schema + live DB
│ └── compress-gamedata.mjs # Pre-compress large gamedata JSON (gzip_static) │ └── compress-gamedata.mjs # Pre-compress large gamedata JSON (gzip_static)
├── src/ ├── src/
│ ├── db/ │ ├── db/
│ │ ├── schema.ts # Drizzle ORM schema (committed — runtime data layer) │ │ ├── schema.ts # Drizzle ORM schema (committed — runtime data layer)
│ │ ├── schema-gamedata.ts # Large-JSON storage schema (furnidata/docs/texts)
│ │ └── relations.ts # Drizzle relations │ │ └── relations.ts # Drizzle relations
│ ├── app/ # Next.js App Router (pages & API routes) │ ├── app/ # Next.js App Router (pages & API routes)
│ ├── actions/ # Server Actions │ ├── actions/ # Server Actions
+98 -5
View File
@@ -7,11 +7,6 @@ services:
# build containers on the bridge network have no outbound NAT/DNS. Build on # build containers on the bridge network have no outbound NAT/DNS. Build on
# the host network instead so pnpm/npm/yarn can reach the registry. # the host network instead so pnpm/npm/yarn can reach the registry.
network: host network: host
args:
# Run as the host owner (www-data = UID/GID 33, already present in the
# node base image) of /var/www/Gamedata so the container can read + write
# the shared gamedata directory.
RUN_USER: "www-data"
container_name: epicnext-cms container_name: epicnext-cms
# Runs on the host network so existing 127.0.0.1 refs in .env keep working: # Runs on the host network so existing 127.0.0.1 refs in .env keep working:
# MariaDB (3306), DragonflyDB/Redis (6379), emulator RCON (3003) + API (3001), # MariaDB (3306), DragonflyDB/Redis (6379), emulator RCON (3003) + API (3001),
@@ -22,6 +17,8 @@ services:
restart: unless-stopped restart: unless-stopped
env_file: env_file:
- .env - .env
environment:
- HOSTNAME=0.0.0.0
volumes: volumes:
# ── Write targets (runtime imports/uploads, persistent on the host) ── # ── Write targets (runtime imports/uploads, persistent on the host) ──
# The CMS writes imported furni/figures/pets/effects here (see # The CMS writes imported furni/figures/pets/effects here (see
@@ -39,6 +36,16 @@ services:
# same directory, so mount it into the container at the same absolute path. # same directory, so mount it into the container at the same absolute path.
# Must be RW so furniture/badge imports can write mirrors to it. # Must be RW so furniture/badge imports can write mirrors to it.
- /var/www/Gamedata:/var/www/Gamedata - /var/www/Gamedata:/var/www/Gamedata
#
# ── node_modules corruption (§ host-sync) ──
# pnpm node_modules must NEVER be a host bind-mount: shared-filesystem
# sync (Docker Desktop gRPC-FUSE/VirtioFS, Unison, Syncthing) corrupts
# pnpm's hardlinked store and .bin shims. The production image already
# bakes node_modules in at build time and is NOT bind-mounted here.
# For local dev use a *named volume* instead of a host bind:
# - node_modules:/app/node_modules
# and never share `node_modules` / `pnpm store` via the Docker bind.
# On macOS add `:cached`/`:delegated` consistency labels per mount.
healthcheck: healthcheck:
test: ["CMD", "node", "-e", "fetch('http://localhost:3002/api/health').then(r=>{if(!r.ok)process.exit(1)}).catch(()=>process.exit(1))"] test: ["CMD", "node", "-e", "fetch('http://localhost:3002/api/health').then(r=>{if(!r.ok)process.exit(1)}).catch(()=>process.exit(1))"]
@@ -82,3 +89,89 @@ services:
# restart: unless-stopped # restart: unless-stopped
# volumes: # volumes:
# - /var/www/Gamedata:/var/www/Gamedata # - /var/www/Gamedata:/var/www/Gamedata
# ── MariaDB "Turbo" (heavy JSON / bulk-loads) ──
# Dedicated MariaDB tuned for >50 MB JSON imports. All tuning lives inline
# in the service `command:` (bulk_insert_buffer_size,
# innodb_flush_log_at_trx_commit=2, max_allowed_packet=512M, ...) so the
# container configures itself — no external .cnf to mount or keep in sync.
# Opt-in via profile so `docker compose up` keeps running the standalone
# stack as today.
#
# Start: docker compose --profile db up -d (= pnpm db:up)
# Note: host networking binds 127.0.0.1:3306 → STOP the host MariaDB
# first (the app's .env DATABASE_URL points at localhost:3306).
# Fixes the "Pulling schema from database..." hang: raise
# net_read/net_write_timeout (done above) + stop imports while
# running `pnpm db:generate` / drizzle-kit introspection.
mariadb-turbo:
image: mariadb:11
container_name: mariadb-turbo
profiles: ["db"]
network_mode: host
restart: unless-stopped
environment:
# mariadb:11 uses MARIADB_*; MYSQL_* also works but is deprecated there.
- MARIADB_ROOT_PASSWORD=${MARIADB_ROOT_PASSWORD:-root}
- MARIADB_DATABASE=${MARIADB_DATABASE:-habbo}
- MARIADB_USER=${MARIADB_USER:-cms}
- MARIADB_PASSWORD=${MARIADB_PASSWORD:-cms}
- MARIADB_AUTO_UPGRADE=1
# MariaDB tunes itself — the Official image appends these flags to mysqld,
# no external .cnf file needed.
command: [
# Charset
"--character-set-server=utf8mb4",
"--collation-server=utf8mb4_unicode_ci",
# 50 MB JSON batches: raise the 16 MB default packet ceiling.
"--max-allowed-packet=512M",
"--net-buffer-length=1M",
# Timeouts — fixes the "Pulling schema from database..." hang
# (drizzle-kit introspection no longer starves behind big imports).
"--connect-timeout=30",
"--wait-timeout=3600",
"--interactive-timeout=3600",
"--net-read-timeout=600",
"--net-write-timeout=600",
# InnoDB: durability/performance trade-off for bulk writes.
"--innodb-flush-log-at-trx-commit=2",
"--innodb-buffer-pool-size=2G",
"--innodb-buffer-pool-instances=4",
"--innodb-log-file-size=1G",
"--innodb-log-buffer-size=64M",
"--innodb-flush-method=O_DIRECT",
"--innodb-autoextend-increment=512",
"--innodb-max-dirty-pages-pct=90",
"--bulk-insert-buffer-size=512M",
# Raise so the engine nearly never double-writes during a 50 MB load.
"--innodb-doublewrite=0",
# libaio/native_aio misbehaves in some containers; io_uring path is fine.
"--innodb-use-native-aio=0",
# Temp tables used when MariaDB scans JSON blobs cannot be indexed.
"--tmp-table-size=256M",
"--max-heap-table-size=256M",
"--read-buffer-size=4M",
"--read-rnd-buffer-size=16M",
# Save ~1-2 GB RAM; bulk loads don't need the instrumentation.
"--performance-schema=OFF",
"--skip-name-resolve",
]
volumes:
# Named volume, NOT a host bind — shared-filesystem sync trashes InnoDB
# files the same way it trashes pnpm's node_modules. Named volumes live
# inside the container filesystem, so 50 MB of JSON writes never cross a
# host-sync boundary.
- mariadb-turbo-data:/var/lib/mysql
healthcheck:
# healthcheck.sh ships in the official mariadb image.
test: ["CMD-SHELL", "healthcheck.sh --connect --innodb_initialized || mariadb-admin ping --silent"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
mem_limit: 4g
# Named volumes declared once; used by the MariaDB service above.
volumes:
mariadb-turbo-data:
driver: local
+5
View File
@@ -21,6 +21,11 @@
"test:e2e": "playwright test", "test:e2e": "playwright test",
"typecheck": "tsc --noEmit", "typecheck": "tsc --noEmit",
"db:generate": "drizzle-kit generate", "db:generate": "drizzle-kit generate",
"db:introspect": "drizzle-kit introspect",
"db:bulk": "tsx scripts/bulk-import-json.ts",
"db:up": "docker compose --profile db up -d mariadb-turbo",
"db:down": "docker compose --profile db stop mariadb-turbo",
"db:schema:generate": "node scripts/generate-drizzle-schema.mjs",
"db:migrate": "tsx scripts/apply-migrations.ts", "db:migrate": "tsx scripts/apply-migrations.ts",
"db:migrate:status": "tsx scripts/apply-migrations.ts --status", "db:migrate:status": "tsx scripts/apply-migrations.ts --status",
"db:studio": "drizzle-kit studio", "db:studio": "drizzle-kit studio",
+254
View File
@@ -0,0 +1,254 @@
// Bulk-load >50 MB JSON into MariaDB in resumable chunks (2000 rows/batch).
//
// Drizzle multi-row INSERTs are wrapped in ONE statement per chunk, so a
// 50 MB dump becomes a few dozen statements instead of hundreds of thousands
// of round-trips — no connect/packet timeouts, no "Pulling schema from
// database..." freeze (that hang is introspection racing a saturated server).
//
// Usage (via pnpm, as required):
// pnpm exec tsx scripts/bulk-import-json.ts \
// --file=/var/www/Gamedata/.../FurnitureData.json [--chunk-size=2000]
//
// Input shapes supported automatically:
// A) habbo furnidata_json → { roomitemtypes:{furnitype:[…]}, wallitemtypes:{…} }
// B) flat array of objects → [ {…}, {…} ]
//
// Flags:
// --file=<path> JSON file to load (required)
// --table=<name> one of: furnidata | docs | texts (default: furnidata)
// --category=<x> category value for docs/texts tables
// --source=<x> source value for furnidata table (default: habbo)
// --chunk-size=N rows per multi-row INSERT (default: 2000)
// --limit=N stop after N rows (dry-test without loading everything)
// --truncate DELETE all rows for this table's source/category first
//
// Idempotent by default: rows re-import on their unique key with
// ON DUPLICATE KEY UPDATE, so interrupted runs resume safely.
import "./load-env";
import { createHash } from "node:crypto";
import { resolve } from "node:path";
import { sql } from "drizzle-orm";
import { drizzle } from "drizzle-orm/mysql2";
import mysql from "mysql2/promise";
import {
GamedataDocs,
GamedataFurnidata,
GamedataTexts,
} from "@/db/schema-gamedata";
import { mysqlConnectionUrl } from "./db-url";
interface Args {
file: string;
table: "furnidata" | "docs" | "texts";
category: string;
source: string;
chunkSize: number;
limit: number;
truncate: boolean;
}
const FURNIDATA_SECTIONS = {
roomitemtypes: "s",
flooritemtypes: "s",
wallitemtypes: "i",
effecttypes: "e",
} as const;
function parseArgs(raw: string[]): Args {
const get = (name: string) => {
const hit = raw.find((a) => a.startsWith(`--${name}=`));
return hit?.slice(name.length + 3);
};
const has = (name: string) => raw.includes(`--${name}`);
const file = get("file");
if (!file) {
console.error(
"Usage: pnpm exec tsx scripts/bulk-import-json.ts --file=<path> [--table=furnidata|docs|texts] [--category=<x>] [--source=<x>] [--chunk-size=2000] [--limit=N] [--truncate]",
);
process.exit(1);
}
const table = (get("table") ?? "furnidata") as Args["table"];
return {
file: resolve(file),
table,
category: get("category") ?? "default",
source: get("source") ?? "habbo",
chunkSize: Number(get("chunk-size") ?? "2000"),
limit: Number(get("limit") ?? "0"),
truncate: has("truncate"),
};
}
// A >50 MB file parses fine with a single JSON.parse (V8 string limit is far
// higher); streaming row-by-row would slow the seed down for no memory win.
async function readJsonFile(file: string): Promise<unknown> {
const { readFile } = await import("node:fs/promises");
return JSON.parse(await readFile(file, "utf8"));
}
interface NormalizedRow {
[key: string]: unknown;
}
// Normalize any supported shape into a flat list of JSON documents.
function normalize(json: unknown, table: Args["table"]): NormalizedRow[] {
if (table !== "furnidata") {
if (Array.isArray(json)) return json as NormalizedRow[];
throw new Error(`Expected a top-level array for --table=${table}`);
}
if (Array.isArray(json)) return json as NormalizedRow[];
// habbo furnidata_json: { roomitemtypes: { furnitype: [...] }, ... }
if (json && typeof json === "object") {
const rows: NormalizedRow[] = [];
for (const [section, kind] of Object.entries(FURNIDATA_SECTIONS)) {
const block = (json as Record<string, unknown>)[section];
if (!block || typeof block !== "object") continue;
const list = (block as Record<string, unknown>).furnitype;
if (!Array.isArray(list)) continue;
for (const item of list) rows.push({ ...item, kind });
}
return rows;
}
throw new Error("Unsupported JSON shape: expected array or furnidata_json");
}
function toFurnidataRow(
row: NormalizedRow,
source: string,
): Record<string, unknown> {
return {
source,
kind: (row.kind as "s" | "i" | "e") ?? "s",
className: String(row.classname ?? row.className ?? ""),
publicName: String(row.public_name ?? row.publicName ?? ""),
spriteId: Number(row.id ?? 0),
payload: JSON.stringify(row),
};
}
function toDocsRow(
row: NormalizedRow,
category: string,
): Record<string, unknown> {
const payload = JSON.stringify(row);
return {
category,
docKey: String(row.key ?? row.docKey ?? row.id ?? ""),
payload,
payloadSha1: createHash("sha1").update(payload).digest("hex"),
};
}
function toTextsRow(
row: NormalizedRow,
category: string,
): Record<string, unknown> {
return {
category,
textKey: String(row.key ?? row.id ?? ""),
value: String(row.value ?? row.text ?? row),
};
}
async function main() {
const args = parseArgs(process.argv.slice(2));
const url = process.env.DATABASE_URL;
if (!url)
throw new Error("DATABASE_URL is required (load-env.ts loaded .env)");
const conn = await mysql.createConnection(mysqlConnectionUrl(url));
const db = drizzle(conn);
// Session bulk-load flags: worst case per chunk is a lost commit, not a
// corrupt table. FOREIGN_KEY_CHECKS off keeps each 50 MB chunk off FK
// validation work; UNIQUE_CHECK is a per-statement no-op vs ON DUPLICATE.
await conn.query("SET SESSION FOREIGN_KEY_CHECKS=0");
await conn.query("SET SESSION sql_mode='NO_ENGINE_SUBSTITUTION'");
const tableRef =
args.table === "furnidata"
? GamedataFurnidata
: args.table === "docs"
? GamedataDocs
: GamedataTexts;
const keyWhere =
args.table === "furnidata"
? sql`${GamedataFurnidata.source} = ${args.source}`
: args.table === "docs"
? sql`${GamedataDocs.category} = ${args.category}`
: sql`${GamedataTexts.category} = ${args.category}`;
if (args.truncate) {
await db.execute(sql`DELETE FROM ${tableRef} WHERE ${keyWhere}`);
console.log(`[bulk] truncated rows for ${args.table}`);
}
console.log(`[bulk] reading ${args.file} …`);
const json = await readJsonFile(args.file);
const rows = normalize(json, args.table);
console.log(`[bulk] parsed ${rows.length} documents (${args.table})`);
const mapper: (r: NormalizedRow) => Record<string, unknown> =
args.table === "furnidata"
? (r) => toFurnidataRow(r, args.source)
: args.table === "docs"
? (r) => toDocsRow(r, args.category)
: (r) => toTextsRow(r, args.category);
const values = rows
.slice(0, args.limit || rows.length)
.map<Record<string, unknown>>(mapper);
const total = values.length;
const chunkSize = args.chunkSize;
const started = Date.now();
let inserted = 0;
for (let i = 0; i < total; i += chunkSize) {
const chunk = values.slice(i, Math.min(i + chunkSize, total));
// `tableRef` is a 3-way union; drizzle's insert is typed per single
// table, so narrow through a custom shape here (runtime is unaffected).
const insert = db.insert(tableRef) as unknown as {
values: (rows: typeof chunk) => {
onDuplicateKeyUpdate: (opts: {
set: Record<string, unknown>;
}) => Promise<unknown>;
};
};
await insert.values(chunk).onDuplicateKeyUpdate({
// MariaDB `VALUES(col)` = the incoming row value; re-runs overwrite.
set: Object.fromEntries(
Object.keys(chunk[0] ?? {}).map((k) => [
k,
sql.raw(`values(\`${k}\`)`),
]),
),
});
inserted += chunk.length;
const remaining = total - i - chunk.length;
if (inserted % (chunkSize * 10) < chunkSize || remaining <= 0) {
const elapsedSec = (Date.now() - started) / 1000;
const rate = Math.round(inserted / Math.max(elapsedSec, 0.001));
const etaMin = Math.round(remaining / Math.max(rate, 1) / 60);
console.log(
`[bulk] ${inserted}/${total} (${Math.round((inserted / total) * 100)}%) ${rate} rows/s, ETA ~${etaMin}m`,
);
}
}
const sec = ((Date.now() - started) / 1000).toFixed(1);
console.log(`[bulk] DONE: ${inserted}/${total} rows in ${sec}s`);
if (total === args.limit && args.limit > 0) {
console.log(
" NOTE: --limit was set; run again without it for the full dump.",
);
}
await conn.end();
}
main().catch((err) => {
console.error("[bulk] FATAL:", err instanceof Error ? err.message : err);
process.exit(1);
});
+7 -1
View File
@@ -74,7 +74,13 @@ pnpm install --frozen-lockfile
pnpm exec playwright install chromium pnpm exec playwright install chromium
echo "Building $image" echo "Building $image"
DOCKER_BUILDKIT=1 docker build --network=host --progress=plain --cache-from epicnext-cms:latest -t "$image" . # Throw away the previous build cache before each build so disk usage doesn't
# grow unbounded across deployments. The current release image (`epicnext-cms`)
# is still reused as a base layer via `--cache-from`; only the accumulated
# BuildKit intermediate cache is discarded.
docker builder prune -af --filter "until=1h" --keep-storage=0 2>/dev/null || true
DOCKER_BUILDKIT=1 docker build --network=host --progress=plain --cache-from epicnext-cms:latest \
--build-arg NEXT_DEPLOYMENT_ID="$sha" -t "$image" .
check_current check_current
pnpm db:migrate pnpm db:migrate
check_current check_current
+139
View File
@@ -0,0 +1,139 @@
#!/usr/bin/env bash
# ==============================================================================
# docker-preflight.sh — Verify a VPS is ready to run the Dockerized EpicNext-CMS.
#
# Lets any supplied .env drive the checks. Checks (all idempotent, none mutating):
# 1. Docker + compose available and usable.
# 2. Required runtime dirs exist (RW for UID 33 where the CMS writes).
# 3. Volume owners are UID/GID 33 (www-data) on the host.
# 4. .env exists and required host-network services are reachable.
# 5. Port 3002 free (or the host CMS that must be stopped).
#
# Usage: ./scripts/docker-preflight.sh [--fix]
# --fix attempts to auto-correct permission/owner issues (chown).
#
# Exit codes: 0 ready, 1 not ready (or fixed nothing).
# ==============================================================================
set -uo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$DIR" || exit 1
FIX=0
[ "${1:-}" = "--fix" ] && FIX=1
ENV_FILE="${ENV_FILE:-$DIR/.env}"
fail=0
warn=0
# Ports that are expected to be reachable ON THE HOST (network_mode: host).
# These mirror the defaults in .env / the emulator stack; override via env.
CMS_PORT="${CMS_PORT:-3002}"
MYSQL_PORT="${MYSQL_PORT:-3306}"
REDIS_PORT="${REDIS_PORT:-6379}"
RCON_PORT="${RCON_PORT:-3003}"
API_PORT="${API_PORT:-3001}"
IMAGER_PORT="${IMAGER_PORT:-8082}"
# Relative dirs the CMS writes to, plus the absolute shared gamedata root.
RW_DIRS=(
"$DIR/public/nitro-assets"
"$DIR/public/swf"
"$DIR/storage"
"/var/www/Gamedata"
)
say() { printf ' %s\n' "$*"; }
ok() { printf ' \033[32m[OK] \033[0m%s\n' "$*"; }
err() { printf ' \033[31m[FAIL]\033[0m %s\n' "$*"; fail=1; }
wrn() { printf ' \033[33m[WARN]\033[0m %s\n' "$*"; warn=1; }
doing(){ printf '\n\033[1m%s\033[0m\n' "$*"; }
doing "1. Docker engine + compose"
if command -v docker >/dev/null 2>&1; then
ok "docker binary present"
if docker info >/dev/null 2>&1; then ok "daemon reachable"; else err "daemon NOT reachable (is it running / does this user have access?)"; fi
else
err "docker binary missing"
fi
if docker compose version >/dev/null 2>&1; then
ok "docker compose plugin present"
else
err "docker compose plugin missing (install docker-compose-plugin)"
fi
doing "2. Runtime directories exist + RW for UID 33"
for d in "${RW_DIRS[@]}"; do
if [ ! -e "$d" ]; then
err "missing dir: $d"
if [ "$FIX" -eq 1 ]; then
mkdir -p "$d" && chown 33:33 "$d" && say " created + chown 33:33 $d" || err " could not create $d"
fi
continue
fi
if [ ! -d "$d" ]; then err "not a directory: $d"; continue; fi
# Test write as the target owner via a temp file drop (as root), then remove.
if [ "$(id -u)" -eq 0 ]; then
if touch "$d/.preflight-write-test" 2>/dev/null; then
rm -f "$d/.preflight-write-test"
ok "writable: $d"
else
err "not writable: $d (needs owner 33:33)"
[ "$FIX" -eq 1 ] && { chown -R 33:33 "$d" && say " chown -R 33:33 $d" || err " chown failed"; }
fi
else
if [ -w "$d" ]; then ok "writable: $d"; else err "not writable: $d"; fi
fi
done
doing "3. Volume ownership (UID/GID 33 = www-data)"
for d in "${RW_DIRS[@]}"; do
[ -e "$d" ] || continue
owner="$(stat -c '%u:%g' "$d" 2>/dev/null || true)"
if [ "$owner" = "33:33" ]; then
ok "owner 33:33: $d"
else
err "owner ${owner:-unknown} (want 33:33): $d"
[ "$FIX" -eq 1 ] && { chown 33:33 "$d" && say " chown 33:33 $d" || err " chown failed"; }
fi
done
doing "4. Configuration + required services (.env, host network)"
if [ -f "$ENV_FILE" ]; then
ok ".env present: $ENV_FILE"
### shellcheck disable=SC1090
set -a; . "$ENV_FILE"; set +a
else
err ".env missing: $ENV_FILE (copy your production env here)"
fi
check_port() { # name port
if command -v nc >/dev/null 2>&1; then
if nc -z 127.0.0.1 "$2" >/dev/null 2>&1; then ok "reachable 127.0.0.1:$2 ($1)"; else err "NOT reachable 127.0.0.1:$2 ($1)"; fi
else
if (echo >/dev/tcp/127.0.0.1/"$2") >/dev/null 2>&1; then ok "reachable 127.0.0.1:$2 ($1)"; else err "NOT reachable 127.0.0.1:$2 ($1)"; fi
fi
}
check_port "MariaDB" "$MYSQL_PORT"
check_port "Redis" "$REDIS_PORT"
check_dep() { # name
if command -v "$1" >/dev/null 2>&1; then ok "host binary: $1"; else wrn "host binary not found: $1 (may still work via container)"; fi
}
check_dep git
doing "5. Port 3002 state (host CMS clash)"
if (echo >/dev/tcp/127.0.0.1/"$CMS_PORT") >/dev/null 2>&1; then
err "port $CMS_PORT already in use on host — stop the host-side CMS (pm2 stop next) before starting the container"
else
ok "port $CMS_PORT free"
fi
printf '\n'
if [ "$fail" -eq 1 ]; then
printf '\033[31m=== NOT READY: %s issue(s) found%s ===\033[0m\n' "$fail" "$([ "$FIX" -eq 1 ] && echo ' after --fix' || echo ' (re-run with --fix to auto-correct)')"
exit 1
fi
if [ "$warn" -eq 1 ]; then printf '\033[33m=== READY (with %s warning(s)) ===\033[0m\n' "$warn"; exit 0; fi
printf '\033[32m=== READY ===\033[0m\n'
exit 0
+31 -5
View File
@@ -32,7 +32,13 @@ touch "$LOG_FILE"
log "=== Start docker-update ===" log "=== Start docker-update ==="
# --- 0. Guard: uncommitted changes would break git pull / taint deploys --- # --- 0. Preflight: verify this VPS is ready (permissions, ports, deps) ---
if ! "$DIR/scripts/docker-preflight.sh"; then
die "preflight failed — fix issues first (see '--fix' flag)" 1
fi
log "preflight OK"
# --- 0b. Guard: uncommitted changes would break git pull / taint deploys ---
if ! { git diff --quiet --exit-code && git diff --cached --quiet --exit-code; }; then if ! { git diff --quiet --exit-code && git diff --cached --quiet --exit-code; }; then
die "working tree has uncommitted changes; commit or stash first" 1 die "working tree has uncommitted changes; commit or stash first" 1
fi fi
@@ -55,15 +61,35 @@ else
fi fi
log "db:migrate OK" log "db:migrate OK"
# --- 3. Rebuild the image --- # --- 3. Node major gate (patches auto, major upgrades need review) ---
# `node:alpine` floats within, then across, Node majors. Patches/minors are
# safe to apply silently; a NEW major (e.g. 26 -> 27) is a breaking risk for
# native addons / Next compatibility, so require an explicit review before it
# goes live. Compare the major of the deployed runtime image vs the floating
# tag; abort (not deploy) when they differ.
deployed_major="$(docker inspect --format '{{.Config.Image}}' epicnext-cms 2>/dev/null || true)"
# Resolve the currently-deployed Node major from its image.
if [ -n "$deployed_major" ] && docker image inspect "$deployed_major" >/dev/null 2>&1; then
deployed_major="$(docker run --rm --entrypoint sh "$deployed_major" -c 'node -p "process.versions.node.split(\".\")[0]"' 2>/dev/null || true)"
fi
float_major="$(docker run --rm --entrypoint sh node:alpine -c 'node -p "process.versions.node.split(\".\")[0]"' 2>/dev/null || true)"
if [ -n "$deployed_major" ] && [ -n "$float_major" ] && [ "$deployed_major" != "$float_major" ]; then
die "Node major change detected (deployed v$deployed_major, floating tag v$float_major). Major upgrades require review; update engines/Dockerfile deliberately first." 1
fi
log "Node major gate OK (major=${float_major:-?})"
# --- 4. Rebuild the image ---
# Reset the BuildKit cache first so the build doesn't accumulate unbounded
# layers on disk across daily rebuilds.
docker builder prune -af --filter "until=1h" --keep-storage=0 2>>"$LOG_FILE" || true
docker compose build >>"$LOG_FILE" 2>&1 || die "docker compose build failed" 2 docker compose build >>"$LOG_FILE" 2>&1 || die "docker compose build failed" 2
log "docker compose build OK" log "docker compose build OK"
# --- 4. Recreate the container --- # --- 5. Recreate the container ---
docker compose up -d >>"$LOG_FILE" 2>&1 || die "docker compose up failed" 2 docker compose up -d >>"$LOG_FILE" 2>&1 || die "docker compose up failed" 2
log "docker compose up OK" log "docker compose up OK"
# --- 5. Wait for health (up to ~4 min) --- # --- 6. Wait for health (up to ~4 min) ---
healthy=0 healthy=0
for i in $(seq 1 16); do for i in $(seq 1 16); do
status="$(docker inspect --format='{{.State.Health.Status}}' epicnext-cms 2>/dev/null || true)" status="$(docker inspect --format='{{.State.Health.Status}}' epicnext-cms 2>/dev/null || true)"
@@ -82,7 +108,7 @@ else
exit 3 exit 3
fi fi
# --- 6. Make sure the stale host-side PM2 CMS stays stopped --- # --- 7. Make sure the stale host-side PM2 CMS stays stopped ---
if command -v pm2 >/dev/null 2>&1 && pm2 jlist >/dev/null 2>&1; then if command -v pm2 >/dev/null 2>&1 && pm2 jlist >/dev/null 2>&1; then
if pm2 list 2>/dev/null | grep -q "${PM2_APP}"; then if pm2 list 2>/dev/null | grep -q "${PM2_APP}"; then
pm2 stop "$PM2_APP" >/dev/null 2>&1 && log "pm2 '${PM2_APP}' kept stopped (avoids port 3002 clash)" pm2 stop "$PM2_APP" >/dev/null 2>&1 && log "pm2 '${PM2_APP}' kept stopped (avoids port 3002 clash)"
+106
View File
@@ -0,0 +1,106 @@
// Gamedata / large-JSON storage schema for MariaDB.
//
// MariaDB's JSON type is an alias for LONGTEXT (+ a `CHECK (json_valid(...))`
// constraint), so it can NEVER be indexed directly. The "smart" pattern:
// 1. keep the raw JSON document in longtext / mediumtext — full fidelity,
// no validation overhead, no 4 GB fetch if only a key is needed;
// 2. promote the fields you actually WHERE/ORDER BY into real columns;
// 3. index those columns (optionally as VIRTUAL generated columns using
// JSON_EXTRACT so you don't duplicate data in the seed script).
//
// Not part of drizzle's auto-generated src/db/schema.ts (which is regenerated
// from the live emulator DB) — this file is a standalone schema module you
// pass to `drizzle(conn, { schema })` in bulk scripts or import directly.
import { sql } from "drizzle-orm";
import {
index,
int,
longtext,
mediumtext,
mysqlEnum,
mysqlTable,
uniqueIndex,
varchar,
} from "drizzle-orm/mysql-core";
// ── Furnidata: one row per furniture/clothing item ───────────────────────────
// Compatible with the habbo furnidata_json shape:
// { "roomitemtypes": { "furnitype": [ { id, classname, public_name, ... } ] },
// "wallitemtypes": { "furnitype": [ ... ] },
// "effecttypes": { "effecttype": [ ... ] } }
export const GamedataFurnidata = mysqlTable(
"gamedata_furnidata",
{
id: int("id", { unsigned: true }).autoincrement().primaryKey().notNull(),
source: varchar("source", { length: 64 }).notNull().default("habbo"),
spriteId: int("sprite_id", { unsigned: true }).notNull(),
// "s" = floor/room item, "i" = wall item, "e" = clothing/effect.
kind: mysqlEnum("kind", ["s", "i", "e"]).notNull().default("s"),
className: varchar("class_name", { length: 128 }).notNull(),
// Promoted from JSON so hot lookups & LIST/WHERE never touch the blob.
publicName: varchar("public_name", { length: 128 }).notNull().default(""),
// Raw JSON document, kept intact for exact round-trips (catalog export).
payload: longtext("payload").notNull(),
},
(t) => [
// Idempotent seeds upsert on this key (source, sprite_id).
uniqueIndex("gamedata_furnidata_source_sprite").on(t.source, t.spriteId),
index("gamedata_furnidata_class").on(t.className),
index("gamedata_furnidata_kind").on(t.kind),
],
);
// ── Generic JSON document store ───────────────────────────────────────────────
// Use for any >50 MB JSON dump split into per-key documents (external_texts,
// product data, achievements, etc.). `title` is a VIRTUAL generated column
// extracted from the payload via JSON_EXTRACT and indexed — MariaDB supports
// secondary indexes on virtual columns (10.2+), so filtering by a payload
// field stays fast without storing a duplicate copy.
export const GamedataDocs = mysqlTable(
"gamedata_docs",
{
id: int("id", { unsigned: true }).autoincrement().primaryKey().notNull(),
category: varchar("category", { length: 64 }).notNull(),
docKey: varchar("doc_key", { length: 128 }).notNull(),
payload: longtext("payload").notNull(),
// SHA-1 of the raw payload → cheap change detection on re-runs.
payloadSha1: varchar("payload_sha1", { length: 40 }).notNull().default(""),
// VIRTUAL (default) column; the JSON_EXTRACT runs at query/index time.
// NOT NULL forces the extract to resolve — use .notNull() only when
// every document guarantees the field.
title: varchar("title", { length: 255 })
.generatedAlwaysAs(
sql`CAST(JSON_UNQUOTE(JSON_EXTRACT(payload, '$.title')) AS CHAR(255))`,
{ mode: "virtual" },
)
.notNull(),
},
(t) => [
uniqueIndex("gamedata_docs_cat_key").on(t.category, t.docKey),
index("gamedata_docs_title").on(t.title),
],
);
// ── Key/value texts (external_texts style, split by category) ───────────────
// One row per string; bulk-loads stay resumable via (category, text_key).
export const GamedataTexts = mysqlTable(
"gamedata_texts",
{
id: int("id", { unsigned: true }).autoincrement().primaryKey().notNull(),
category: varchar("category", { length: 64 }).notNull(),
textKey: varchar("text_key", { length: 255 }).notNull(),
value: mediumtext("value").notNull(),
},
(t) => [
uniqueIndex("gamedata_texts_cat_key").on(t.category, t.textKey),
index("gamedata_texts_key").on(t.textKey),
],
);
export type GamedataFurnidataRow = typeof GamedataFurnidata.$inferSelect;
export type GamedataFurnidataInsert = typeof GamedataFurnidata.$inferInsert;
export type GamedataDocsRow = typeof GamedataDocs.$inferSelect;
export type GamedataDocsInsert = typeof GamedataDocs.$inferInsert;
export type GamedataTextsRow = typeof GamedataTexts.$inferSelect;
export type GamedataTextsInsert = typeof GamedataTexts.$inferInsert;
+9 -6
View File
@@ -6,18 +6,21 @@ const dockerfile = readFileSync("Dockerfile", "utf8");
describe("Docker build cache", () => { describe("Docker build cache", () => {
it("installs frozen dependencies before copying application source", () => { it("installs frozen dependencies before copying application source", () => {
const manifests = dockerfile.indexOf( const manifests = dockerfile.indexOf(
"COPY package.json *lock* pnpm-workspace.yaml .npmrc ./", "COPY package.json pnpm-lock.yaml* pnpm-workspace.yaml* .npmrc* ./",
); );
const fetch = dockerfile.indexOf("pnpm fetch --ignore-scripts");
const install = dockerfile.indexOf("pnpm install --frozen-lockfile"); const install = dockerfile.indexOf("pnpm install --frozen-lockfile");
const source = dockerfile.indexOf("COPY . ."); const source = dockerfile.indexOf("COPY . .");
expect(manifests).toBeGreaterThan(-1); expect(manifests).toBeGreaterThan(-1);
expect(install).toBeGreaterThan(manifests); expect(fetch).toBeGreaterThan(manifests);
expect(install).toBeGreaterThan(fetch);
expect(source).toBeGreaterThan(install); expect(source).toBeGreaterThan(install);
}); });
it("retains the package store and Next compiler cache across source changes", () => { it("keeps dependency downloads in a lockfile-only cached layer", () => {
expect(dockerfile).toContain("id=epicnext-pnpm,target=/pnpm/store"); expect(dockerfile).toContain("pnpm fetch --ignore-scripts");
expect(dockerfile).toContain("--store-dir=/pnpm/store"); expect(dockerfile).toContain(
expect(dockerfile).toContain("id=epicnext-next,target=/app/.next/cache"); "pnpm install --frozen-lockfile --ignore-scripts --offline",
);
}); });
it("ships standalone output without a redundant dependency pruning step", () => { it("ships standalone output without a redundant dependency pruning step", () => {
expect(dockerfile).toContain("/app/.next/standalone ./"); expect(dockerfile).toContain("/app/.next/standalone ./");