perf: pre-compress gamedata JSON, tune MariaDB, fix avatar imager, docs
- scripts/compress-gamedata.mjs: pre-compress large gamedata JSON to .gz (gzip level 9, idempotent mtime check) served via nginx gzip_static (48 MB FurnitureData.json -> ~2.4 MB, ~0.3s -> ~0.005s per request) - package.json: add gamedata:compress script - fix(imaging): accept real Habbo figure strings in avatar route (allow optional second number per part, e.g. hd-180-1.ch-210-66) - README: document avatar imager container (avatar-imaging-pixinode, port 8082, /docker/Polaris-imager), nginx /imaging proxying, MariaDB tuning, gamedata pre-compression cron and caching layers
This commit is contained in:
openhands committed
1 parent
251738fdda
commit
9dc9d1fa4b
4 files changed
+260
-9
No files matched your search
@@ -112,7 +112,7 @@ Open `http://localhost:3002` in your browser.
|
||||
|
||||
## Docker Deployment
|
||||
|
||||
The CMS ships with a multi-stage Dockerfile that automatically detects your package manager (pnpm, npm, or yarn) based on which lockfile is present. Only the **CMS** runs in Docker; the Nitro/Octane client and its renderer stay outside and are served by nginx on the host.
|
||||
The CMS ships with a multi-stage Dockerfile that automatically detects your package manager (pnpm, npm, or yarn) based on which lockfile is present. The **CMS** runs in Docker; the Nitro/Octane client stays outside and is served by nginx on the host. The server-side **avatar imager** also runs in its own Docker container (`avatar-imaging-pixinode`, port 8082) — see [Avatar Imaging](#avatar-imaging).
|
||||
|
||||
### Prerequisites
|
||||
|
||||
@@ -212,7 +212,7 @@ This means you can use any package manager on your host machine — the Docker b
|
||||
|
||||
### Volumes
|
||||
|
||||
Only the CMS runs in Docker. The heavy client assets and gamedata stay on the host and are shared into the container so imports and uploads persist:
|
||||
Only the CMS (and the optional avatar imager container, see [Avatar Imaging](#avatar-imaging)) run in Docker. The heavy client assets and gamedata stay on the host and are shared into the container so imports and uploads persist:
|
||||
|
||||
| Container Path | Host Path | Mode | Purpose |
|
||||
| -------------------------- | -------------------------- | ---- | ------------------------------------ |
|
||||
@@ -228,7 +228,7 @@ Only the CMS runs in Docker. The heavy client assets and gamedata stay on the ho
|
||||
| Component | Host path | How it's served |
|
||||
| -------------------- | ----------------------- | ---------------------------------------------------------------------- |
|
||||
| Nitro/Octane client | `/var/www/Octane/dist` | nginx `location ^~ /client/` and `/nitro-client/` alias |
|
||||
| Octane-Renderer | `/var/www/Octane-Renderer` | Optional. Run separately (or as the commented-out `imager` service) proxied by nginx `/imaging` |
|
||||
| Avatar imager | `/docker/Polaris-imager`| Docker container `avatar-imaging-pixinode` on port `8082` — see [Avatar Imaging](#avatar-imaging) |
|
||||
| Camera uploads | `/var/www/Camera` | nginx `/camera/` alias |
|
||||
|
||||
**Container user & write permissions:** the CMS container runs as `www-data` (UID/GID 33) by default to match the host owner of `/var/www/Gamedata`. Ensure the other write volumes (`./public/nitro-assets`, `./public/swf`, `./storage`) are also owned by `www-data` on the host:
|
||||
@@ -386,9 +386,11 @@ server {
|
||||
add_header Cache-Control "public, max-age=31536000, immutable";
|
||||
}
|
||||
|
||||
location /imaging {
|
||||
proxy_pass http://127.0.0.1:3030;
|
||||
add_header Cache-Control "public, max-age=3600, s-maxage=86400";
|
||||
location /imaging/ {
|
||||
proxy_pass http://127.0.0.1:8082/;
|
||||
proxy_set_header Connection "";
|
||||
proxy_http_version 1.1;
|
||||
add_header Cache-Control "public, max-age=300";
|
||||
}
|
||||
|
||||
location /health {
|
||||
@@ -399,6 +401,122 @@ server {
|
||||
}
|
||||
```
|
||||
|
||||
> **`/imaging` trailing-slash gotcha:** `location /imaging` (no trailing slash) combined with `proxy_pass http://…8082/;` rewrites `/imaging/avatarimage` into `//avatarimage` and 404s. Always use `location /imaging/` + a trailing-slash `proxy_pass` so `/imaging/avatarimage` reaches the imager's `/avatarimage` route.
|
||||
|
||||
---
|
||||
|
||||
## Avatar Imaging
|
||||
|
||||
Avatar images (profile figures, chat badges, forum avatars, admin previews) are rendered server-side by **Polaris-imager** ([duckietm/Polaris-imager](https://github.com/duckietm/Polaris-imager)) — a headless Nitro/Octane renderer that runs `@pixi/node` (real WebGL via `gl` + Xvfb) inside a Docker container named **`avatar-imaging-pixinode`**. It serves `GET /avatarimage?figure=…` on port **8082** and can emit PNG/APNG/GIF, gestures, actions, scenes and more.
|
||||
|
||||
### Request flow
|
||||
|
||||
```
|
||||
browser → Cloudflare → (page rule /origin for /imaging/*) → host:8082
|
||||
browser → nginx origin → location /imaging/ → 127.0.0.1:8082/avatarimage
|
||||
```
|
||||
|
||||
URLs look like `https://<hotel>/imaging/avatarimage?figure=hd-180-1.ch-210-66&img_format=png&size=l&direction=4`. The public `NEXT_PUBLIC_IMAGER_URL` points at that endpoint (see `src/lib/imager.ts`). Responses carry `Cache-Control: public, max-age=300`, so browsers and Cloudflare reuse a render for 5 minutes.
|
||||
|
||||
### Requirements (host)
|
||||
|
||||
- Docker (daemon with `build.network: host`, same as the CMS build — this host disables Docker iptables).
|
||||
- An nginx that serves `/gamedata/` (FigureData/FigureMap/EffectMap…) and `/gamedata/bundled` (`.nitro` assets) over HTTP so the renderer can fetch them. The compose file points at `host.docker.internal:8081`.
|
||||
|
||||
### Configuration — `/docker/Polaris-imager/.env`
|
||||
|
||||
```dotenv
|
||||
NITRO_GAMEDATA_URL=http://host.docker.internal:8081/gamedata/config
|
||||
NITRO_ASSET_URL=http://host.docker.internal:8081/gamedata/bundled
|
||||
AVATAR_IMAGING_FPS=12
|
||||
AVATAR_IMAGING_MAX_FRAMES=60
|
||||
AVATAR_IMAGING_HOST=0.0.0.0
|
||||
AVATAR_IMAGING_PORT=8082
|
||||
AVATAR_IMAGING_RATELIMIT_MAX=600
|
||||
AVATAR_IMAGING_SCENE=1
|
||||
AVATAR_IMAGING_WARDROBE=1
|
||||
AVATAR_IMAGING_HABBO_FONTS=1
|
||||
AVATAR_IMAGING_CHAT_BUBBLES=1
|
||||
AVATAR_IMAGING_HOTEL_URL=https://<hotel>
|
||||
AVATAR_IMAGING_HOTEL_NAME=<hotel>
|
||||
```
|
||||
|
||||
### Build & start
|
||||
|
||||
The image is self-contained — the `Dockerfile` clones the pinned Nitro/Octane renderer and runs the full `yarn`/`vite` bundle at build time, then produces a slim Debian runtime with Xvfb + Mesa software rendering:
|
||||
|
||||
```bash
|
||||
cd /docker/Polaris-imager
|
||||
docker compose build
|
||||
docker compose up -d # container: avatar-imaging-pixinode
|
||||
docker logs -f avatar-imaging-pixinode # expect: "listening on http://0.0.0.0:8082"
|
||||
```
|
||||
|
||||
The container runs `restart: unless-stopped` and ships a healthcheck that probes `/health` (reports `{ "ready": true }` once the headless renderer has booted). Verify a render:
|
||||
|
||||
```bash
|
||||
curl -o avatar.png 'http://127.0.0.1:8082/avatarimage?figure=hr-893-45.hd-600-1.ch-255-66.lg-280-110.sh-295-62&img_format=png&size=l'
|
||||
```
|
||||
|
||||
A production issue that this section documents: **the `avatar-imaging-pixinode` image can be removed by a `docker image prune`**, which takes the whole site's avatars offline (502 on `/imaging/*`). Keep the image tagged and re-build it via the commands above if it ever disappears (`docker compose build && docker compose up -d`).
|
||||
|
||||
### Figure string validation (`src/app/api/imaging/avatar/route.ts`)
|
||||
|
||||
The CMS-side route re-validates the incoming figure against `/^[a-z]{2}-\d+(-\d+)?(\.[a-z]{2}-\d+(-\d+)?)*$/i`. Real Habbo figure strings use `type-id-color` parts separated by dots (`hd-180-1.ch-210-66`), so each part allows an optional second number. Invalid input is rejected with `400` before any render happens.
|
||||
|
||||
---
|
||||
|
||||
## Performance Tuning (production)
|
||||
|
||||
The CMS shares its MySQL/MariaDB database with the emulator, so the hot wins come from caching, asset serving and database knob-tuning — not from a storage-engine migration.
|
||||
|
||||
### 1. gzip_static pre-compression of gamedata JSON
|
||||
|
||||
The client downloads `FurnitureData.json` (~48 MB, one file per supported language) plus other large JSON from `/gamedata/`. nginx has `gzip` and `gzip_static on` (see `/etc/nginx/nginx.conf`), but with no pre-built `.gz` file it re-compressed the 48 MB **on every request** (~0.25–0.30 s of CPU per miss).
|
||||
|
||||
`scripts/compress-gamedata.mjs` pre-compresses every JSON >1 MB in `<Gamedata>/config` and `<Gamedata>/bundled/config` once (gzip level 9): `FurnitureData*.json` go from ~48 MB to ~2.4 MB (~95 % smaller). It is idempotent — a file is only re-compressed when the source mtime is newer than its `.gz` (e.g. after a Studio catalog export).
|
||||
|
||||
```bash
|
||||
pnpm gamedata:compress # manual run
|
||||
```
|
||||
|
||||
Once the `.gz` exists, nginx `gzip_static` serves it straight from disk with `Content-Encoding: gzip` and effectively zero CPU. **Measured result on the production origin: 0.25–0.30 s → ~0.005 s per file.**
|
||||
|
||||
The production host runs the script on a cron so newly exported catalogs are compressed shortly after they land:
|
||||
|
||||
```bash
|
||||
crontab -e
|
||||
# */15 * * * * node /var/www/atom-nexst/scripts/compress-gamedata.mjs >> /var/www/atom-nexst/logs/gamedata-compress.log 2>&1
|
||||
```
|
||||
|
||||
### 2. MariaDB tuning
|
||||
|
||||
The whole `habbo` datadir is roughly 0.5 GB; the default InnoDB buffer pool (128 MB) pushed index/row reads to disk. The pool is raised to **4 GB** and the slow-query log is enabled (threshold 2 s), both live (`SET GLOBAL …`) and persisted for boot in `/etc/mysql/mariadb.conf.d/zz-cms-performance.cnf`:
|
||||
|
||||
```ini
|
||||
[mysqld]
|
||||
innodb_buffer_pool_size = 4G
|
||||
slow_query_log = 1
|
||||
long_query_time = 2
|
||||
slow_query_log_file = /var/log/mysql/mariadb-slow.log
|
||||
```
|
||||
|
||||
The slow-query log surfaces hot-spot SQL for a follow-up index/query audit. The CMS connection pool itself is already tuned (pool size 25, `connection_limit` in `DATABASE_URL`).
|
||||
|
||||
### 3. Asset caching headers
|
||||
|
||||
| Path | Cache-Control |
|
||||
| ------------------------------ | -------------------------------------------------------------- |
|
||||
| `/swf/**`, `/nitro-assets/**` | `public, max-age=604800, stale-while-revalidate=2592000` |
|
||||
| `/gamedata/**`, `/client/**` | `public` + `expires 7–30 d` (nginx) |
|
||||
| `/imaging/*` | `public, max-age=300` |
|
||||
| `/camera/**` | `public, max-age=31536000, immutable` |
|
||||
| `/_next/static/**` | `public, max-age=31536000, immutable` |
|
||||
|
||||
### 4. Cache layers (Redis + CDN)
|
||||
|
||||
The CMS caches through `cached()` / `cachedQuery()` (`src/lib/cache.ts`): an in-memory fast path with a Redis (DragonflyDB-compatible) fallback and single-flight protection against cache stampedes. Public API routes (home, leaderboard, online count, radio, photos, …) fill Redis keys per route; verify with `redis-cli DBSIZE`. With Cloudflare in front, static and `/imaging/*` responses are additionally cached edge-side (`cf-cache-status`), cutting origin load further.
|
||||
|
||||
---
|
||||
|
||||
## Production Deployment (PM2)
|
||||
@@ -435,6 +553,7 @@ pm2 restart next
|
||||
| `pnpm db:generate` | Draft SQL via drizzle-kit → `drizzle/drafts/` |
|
||||
| `pnpm db:studio` | Drizzle Studio (dev) |
|
||||
| `pnpm db:introspect` | drizzle-kit introspect (draft) |
|
||||
| `pnpm gamedata:compress` | Pre-compress large gamedata JSON to `.gz` (gzip_static) |
|
||||
| `pnpm analyze` | Build + open bundle analyzer |
|
||||
| `pnpm jobs:worker` | Start background task worker |
|
||||
| `pnpm biome:check` | Lint and format code |
|
||||
@@ -456,6 +575,7 @@ pm2 restart next
|
||||
| **Streaming & Suspense** | Next.js App Router streaming for fast page loads |
|
||||
| **Automatic Image Optimization** | `next/image` with Sharp for resizing and WebP/AVIF |
|
||||
| **Compression** | Gzip compression enabled on all responses |
|
||||
| **gzip_static (gamedata)** | 48 MB FurnitureData JSON served from pre-built `.gz` — no per-request CPU |
|
||||
| **PWA** | Service worker with skip-waiting, stale CSS chunk recovery |
|
||||
| **Biome** | Fast linting and formatting (replaces ESLint + Prettier) |
|
||||
|
||||
@@ -470,7 +590,8 @@ pm2 restart next
|
||||
├── scripts/
|
||||
│ ├── apply-migrations.ts # SQL migration runner (apply + status)
|
||||
│ ├── jobs-worker.ts # Background task scheduler
|
||||
│ └── generate-drizzle-schema.mjs # Regen src/db/schema.ts from schema + live DB
|
||||
│ ├── generate-drizzle-schema.mjs # Regen src/db/schema.ts from schema + live DB
|
||||
│ └── compress-gamedata.mjs # Pre-compress large gamedata JSON (gzip_static)
|
||||
├── src/
|
||||
│ ├── db/
|
||||
│ │ ├── schema.ts # Drizzle ORM schema (committed — runtime data layer)
|
||||
@@ -577,4 +698,3 @@ pnpm biome:check # Lint and format
|
||||
## License
|
||||
|
||||
CC BY-NC-SA 4.0. See `LICENSE` for details.
|
||||
Test aanpassing voor Gitea Actions
|
||||
@@ -24,6 +24,7 @@
|
||||
"db:migrate": "tsx scripts/apply-migrations.ts",
|
||||
"db:migrate:status": "tsx scripts/apply-migrations.ts --status",
|
||||
"db:studio": "drizzle-kit studio",
|
||||
"gamedata:compress": "node scripts/compress-gamedata.mjs",
|
||||
"hk:matrix:check": "tsx scripts/verify-housekeeping-matrix.ts",
|
||||
"test:housekeeping": "vitest run --coverage.enabled=false src/features/housekeeping src/lib/admin-theme-source-audit.test.ts src/lib/admin/authorization-contract.test.ts",
|
||||
"prepare": "node scripts/prepare-hooks.mjs",
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
/**
|
||||
* Pre-compress large gamedata JSON files to .gz so nginx `gzip_static`
|
||||
* serves them via sendfile instead of re-compressing up to 48 MB on every
|
||||
* request (see /etc/nginx/nginx.conf: `gzip_static on`).
|
||||
*
|
||||
* Idempotent: a file is only re-compressed when its source mtime is newer
|
||||
* than the existing .gz (e.g. after a Studio catalog export rewrites it).
|
||||
*
|
||||
* Usage: node scripts/compress-gamedata.mjs
|
||||
* Env: GAMEDATA_ROOT (default /var/www/Gamedata)
|
||||
* GAMEDATA_GZIP_MIN_BYTES (default 1048576)
|
||||
*/
|
||||
|
||||
import { createReadStream, createWriteStream, promises as fs } from "node:fs";
|
||||
import { cpus } from "node:os";
|
||||
import { join, relative } from "node:path";
|
||||
import { pipeline } from "node:stream/promises";
|
||||
import { createGzip } from "node:zlib";
|
||||
|
||||
const GAMEDATA_ROOT = process.env.GAMEDATA_ROOT ?? "/var/www/Gamedata";
|
||||
const MIN_BYTES = Number(process.env.GAMEDATA_GZIP_MIN_BYTES ?? 1024 * 1024);
|
||||
const GZIP_LEVEL = 9;
|
||||
const TARGET_DIRS = ["config", "bundled/config"];
|
||||
const CONCURRENCY = Math.max(1, Math.min(4, cpus().length));
|
||||
|
||||
function isCompressible(name) {
|
||||
return (
|
||||
name.endsWith(".json") &&
|
||||
!name.endsWith(".gz") &&
|
||||
!name.endsWith(".min.json")
|
||||
);
|
||||
}
|
||||
|
||||
async function collectCandidates() {
|
||||
const files = [];
|
||||
for (const dir of TARGET_DIRS) {
|
||||
const root = join(GAMEDATA_ROOT, dir);
|
||||
let entries;
|
||||
try {
|
||||
entries = await fs.readdir(root, { withFileTypes: true });
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
for (const entry of entries) {
|
||||
if (!entry.isFile() || !isCompressible(entry.name)) continue;
|
||||
const full = join(root, entry.name);
|
||||
const stat = await fs.stat(full);
|
||||
if (stat.size < MIN_BYTES) continue;
|
||||
files.push({ src: full, size: stat.size, mtimeMs: stat.mtimeMs });
|
||||
}
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
async function needsCompression({ src, mtimeMs }) {
|
||||
const gz = `${src}.gz`;
|
||||
try {
|
||||
const stat = await fs.stat(gz);
|
||||
return stat.mtimeMs < mtimeMs;
|
||||
} catch {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
async function compressOne({ src }) {
|
||||
const gz = `${src}.gz`;
|
||||
const tmp = `${gz}.tmp-${process.pid}`;
|
||||
await pipeline(
|
||||
createReadStream(src),
|
||||
createGzip({ level: GZIP_LEVEL }),
|
||||
createWriteStream(tmp),
|
||||
);
|
||||
await fs.rename(tmp, gz);
|
||||
try {
|
||||
await fs.chmod(gz, 0o644);
|
||||
} catch {
|
||||
// Best-effort; ownership is up to the caller.
|
||||
}
|
||||
return gz;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const candidates = await collectCandidates();
|
||||
const work = [];
|
||||
const skipped = [];
|
||||
for (const candidate of candidates) {
|
||||
if (await needsCompression(candidate)) {
|
||||
work.push(candidate);
|
||||
} else {
|
||||
skipped.push(candidate);
|
||||
}
|
||||
}
|
||||
|
||||
const results = [];
|
||||
let idx = 0;
|
||||
async function worker() {
|
||||
while (idx < work.length) {
|
||||
const item = work[idx++];
|
||||
try {
|
||||
const gz = await compressOne(item);
|
||||
const stat = await fs.stat(gz);
|
||||
results.push(
|
||||
`compressed ${relative(GAMEDATA_ROOT, gz)} (${item.size} -> ${stat.size} bytes, ${Math.round((1 - stat.size / item.size) * 1000) / 10}% smaller)`,
|
||||
);
|
||||
} catch (err) {
|
||||
results.push(
|
||||
`FAILED ${relative(GAMEDATA_ROOT, item.src)}: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const workers = Array.from(
|
||||
{ length: Math.min(CONCURRENCY, work.length) },
|
||||
() => worker(),
|
||||
);
|
||||
await Promise.all(workers);
|
||||
|
||||
console.log(
|
||||
`gamedata: ${work.length} to compress, ${skipped.length} up to date`,
|
||||
);
|
||||
for (const line of results) console.log(`gamedata: ${line}`);
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(
|
||||
`gamedata: FATAL ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,7 +1,7 @@
|
||||
import { type NextRequest, NextResponse } from "next/server";
|
||||
import { resolveImagerBase } from "@/lib/imager";
|
||||
|
||||
const FIGURE_RE = /^([a-z]{2}-\d+)(\.[a-z]{2}-\d+)*$/i;
|
||||
const FIGURE_RE = /^[a-z]{2}-\d+(-\d+)?(\.[a-z]{2}-\d+(-\d+)?)*$/i;
|
||||
const FIGURE_MAX_LEN = 512;
|
||||
const FIGURE_MAX_PARTS = 24;
|
||||
const UPSTREAM_TIMEOUT_MS = 10_000;
|
||||
|
||||
Reference in new issue
Block a user