feat(db): add self-tuned mariadb-turbo container and bulk JSON importer
- docker-compose: opt-in mariadb-turbo service (profile 'db') with inline mysqld tuning (max_allowed_packet=512M, innodb_flush_log_at_trx_commit=2, 2G buffer pool, net_read/write_timeout=600) for >50 MB JSON bulk loads; named volume for the datadir + node_modules host-sync guidance - scripts/bulk-import-json.ts: chunked (2000-row) idempotent importer with ON DUPLICATE KEY UPDATE, resumable, habbo furnidata or flat-array input - src/db/schema-gamedata.ts: promote+index JSON storage schema (furnidata, docs, texts) incl. VIRTUAL generated columns for MariaDB - package.json: add db:bulk, db:up, db:down, db:introspect, db:schema:generate
This commit is contained in:
openhands committed
1 parent
a640e40a5d
commit
52459c0b74
5 files changed
+540
-2
No files matched your search
@@ -1,4 +1,4 @@
|
||||
# EpicNext-CMS v1.0.1
|
||||
# EpicNext-CMS v1.0.3
|
||||
|
||||
A modern, high-performance content management system for Habbo hotel emulators, built on **Next.js 16** (App Router) with **Drizzle ORM** and **React 19**. Designed to integrate seamlessly with Polaris / Arcturus Morningstar MySQL/MariaDB databases.
|
||||
|
||||
@@ -196,6 +196,7 @@ an exit of 0 means the CMS is healthy on the new commit.
|
||||
| `docker compose pull && docker compose up -d --build` | Update and redeploy |
|
||||
| `pnpm db:migrate` (on the **host**) | Run database migrations (slim image has no source) |
|
||||
| `./scripts/docker-update.sh` | Full automated update (manual or cron) |
|
||||
| `pnpm db:up` / `pnpm db:down` | Start / stop the `mariadb-turbo` bulk-load container |
|
||||
|
||||
### How Package Manager Detection Works
|
||||
|
||||
@@ -258,6 +259,45 @@ The container includes a health check that hits `/api/health` on port 3002 every
|
||||
docker inspect --format='{{.State.Health.Status}}' epicnext-cms
|
||||
```
|
||||
|
||||
### MariaDB Turbo (bulk loads)
|
||||
|
||||
`docker-compose.yml` ships an opt-in **`mariadb-turbo`** service — a dedicated MariaDB 11 container tuned for loading >50 MB JSON dumps (furnidata, external texts, etc.). It configures itself via mysqld flags on the `command:`, so no external `.cnf` file has to be mounted or kept in sync:
|
||||
|
||||
```bash
|
||||
pnpm db:up # docker compose --profile db up -d (starts mariadb-turbo)
|
||||
```
|
||||
|
||||
Key settings (see the `command:` block in `docker-compose.yml`):
|
||||
|
||||
- `max_allowed_packet = 512M` — 50 MB JSON batches no longer hit the 16 MB packet ceiling.
|
||||
- `innodb_flush_log_at_trx_commit = 2` + `innodb_doublewrite = 0` — durability relaxed for bulk writes.
|
||||
- `innodb_buffer_pool_size = 2G`, `innodb_log_file_size = 1G`, `bulk_insert_buffer_size = 512M`.
|
||||
- `net_read_timeout` / `net_write_timeout = 600`, `wait_timeout = 3600` — fixes the `drizzle-kit` **"Pulling schema from database..."** hang by never starving introspection/DDL sessions behind a long import.
|
||||
- `performance_schema = OFF` — saves ~1-2 GB RAM.
|
||||
|
||||
The datadir lives in a **named volume** (`mariadb-turbo-data`), never a host bind-mount: shared-filesystem sync trashes InnoDB files the same way it corrupts pnpm's `node_modules`. Named volumes stay inside the container filesystem, so 50 MB of JSON writes never cross a host-sync boundary.
|
||||
|
||||
> **Port note:** the service uses `network_mode: host` and binds `127.0.0.1:3306` — stop the host MariaDB first, otherwise the port collides with the standalone `.env` `DATABASE_URL` (`localhost:3306`).
|
||||
|
||||
#### node_modules & host-sync corruption
|
||||
|
||||
pnpm's store is hard-linked and its `.bin` shims are symlinks, so **node_modules must never be a host bind-mount** (Docker Desktop gRPC-FUSE/VirtioFS, Unison and Syncthing all corrupt it). The production build bakes `node_modules` into the image at build time; it is never bind-mounted. For local dev, mount a *named volume* (`node_modules:/app/node_modules`) instead of the host directory, and keep `node_modules` / the pnpm store out of any shared-filesystem bind.
|
||||
|
||||
#### Loading a 50 MB JSON dump
|
||||
|
||||
```bash
|
||||
# Habbo furnidata (auto-detects roomitemtypes / wallitemtypes / effecttypes)
|
||||
pnpm db:bulk --file=/var/www/Gamedata/config/FurnitureData.json
|
||||
|
||||
# Flat array of documents → generic JSON store
|
||||
pnpm db:bulk --file=/data/products.json --table=docs --category=furni
|
||||
|
||||
# Key/value texts
|
||||
pnpm db:bulk --file=/data/external_texts.json --table=texts --category=default
|
||||
```
|
||||
|
||||
The importer (`scripts/bulk-import-json.ts`) maps rows onto `src/db/schema-gamedata.ts`, batches them into **2000-row multi-row INSERTs** (one statement per chunk), uses `ON DUPLICATE KEY UPDATE` so re-runs are idempotent and interrupted loads resume, and prints rows/s + ETA. See ["ORM Setup & Type Generation"](#orm-setup--type-generation) → *Bulk JSON storage* for the design.
|
||||
|
||||
|
||||
---
|
||||
|
||||
@@ -284,10 +324,41 @@ const found = await db.select()
|
||||
| `pnpm db:generate` | Draft SQL from Drizzle schema into `drizzle/drafts/` (review + copy) |
|
||||
| `pnpm db:studio` | Open Drizzle Studio (dev only) |
|
||||
| `pnpm db:introspect` | Reverse-engineer an existing DB into a Drizzle schema draft |
|
||||
| `pnpm db:bulk` | Batch-import >50 MB JSON (2000-row chunks, resumable) via `scripts/bulk-import-json.ts` |
|
||||
| `pnpm db:schema:generate` | Regen committed `src/db/schema.ts` from previous schema + live DB |
|
||||
|
||||
> The CMS does **not** use `drizzle-kit push` or `drizzle-kit migrate` — the database is shared with the emulator. Apply CMS DDL only via `pnpm db:migrate`.
|
||||
|
||||
#### Bulk JSON storage (MariaDB)
|
||||
|
||||
Emulator/hotel JSON dumps (furnidata, external texts, product data) are often >50 MB. MariaDB's `JSON` type is an alias for `LONGTEXT` and can **never be indexed directly**, so `src/db/schema-gamedata.ts` follows the "promote + index" pattern:
|
||||
|
||||
- The raw JSON document stays intact in a `longtext` / `mediumtext` column (`payload`, `value`).
|
||||
- The fields you actually `WHERE` / `ORDER BY` on are promoted into real columns (`sprite_id`, `class_name`, `kind`, `title`, `text_key`) and indexed.
|
||||
- Optional **VIRTUAL generated columns** extract indexed fields from the payload with `JSON_EXTRACT` (MariaDB supports secondary indexes on virtual columns, 10.2+), so no duplicate data has to be written by the importer.
|
||||
|
||||
Three tables are exported:
|
||||
|
||||
| Table | Purpose | Unique key |
|
||||
| -------------------- | ---------------------------------------------- | --------------------------- |
|
||||
| `gamedata_furnidata` | One row per furni/clothing item (habbo furnidata_json shape) | `(source, sprite_id)` |
|
||||
| `gamedata_docs` | Generic JSON document store (per-key documents) | `(category, doc_key)` |
|
||||
| `gamedata_texts` | External-texts style key/value pairs | `(category, text_key)` |
|
||||
|
||||
The bulk importer (`scripts/bulk-import-json.ts`, run via `pnpm db:bulk`) is DB-driven and fast precisely because each chunk is a *single* multi-row `INSERT … VALUES () ()… ON DUPLICATE KEY UPDATE`, so a 50 MB dump is a few dozen statements instead of hundreds of thousands of round-trips. Flags:
|
||||
|
||||
| Flag | Default | Description |
|
||||
| -------------------- | ----------- | ------------------------------------------------- |
|
||||
| `--file=<path>` | — (required)| JSON file (habbo furnidata_json or flat array) |
|
||||
| `--table=<name>` | `furnidata` | One of `furnidata` \| `docs` \| `texts` |
|
||||
| `--source=<x>` | `habbo` | `source` value for `furnidata` |
|
||||
| `--category=<x>` | `default` | `category` value for `docs` / `texts` |
|
||||
| `--chunk-size=N` | `2000` | Rows per multi-row INSERT |
|
||||
| `--limit=N` | `0` | Stop after N rows (dry-test) |
|
||||
| `--truncate` | off | DELETE rows for this source/category first |
|
||||
|
||||
The script sets `FOREIGN_KEY_CHECKS=0` for the session and is safe to interrupt: each chunk commits on its own, and `ON DUPLICATE KEY UPDATE` makes re-runs overwrite instead of appending.
|
||||
|
||||
---
|
||||
|
||||
## DragonflyDB (Caching, Rate Limiting, SSE)
|
||||
@@ -503,6 +574,8 @@ slow_query_log_file = /var/log/mysql/mariadb-slow.log
|
||||
|
||||
The slow-query log surfaces hot-spot SQL for a follow-up index/query audit. The CMS connection pool itself is already tuned (pool size 25, `connection_limit` in `DATABASE_URL`).
|
||||
|
||||
For bulk-written tables (game-data JSON, imports) a containerized **`mariadb-turbo`** variant is available — see [MariaDB Turbo](#mariadb-turbo-bulk-loads).
|
||||
|
||||
### 3. Asset caching headers
|
||||
|
||||
| Path | Cache-Control |
|
||||
@@ -553,6 +626,8 @@ pm2 restart next
|
||||
| `pnpm db:generate` | Draft SQL via drizzle-kit → `drizzle/drafts/` |
|
||||
| `pnpm db:studio` | Drizzle Studio (dev) |
|
||||
| `pnpm db:introspect` | drizzle-kit introspect (draft) |
|
||||
| `pnpm db:bulk` | Batch-import >50 MB JSON via `scripts/bulk-import-json.ts` |
|
||||
| `pnpm db:up` / `pnpm db:down` | Start / stop the `mariadb-turbo` container |
|
||||
| `pnpm gamedata:compress` | Pre-compress large gamedata JSON to `.gz` (gzip_static) |
|
||||
| `pnpm analyze` | Build + open bundle analyzer |
|
||||
| `pnpm jobs:worker` | Start background task worker |
|
||||
@@ -589,12 +664,14 @@ pm2 restart next
|
||||
│ └── drafts/ # drizzle-kit generate output (never auto-applied)
|
||||
├── scripts/
|
||||
│ ├── apply-migrations.ts # SQL migration runner (apply + status)
|
||||
│ ├── bulk-import-json.ts # 50 MB+ JSON importer (2000-row chunks, UPSERT)
|
||||
│ ├── jobs-worker.ts # Background task scheduler
|
||||
│ ├── generate-drizzle-schema.mjs # Regen src/db/schema.ts from schema + live DB
|
||||
│ └── compress-gamedata.mjs # Pre-compress large gamedata JSON (gzip_static)
|
||||
├── src/
|
||||
│ ├── db/
|
||||
│ │ ├── schema.ts # Drizzle ORM schema (committed — runtime data layer)
|
||||
│ │ ├── schema-gamedata.ts # Large-JSON storage schema (furnidata/docs/texts)
|
||||
│ │ └── relations.ts # Drizzle relations
|
||||
│ ├── app/ # Next.js App Router (pages & API routes)
|
||||
│ ├── actions/ # Server Actions
|
||||
|
||||
Reference in new issue
Block a user