feat(catalog): add scripts/catalog-sync.ts CLI + interactive menu
CI / check (push) Successful in 32s
CI / tests-integration (push) Successful in 1m55s
CI / tests-unit (push) Successful in 2m1s
CI / tests-imaging-browsers (push) Successful in 2m7s
CI / tests-ui (push) Successful in 2m44s
CI / preflight (push) Skipped
CI / tests-quality (push) Successful in 1m46s
CI / deploy (push) Successful in 3m28s
CI / check (push) Successful in 32s
CI / tests-integration (push) Successful in 1m55s
CI / tests-unit (push) Successful in 2m1s
CI / tests-imaging-browsers (push) Successful in 2m7s
CI / tests-ui (push) Successful in 2m44s
CI / preflight (push) Skipped
CI / tests-quality (push) Successful in 1m46s
CI / deploy (push) Successful in 3m28s
Updating the catalog was only possible from the admin "Catalog audit" page, which streams `runCatalogAudit` over SSE and always compares against every enabled clone source. This adds the same engine to the terminal so it can be run by hand or from cron: - `scripts/catalog-sync.ts` — CLI driver around `runCatalogAudit`. Prints the live progress, the per-source "on the source but not in this hotel" counts and a summary block; generated catalog SQL is saved to `storage/catalog-sync/*.sql` (gitignored). Supports `--list-sources`, `--sources=<a,b>`, `--icons`, `--nitros`, `--furnidata`, `--structure`, `--check-ids`, `--sql`, `--apply-sql`, `--organize-sql`, `--all`, `--json` and `--fail-on-errors`. - `scripts/catalog-sync.sh` — interactive menu (`pnpm catalog:sync`). Lists the configured sources (so the menu always matches `clone_sources`), lets Habbo (COM) be taken with Enter, and offers audit-only through full repair + applying the catalog SQL, each with a distinct confirmation. With no arguments it needs a terminal; with any flag it passes straight through to the CLI. - `runCatalogAudit` gains `sourceIds` to restrict which sites are compared; omitting it keeps the previous "compare against everything enabled" behaviour. An audit writes nothing, and `--apply-sql` is the only flag that changes the catalog. Verified against the live DB: 16370 items_base rows, 16479 catalog items, 2871 Habbo (COM) entries missing from this hotel.
This commit is contained in:
1 parent
61107848f8
commit
4b575d3a1d
7 files changed
+739
-1
No files matched your search
@@ -1161,6 +1161,7 @@ the headroom left for everything else on the machine; the effective ceiling is
|
|||||||
| `pnpm db:studio` | Drizzle Studio (dev) |
|
| `pnpm db:studio` | Drizzle Studio (dev) |
|
||||||
| `pnpm db:introspect` | drizzle-kit introspect (draft) |
|
| `pnpm db:introspect` | drizzle-kit introspect (draft) |
|
||||||
| `pnpm db:bulk` | Batch-import >50 MB JSON via `scripts/bulk-import-json.ts` |
|
| `pnpm db:bulk` | Batch-import >50 MB JSON via `scripts/bulk-import-json.ts` |
|
||||||
|
| `pnpm catalog:sync` | Sync the catalog against a clone source (interactive) |
|
||||||
| `pnpm db:up` / `pnpm db:down` | Start / stop the `mariadb-turbo` container |
|
| `pnpm db:up` / `pnpm db:down` | Start / stop the `mariadb-turbo` container |
|
||||||
| `pnpm gamedata:compress` | Pre-compress large gamedata JSON to `.gz` (gzip_static) |
|
| `pnpm gamedata:compress` | Pre-compress large gamedata JSON to `.gz` (gzip_static) |
|
||||||
| `pnpm analyze` | Build + report per-route bundle sizes |
|
| `pnpm analyze` | Build + report per-route bundle sizes |
|
||||||
|
|||||||
@@ -29,6 +29,8 @@
|
|||||||
"db:schema:generate": "node scripts/generate-drizzle-schema.mjs",
|
"db:schema:generate": "node scripts/generate-drizzle-schema.mjs",
|
||||||
"db:migrate": "tsx scripts/apply-migrations.ts",
|
"db:migrate": "tsx scripts/apply-migrations.ts",
|
||||||
"db:migrate:status": "tsx scripts/apply-migrations.ts --status",
|
"db:migrate:status": "tsx scripts/apply-migrations.ts --status",
|
||||||
|
"catalog:sync": "bash scripts/catalog-sync.sh",
|
||||||
|
"catalog:sync:cli": "node --conditions=react-server --import tsx scripts/catalog-sync.ts",
|
||||||
"db:studio": "drizzle-kit studio",
|
"db:studio": "drizzle-kit studio",
|
||||||
"gamedata:compress": "node scripts/compress-gamedata.mjs",
|
"gamedata:compress": "node scripts/compress-gamedata.mjs",
|
||||||
"hk:matrix:check": "tsx scripts/verify-housekeeping-matrix.ts",
|
"hk:matrix:check": "tsx scripts/verify-housekeeping-matrix.ts",
|
||||||
|
|||||||
@@ -25,6 +25,7 @@ This directory contains utility scripts for managing the EpicNext-Cms project.
|
|||||||
| `doctor.sh` | Runs a system diagnostic report. |
|
| `doctor.sh` | Runs a system diagnostic report. |
|
||||||
| `setup-cron.sh` | Configures automated cron jobs. |
|
| `setup-cron.sh` | Configures automated cron jobs. |
|
||||||
| `alert.sh` | Sends notifications to a configured webhook. |
|
| `alert.sh` | Sends notifications to a configured webhook. |
|
||||||
|
| `catalog-sync.sh` | Interactive catalog update against a clone source (see below). |
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
Run any script directly, or use the dashboard:
|
Run any script directly, or use the dashboard:
|
||||||
@@ -46,3 +47,32 @@ It manages the `pnpm` and `Node.js` rows of the System Requirements table and
|
|||||||
the "Core stack" line. The CI `check` job runs the `--check` variant, so a
|
the "Core stack" line. The CI `check` job runs the `--check` variant, so a
|
||||||
version change cannot reach `main` with a stale README — the pipeline fails
|
version change cannot reach `main` with a stale README — the pipeline fails
|
||||||
until you run `pnpm readme:versions`.
|
until you run `pnpm readme:versions`.
|
||||||
|
|
||||||
|
### Updating the catalog (`catalog-sync.sh`)
|
||||||
|
|
||||||
|
Compares this hotel's catalog against a clone source (Habbo and the retros) and
|
||||||
|
fixes what is broken: missing furniture icons, missing `.nitro` bundles,
|
||||||
|
`FurnitureData.json` drift, orphaned/duplicate catalog rows, and items that
|
||||||
|
exist in `items_base` but have no catalog offer.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm catalog:sync # interactive menu
|
||||||
|
./scripts/catalog-sync.sh --list-sources # which sites can be compared
|
||||||
|
./scripts/catalog-sync.sh --all --sources=default-habbo-com
|
||||||
|
./scripts/catalog-sync.sh --all --sql --apply-sql # catalogs changes are written
|
||||||
|
```
|
||||||
|
|
||||||
|
The menu asks which sources to compare (Habbo (COM) is the preset) and what may
|
||||||
|
be written. With no arguments it needs a terminal; with any flag it skips the
|
||||||
|
menu, so it is usable from cron.
|
||||||
|
|
||||||
|
Safety notes:
|
||||||
|
|
||||||
|
- **An audit writes nothing** — no repair flag means read-only.
|
||||||
|
- `--apply-sql` is the only flag that changes the catalog, and the generated SQL
|
||||||
|
is always written to `storage/catalog-sync/*.sql` (gitignored) so it can be
|
||||||
|
reviewed or replayed by hand.
|
||||||
|
- The engine is the same one the admin *Catalog audit* page streams
|
||||||
|
(`runCatalogAudit`), so a terminal run and an in-admin run agree.
|
||||||
|
- `--json` prints one machine-readable report; `--fail-on-errors` exits `3` when
|
||||||
|
error-severity issues remain, which is what a scheduled check should use.
|
||||||
Executable
+157
@@ -0,0 +1,157 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Update the hotel catalog from a clone source (Habbo, retros, …).
|
||||||
|
#
|
||||||
|
# Two ways to use it:
|
||||||
|
# ./scripts/catalog-sync.sh interactive menu
|
||||||
|
# ./scripts/catalog-sync.sh --all --sql same flags as the CLI, no menu
|
||||||
|
#
|
||||||
|
# The menu asks which sites to compare (Habbo is the preset) and what may be
|
||||||
|
# written. An audit is read-only; only a repair option touches the catalog, and
|
||||||
|
# applying the generated catalog SQL always needs its own choice.
|
||||||
|
|
||||||
|
cd "$(dirname "$0")/.." || exit 1
|
||||||
|
CLI="scripts/catalog-sync.ts"
|
||||||
|
# The react-server export condition resolves `server-only` to its no-op build,
|
||||||
|
# exactly like `pnpm jobs:worker` — the audit engine imports server modules.
|
||||||
|
RUN=(node --conditions=react-server --import tsx "$CLI")
|
||||||
|
|
||||||
|
# Any flag passed straight through means "no menu", so the same entry point
|
||||||
|
# works for cron/CI where only the flag path is safe.
|
||||||
|
if [ $# -gt 0 ]; then
|
||||||
|
exec "${RUN[@]}" "$@"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ ! -t 0 ]; then
|
||||||
|
echo "Not a terminal. Use the non-interactive form, e.g.:" >&2
|
||||||
|
echo " $0 --all --sources=default-habbo-com" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "=== Catalog sync ==="
|
||||||
|
echo
|
||||||
|
|
||||||
|
# ── Step 1: which sites to compare ───────────────────────────────────────────
|
||||||
|
# Ask the CLI for the configured sources so the menu always matches the live
|
||||||
|
# `clone_sources` setting (and its defaults) instead of a hardcoded list.
|
||||||
|
mapfile -t SOURCE_ROWS < <("${RUN[@]}" --list-sources --tsv)
|
||||||
|
if [ "${#SOURCE_ROWS[@]}" -eq 0 ]; then
|
||||||
|
echo "No clone sources are configured." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
IDS=()
|
||||||
|
NAMES=()
|
||||||
|
echo "Compare this catalog against:"
|
||||||
|
for i in "${!SOURCE_ROWS[@]}"; do
|
||||||
|
IFS=$'\t' read -r SOURCE_ID SOURCE_NAME _SOURCE_URL <<<"${SOURCE_ROWS[$i]}"
|
||||||
|
IDS+=("$SOURCE_ID")
|
||||||
|
NAMES+=("$SOURCE_NAME")
|
||||||
|
printf " %2d) %-24s %s\n" "$((i + 1))" "$SOURCE_NAME" "$SOURCE_ID"
|
||||||
|
done
|
||||||
|
echo
|
||||||
|
echo " a) all sources above"
|
||||||
|
echo " Enter) preset: ${NAMES[0]}"
|
||||||
|
|
||||||
|
read -r -p "Sources [a, 1-${#IDS[@]}, Enter]: " SOURCE_CHOICE
|
||||||
|
|
||||||
|
SELECTED_FLAG=""
|
||||||
|
case "$SOURCE_CHOICE" in
|
||||||
|
"" | "Habbo" | "habbo" | "habbo.com")
|
||||||
|
# Habbo is the first default preset (Habbo (COM)).
|
||||||
|
SELECTED_FLAG="--sources=${IDS[0]}"
|
||||||
|
SOURCE_LABEL="${NAMES[0]}"
|
||||||
|
;;
|
||||||
|
a | A | all)
|
||||||
|
SOURCE_LABEL="all sources"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
SELECTED_IDS=()
|
||||||
|
IFS=', ' read -r -a CHOICES <<<"$SOURCE_CHOICE"
|
||||||
|
for CHOICE in "${CHOICES[@]}"; do
|
||||||
|
[ -z "$CHOICE" ] && continue
|
||||||
|
case "$CHOICE" in
|
||||||
|
*[!0-9]* | "")
|
||||||
|
echo "Invalid choice: '$CHOICE'" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
if [ "$CHOICE" -lt 1 ] || [ "$CHOICE" -gt "${#IDS[@]}" ]; then
|
||||||
|
echo "Out of range: $CHOICE" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
SELECTED_IDS+=("${IDS[$((CHOICE - 1))]}")
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
if [ "${#SELECTED_IDS[@]}" -eq 0 ]; then
|
||||||
|
echo "No sources selected." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
SELECTED_FLAG="--sources=$(IFS=,; echo "${SELECTED_IDS[*]}")"
|
||||||
|
SOURCE_LABEL="${#SELECTED_IDS[@]} source(s)"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# ── Step 2: what may be changed ──────────────────────────────────────────────
|
||||||
|
cat <<EOF
|
||||||
|
|
||||||
|
What should be done? (comparing with: $SOURCE_LABEL)
|
||||||
|
1) Audit only — nothing is written
|
||||||
|
2) Repair furniture icons + nitro bundles
|
||||||
|
3) 2 + FurnitureData.json + catalog structure
|
||||||
|
4) 3 + generate the catalog SQL (saved, not applied)
|
||||||
|
5) 4 + APPLY the catalog SQL to the catalog
|
||||||
|
q) Cancel
|
||||||
|
EOF
|
||||||
|
|
||||||
|
read -r -p "Action [1]: " ACTION_CHOICE
|
||||||
|
ACTION_CHOICE="${ACTION_CHOICE:-1}"
|
||||||
|
|
||||||
|
ACTIONS=(audit)
|
||||||
|
CONFIRM=""
|
||||||
|
# Empty for the audit-only branch on purpose: no repair flags are passed.
|
||||||
|
ACTION_FLAGS=()
|
||||||
|
case "$ACTION_CHOICE" in
|
||||||
|
1)
|
||||||
|
ACTIONS=(audit "[nothing written]")
|
||||||
|
;;
|
||||||
|
2)
|
||||||
|
ACTIONS=(audit "icons + nitro bundles downloaded")
|
||||||
|
ACTION_FLAGS=(--icons --nitros)
|
||||||
|
;;
|
||||||
|
3)
|
||||||
|
ACTIONS=(audit "icons + nitro bundles + FurnitureData.json + structure")
|
||||||
|
ACTION_FLAGS=(--all)
|
||||||
|
;;
|
||||||
|
4)
|
||||||
|
ACTIONS=(audit "everything + catalog SQL generated")
|
||||||
|
ACTION_FLAGS=(--all --sql)
|
||||||
|
;;
|
||||||
|
5)
|
||||||
|
ACTIONS=(audit "everything + catalog SQL APPLIED")
|
||||||
|
ACTION_FLAGS=(--all --sql --apply-sql)
|
||||||
|
CONFIRM="This writes to the catalog. Continue? [y/N]: "
|
||||||
|
;;
|
||||||
|
q | Q)
|
||||||
|
echo "Cancelled."
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Invalid action: '$ACTION_CHOICE'" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
echo
|
||||||
|
printf 'Running: %s against %s\n' "${ACTIONS[1]}" "$SOURCE_LABEL"
|
||||||
|
if [ -n "$CONFIRM" ]; then
|
||||||
|
read -r -p "$CONFIRM" REPLY
|
||||||
|
if [ "$REPLY" != "y" ] && [ "$REPLY" != "Y" ]; then
|
||||||
|
echo "Cancelled."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
# shellcheck disable=SC2086
|
||||||
|
"${RUN[@]}" $SELECTED_FLAG "${ACTION_FLAGS[@]}"
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||||
|
import {
|
||||||
|
DEFAULT_CATALOG_SYNC_FLAGS,
|
||||||
|
describeActions,
|
||||||
|
parseCatalogSyncArgs,
|
||||||
|
writeGeneratedSql,
|
||||||
|
} from "./catalog-sync";
|
||||||
|
|
||||||
|
describe("catalog-sync argument parsing", () => {
|
||||||
|
it("defaults to a read-only audit with no filters", () => {
|
||||||
|
expect(parseCatalogSyncArgs([])).toEqual({
|
||||||
|
...DEFAULT_CATALOG_SYNC_FLAGS,
|
||||||
|
sourceIds: [],
|
||||||
|
});
|
||||||
|
expect(describeActions(parseCatalogSyncArgs([]))).toBe("audit");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("accepts comma-separated source ids", () => {
|
||||||
|
const flags = parseCatalogSyncArgs([
|
||||||
|
"--sources=default-habbo-com,default-wibbo",
|
||||||
|
]);
|
||||||
|
expect(flags.sourceIds).toEqual(["default-habbo-com", "default-wibbo"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("accepts display names, including ones with spaces", () => {
|
||||||
|
const flags = parseCatalogSyncArgs(["--sources=Habbo (COM), Wibbo"]);
|
||||||
|
expect(flags.sourceIds).toEqual(["Habbo (COM)", "Wibbo"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("trims and drops empty source entries", () => {
|
||||||
|
expect(parseCatalogSyncArgs(["--sources= a , ,b,"]).sourceIds).toEqual([
|
||||||
|
"a",
|
||||||
|
"b",
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("turns --all into every repair step", () => {
|
||||||
|
const flags = parseCatalogSyncArgs(["--all"]);
|
||||||
|
expect(flags.repairIcons).toBe(true);
|
||||||
|
expect(flags.repairNitros).toBe(true);
|
||||||
|
expect(flags.repairFurniData).toBe(true);
|
||||||
|
expect(flags.repairStructure).toBe(true);
|
||||||
|
expect(flags.checkFurniDataIds).toBe(true);
|
||||||
|
// Generating/applying SQL is never implied by a repair.
|
||||||
|
expect(flags.generateSql).toBe(false);
|
||||||
|
expect(flags.applySql).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("treats applying SQL as implying generation", () => {
|
||||||
|
expect(parseCatalogSyncArgs(["--apply-sql"]).generateSql).toBe(true);
|
||||||
|
expect(parseCatalogSyncArgs(["--organize-sql"]).generateSql).toBe(true);
|
||||||
|
expect(describeActions(parseCatalogSyncArgs(["--apply-sql"]))).toBe(
|
||||||
|
"audit + apply-sql",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("accepts the alias spellings of the repair flags", () => {
|
||||||
|
const flags = parseCatalogSyncArgs([
|
||||||
|
"--repair-icons",
|
||||||
|
"--repair-nitros",
|
||||||
|
"--repair-furnidata",
|
||||||
|
"--repair-structure",
|
||||||
|
]);
|
||||||
|
expect(flags.repairIcons).toBe(true);
|
||||||
|
expect(flags.repairNitros).toBe(true);
|
||||||
|
expect(flags.repairFurniData).toBe(true);
|
||||||
|
expect(flags.repairStructure).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("ignores a bare -- separator", () => {
|
||||||
|
expect(parseCatalogSyncArgs(["--", "--icons"]).repairIcons).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rejects an unknown flag", () => {
|
||||||
|
expect(() => parseCatalogSyncArgs(["--nope"])).toThrow(/Unknown flag/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rejects --sources without a value", () => {
|
||||||
|
expect(() => parseCatalogSyncArgs(["--sources"])).toThrow(/needs a value/);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("generated catalog SQL output", () => {
|
||||||
|
let cwd = "";
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
cwd = process.cwd();
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
vi.restoreAllMocks();
|
||||||
|
process.chdir(cwd);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("writes the SQL under storage/catalog-sync with a trailing newline", async () => {
|
||||||
|
let target = "";
|
||||||
|
try {
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), "catalog-sync-sql-"));
|
||||||
|
target = dir;
|
||||||
|
process.chdir(dir);
|
||||||
|
const file = await writeGeneratedSql("INSERT INTO `catalog_items` (1);");
|
||||||
|
expect(file.startsWith("storage/catalog-sync/")).toBe(true);
|
||||||
|
expect(file.endsWith(".sql")).toBe(true);
|
||||||
|
expect(readFileSync(join(dir, file), "utf8")).toBe(
|
||||||
|
"INSERT INTO `catalog_items` (1);\n",
|
||||||
|
);
|
||||||
|
} finally {
|
||||||
|
if (target) rmSync(target, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,423 @@
|
|||||||
|
// Command-line driver for the catalog audit/repair pipeline.
|
||||||
|
//
|
||||||
|
// The same engine the admin "Catalog audit" UI streams from (`runCatalogAudit`),
|
||||||
|
// for the terminal: audit the hotel catalog against one or more clone sources
|
||||||
|
// (Habbo and the retros), fix the furniture that is broken, and bring missing
|
||||||
|
// items into the catalog — optionally writing and applying the catalog SQL.
|
||||||
|
//
|
||||||
|
// Usage:
|
||||||
|
// pnpm catalog:sync # interactive menu (scripts/catalog-sync.sh)
|
||||||
|
// pnpm exec tsx scripts/catalog-sync.ts --list-sources
|
||||||
|
// pnpm exec tsx scripts/catalog-sync.ts --sources="Habbo (COM)" # audit only
|
||||||
|
// pnpm exec tsx scripts/catalog-sync.ts --all --sources=default-habbo-com
|
||||||
|
// pnpm exec tsx scripts/catalog-sync.ts --all --sql --apply-sql # + write/apply SQL
|
||||||
|
//
|
||||||
|
// Flags:
|
||||||
|
// --list-sources print the configured clone sources and exit
|
||||||
|
// --tsv with the above, print `id\tname\tfurnidataUrl` rows
|
||||||
|
// --sources=<a,b> compare against these sources (id or name, comma-separated)
|
||||||
|
// --icons re-download missing furniture icons from the sources
|
||||||
|
// --nitros re-download missing .nitro / swf bundles
|
||||||
|
// --furnidata repair FurnitureData.json (add missing, drop duplicates)
|
||||||
|
// --structure repair orphaned / duplicate catalog rows
|
||||||
|
// --check-ids check FurnitureData.json for spriteId conflicts
|
||||||
|
// --sql generate the catalog SQL for items missing an offer
|
||||||
|
// --apply-sql apply the generated SQL (implies --sql)
|
||||||
|
// --organize-sql order the generated SQL by catalog page (implies --sql)
|
||||||
|
// --all --icons --nitros --furnidata --structure --check-ids
|
||||||
|
// --json print one machine-readable JSON report
|
||||||
|
// --fail-on-errors exit 3 when error-severity issues remain
|
||||||
|
//
|
||||||
|
// Audit is read-only by default: nothing is written until one of the repair
|
||||||
|
// flags is passed, and `--apply-sql` is the only flag that changes the catalog.
|
||||||
|
|
||||||
|
import "./load-env";
|
||||||
|
import { mkdir, writeFile } from "node:fs/promises";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { pathToFileURL } from "node:url";
|
||||||
|
import type { AuditEvent, AuditSummary } from "@/lib/services/catalog-audit";
|
||||||
|
|
||||||
|
/** Where generated catalog SQL is written (`storage/` is gitignored). */
|
||||||
|
function sqlOutputDir(): string {
|
||||||
|
return join(process.cwd(), "storage", "catalog-sync");
|
||||||
|
}
|
||||||
|
|
||||||
|
export type SourceRef = { id: string; name: string; furnidataUrl: string };
|
||||||
|
|
||||||
|
export type AuditIssue = NonNullable<AuditEvent["issues"]>[number];
|
||||||
|
export type MissingFromSource = NonNullable<
|
||||||
|
AuditEvent["missingFromSources"]
|
||||||
|
>[number];
|
||||||
|
|
||||||
|
export interface AuditReport {
|
||||||
|
sources: SourceRef[];
|
||||||
|
summary?: AuditSummary;
|
||||||
|
issues: AuditIssue[];
|
||||||
|
missingFromSources: MissingFromSource[];
|
||||||
|
sqlFile?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogSyncFlags {
|
||||||
|
listSources: boolean;
|
||||||
|
tsv: boolean;
|
||||||
|
sourceIds: string[];
|
||||||
|
repairIcons: boolean;
|
||||||
|
repairNitros: boolean;
|
||||||
|
repairFurniData: boolean;
|
||||||
|
repairStructure: boolean;
|
||||||
|
checkFurniDataIds: boolean;
|
||||||
|
generateSql: boolean;
|
||||||
|
applySql: boolean;
|
||||||
|
organizeSql: boolean;
|
||||||
|
json: boolean;
|
||||||
|
failOnErrors: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const DEFAULT_CATALOG_SYNC_FLAGS: Omit<CatalogSyncFlags, "sourceIds"> = {
|
||||||
|
listSources: false,
|
||||||
|
tsv: false,
|
||||||
|
repairIcons: false,
|
||||||
|
repairNitros: false,
|
||||||
|
repairFurniData: false,
|
||||||
|
repairStructure: false,
|
||||||
|
checkFurniDataIds: false,
|
||||||
|
generateSql: false,
|
||||||
|
applySql: false,
|
||||||
|
organizeSql: false,
|
||||||
|
json: false,
|
||||||
|
failOnErrors: false,
|
||||||
|
};
|
||||||
|
|
||||||
|
const BOOLEAN_FLAGS: Record<string, keyof CatalogSyncFlags> = {
|
||||||
|
"--list-sources": "listSources",
|
||||||
|
"--tsv": "tsv",
|
||||||
|
"--icons": "repairIcons",
|
||||||
|
"--repair-icons": "repairIcons",
|
||||||
|
"--nitros": "repairNitros",
|
||||||
|
"--repair-nitros": "repairNitros",
|
||||||
|
"--furnidata": "repairFurniData",
|
||||||
|
"--repair-furnidata": "repairFurniData",
|
||||||
|
"--structure": "repairStructure",
|
||||||
|
"--repair-structure": "repairStructure",
|
||||||
|
"--check-ids": "checkFurniDataIds",
|
||||||
|
"--sql": "generateSql",
|
||||||
|
"--apply-sql": "applySql",
|
||||||
|
"--organize-sql": "organizeSql",
|
||||||
|
"--json": "json",
|
||||||
|
"--fail-on-errors": "failOnErrors",
|
||||||
|
};
|
||||||
|
|
||||||
|
export function parseCatalogSyncArgs(raw: string[]): CatalogSyncFlags {
|
||||||
|
const flags: CatalogSyncFlags = {
|
||||||
|
...DEFAULT_CATALOG_SYNC_FLAGS,
|
||||||
|
sourceIds: [],
|
||||||
|
};
|
||||||
|
|
||||||
|
for (const arg of raw) {
|
||||||
|
if (arg === "--") continue;
|
||||||
|
|
||||||
|
if (arg === "--all") {
|
||||||
|
flags.repairIcons = true;
|
||||||
|
flags.repairNitros = true;
|
||||||
|
flags.repairFurniData = true;
|
||||||
|
flags.repairStructure = true;
|
||||||
|
flags.checkFurniDataIds = true;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const separator = arg.indexOf("=");
|
||||||
|
const name = separator >= 0 ? arg.slice(0, separator) : arg;
|
||||||
|
|
||||||
|
const booleanFlag = BOOLEAN_FLAGS[name];
|
||||||
|
if (booleanFlag) {
|
||||||
|
flags[booleanFlag] = true as never;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (name === "--sources" || name === "--source") {
|
||||||
|
if (separator < 0)
|
||||||
|
throw new Error(
|
||||||
|
`${name} needs a value, e.g. --sources=default-habbo-com`,
|
||||||
|
);
|
||||||
|
const value = arg.slice(separator + 1);
|
||||||
|
for (const part of value
|
||||||
|
.split(",")
|
||||||
|
.map((entry) => entry.trim())
|
||||||
|
.filter(Boolean))
|
||||||
|
flags.sourceIds.push(part);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new Error(
|
||||||
|
`Unknown flag: ${arg} (run with --help for the supported flags)`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Applying SQL implies generating it, and organising it implies generating.
|
||||||
|
if (flags.applySql) flags.generateSql = true;
|
||||||
|
if (flags.organizeSql) flags.generateSql = true;
|
||||||
|
return flags;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Human-readable list of what the run will do. */
|
||||||
|
export function describeActions(flags: CatalogSyncFlags): string {
|
||||||
|
const actions = ["audit"];
|
||||||
|
if (flags.repairIcons) actions.push("icons");
|
||||||
|
if (flags.repairNitros) actions.push("nitros");
|
||||||
|
if (flags.repairFurniData) actions.push("furnidata");
|
||||||
|
if (flags.repairStructure) actions.push("structure");
|
||||||
|
if (flags.checkFurniDataIds) actions.push("id-check");
|
||||||
|
if (flags.applySql) actions.push("apply-sql");
|
||||||
|
else if (flags.generateSql) actions.push("generate-sql");
|
||||||
|
if (flags.organizeSql) actions.push("organised");
|
||||||
|
return actions.join(" + ");
|
||||||
|
}
|
||||||
|
|
||||||
|
/** True when the file is being executed, not imported by a test. */
|
||||||
|
function isEntryPoint(): boolean {
|
||||||
|
const entry = process.argv[1];
|
||||||
|
return (
|
||||||
|
Boolean(entry) && import.meta.url === pathToFileURL(String(entry)).href
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadSourceCatalogue(): Promise<SourceRef[]> {
|
||||||
|
const { listSources } = await import("@/lib/services/clone-sources");
|
||||||
|
const sources = await listSources();
|
||||||
|
return sources
|
||||||
|
.filter((source) => source.furnidataUrl)
|
||||||
|
.map((source) => ({
|
||||||
|
id: source.id,
|
||||||
|
name: source.name,
|
||||||
|
furnidataUrl: source.furnidataUrl,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
|
async function commandListSources(flags: CatalogSyncFlags): Promise<number> {
|
||||||
|
const sources = await loadSourceCatalogue();
|
||||||
|
|
||||||
|
if (flags.tsv) {
|
||||||
|
for (const source of sources)
|
||||||
|
console.log([source.id, source.name, source.furnidataUrl].join("\t"));
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`Configured clone sources (${sources.length}):`);
|
||||||
|
for (const source of sources)
|
||||||
|
console.log(` ${source.id}\t${source.name}\t${source.furnidataUrl}`);
|
||||||
|
console.log(
|
||||||
|
"\nCompare with: --sources=<id> (comma-separated; display names work too)",
|
||||||
|
);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Persist the generated SQL so it can be reviewed or replayed by hand. */
|
||||||
|
export async function writeGeneratedSql(sql: string): Promise<string> {
|
||||||
|
await mkdir(sqlOutputDir(), { recursive: true });
|
||||||
|
const stamp = new Date().toISOString().replace(/[:.]/g, "-");
|
||||||
|
const file = join(sqlOutputDir(), `catalog-${stamp}.sql`);
|
||||||
|
await writeFile(file, sql.endsWith("\n") ? sql : `${sql}\n`, "utf8");
|
||||||
|
return file.replace(`${process.cwd()}/`, "");
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function commandRun(flags: CatalogSyncFlags): Promise<number> {
|
||||||
|
const catalogue = await loadSourceCatalogue();
|
||||||
|
|
||||||
|
// Resolve the selection up-front so a typo fails before anything is written.
|
||||||
|
const known = new Set<string>();
|
||||||
|
for (const source of catalogue) {
|
||||||
|
known.add(source.id);
|
||||||
|
known.add(source.name);
|
||||||
|
}
|
||||||
|
const unknown = flags.sourceIds.filter((id) => !known.has(id));
|
||||||
|
if (unknown.length > 0) {
|
||||||
|
console.error(
|
||||||
|
`Unknown source: ${unknown.join(", ")}\nRun --list-sources to see the configured sites.`,
|
||||||
|
);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const selected = flags.sourceIds.length
|
||||||
|
? catalogue.filter(
|
||||||
|
(source) =>
|
||||||
|
flags.sourceIds.includes(source.id) ||
|
||||||
|
flags.sourceIds.includes(source.name),
|
||||||
|
)
|
||||||
|
: catalogue;
|
||||||
|
|
||||||
|
const { runCatalogAudit } = await import("@/lib/services/catalog-audit");
|
||||||
|
|
||||||
|
const report: AuditReport = {
|
||||||
|
sources: selected,
|
||||||
|
issues: [],
|
||||||
|
missingFromSources: [],
|
||||||
|
};
|
||||||
|
const quiet = flags.json;
|
||||||
|
const log = (line: string) => {
|
||||||
|
if (!quiet) console.log(line);
|
||||||
|
};
|
||||||
|
const started = Date.now();
|
||||||
|
let pendingSql: string | undefined;
|
||||||
|
|
||||||
|
log(
|
||||||
|
`Catalog sync — ${selected.length} source(s), ${describeActions(flags)}` +
|
||||||
|
(flags.applySql ? " [SQL WILL BE APPLIED]" : ""),
|
||||||
|
);
|
||||||
|
|
||||||
|
await runCatalogAudit(
|
||||||
|
(event) => {
|
||||||
|
switch (event.type) {
|
||||||
|
case "progress":
|
||||||
|
log(`· ${event.message ?? ""}`);
|
||||||
|
break;
|
||||||
|
case "items_loaded":
|
||||||
|
log(`· items_base: ${event.total} rows`);
|
||||||
|
break;
|
||||||
|
case "catalog_loaded":
|
||||||
|
log(`· catalog_items: ${event.total} rows`);
|
||||||
|
break;
|
||||||
|
case "sql_ready":
|
||||||
|
log(`· catalog SQL generated: ${event.sqlCount ?? 0} statement(s)`);
|
||||||
|
pendingSql = event.sql;
|
||||||
|
break;
|
||||||
|
case "sql_applied":
|
||||||
|
log(
|
||||||
|
`· catalog SQL applied: ${event.applied ?? 0} (failed ${event.sqlFailed ?? 0})`,
|
||||||
|
);
|
||||||
|
break;
|
||||||
|
case "furnidata_repair_complete":
|
||||||
|
log(
|
||||||
|
`· FurnitureData.json: +${event.furniData?.added ?? 0} entries, ${event.furniData?.removedDuplicates ?? 0} duplicate(s) removed`,
|
||||||
|
);
|
||||||
|
break;
|
||||||
|
case "structure_repair_complete":
|
||||||
|
log(
|
||||||
|
`· structure: ${event.structure?.orphanedRemoved ?? 0} orphaned removed, ${event.structure?.duplicatesMerged ?? 0} duplicate(s) merged, ${event.structure?.remappedReferences ?? 0} reference(s) remapped`,
|
||||||
|
);
|
||||||
|
break;
|
||||||
|
case "repair_complete":
|
||||||
|
log(
|
||||||
|
`· asset repair: ${event.repair?.succeeded ?? 0} ok, ${event.repair?.failed ?? 0} failed, ${event.repair?.skipped ?? 0} skipped`,
|
||||||
|
);
|
||||||
|
break;
|
||||||
|
case "batch_complete":
|
||||||
|
report.issues = event.issues ?? [];
|
||||||
|
report.missingFromSources = event.missingFromSources ?? [];
|
||||||
|
report.summary = event.summary;
|
||||||
|
break;
|
||||||
|
case "error":
|
||||||
|
console.error(`! ${event.message ?? "audit error"}`);
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
sourceIds: flags.sourceIds,
|
||||||
|
repair: flags.repairIcons,
|
||||||
|
repairNitros: flags.repairNitros,
|
||||||
|
sql: flags.generateSql,
|
||||||
|
applySql: flags.applySql,
|
||||||
|
organizeSql: flags.organizeSql,
|
||||||
|
repairFurniData: flags.repairFurniData,
|
||||||
|
repairStructure: flags.repairStructure,
|
||||||
|
checkFurniDataIds: flags.checkFurniDataIds,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
if (pendingSql?.trim()) report.sqlFile = await writeGeneratedSql(pendingSql);
|
||||||
|
|
||||||
|
if (flags.json) {
|
||||||
|
console.log(
|
||||||
|
JSON.stringify(
|
||||||
|
{ flags, elapsedMs: Date.now() - started, ...report },
|
||||||
|
null,
|
||||||
|
2,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
printSummary(report, started);
|
||||||
|
if (report.sqlFile) console.log(`Catalog SQL: ${report.sqlFile}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const errors = report.issues.filter((issue) => issue.severity === "error");
|
||||||
|
if (flags.failOnErrors && errors.length > 0) {
|
||||||
|
if (!quiet) console.error(`${errors.length} error(s) remain unresolved.`);
|
||||||
|
return 3;
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
function printSummary(report: AuditReport, started: number): void {
|
||||||
|
const summary = report.summary;
|
||||||
|
const seconds = ((Date.now() - started) / 1000).toFixed(1);
|
||||||
|
|
||||||
|
console.log(`\nDone in ${seconds}s`);
|
||||||
|
if (!summary) {
|
||||||
|
console.log("No summary was produced (the audit stopped early).");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(` items_base ${summary.totalItemsBase}`);
|
||||||
|
console.log(` catalog_items ${summary.totalCatalogItems}`);
|
||||||
|
console.log(` missing catalog entry ${summary.missingCatalog}`);
|
||||||
|
console.log(` orphaned catalog rows ${summary.orphanedCatalog}`);
|
||||||
|
console.log(` duplicate classnames ${summary.duplicates}`);
|
||||||
|
console.log(
|
||||||
|
` missing icons ${summary.missingIcon} (repaired ${summary.iconsRepaired}, still missing ${summary.iconsStillMissing})`,
|
||||||
|
);
|
||||||
|
console.log(
|
||||||
|
` missing nitro bundles ${summary.missingNitro} (repaired ${summary.nitrosRepaired}, still missing ${summary.nitrosStillMissing})`,
|
||||||
|
);
|
||||||
|
console.log(
|
||||||
|
` catalog SQL ${summary.catalogSqlCount} stmt, applied ${summary.catalogSqlApplied}`,
|
||||||
|
);
|
||||||
|
console.log(` furnidata added ${summary.furniDataAdded}`);
|
||||||
|
console.log(
|
||||||
|
` not restorable ${summary.unrepairableCount} (no icon/bundle on any compared source)`,
|
||||||
|
);
|
||||||
|
|
||||||
|
if (report.missingFromSources.length > 0) {
|
||||||
|
const bySource = new Map<string, number>();
|
||||||
|
for (const entry of report.missingFromSources)
|
||||||
|
bySource.set(entry.sourceName, (bySource.get(entry.sourceName) ?? 0) + 1);
|
||||||
|
console.log(
|
||||||
|
`\n on compared sources but not in this hotel: ${report.missingFromSources.length}`,
|
||||||
|
);
|
||||||
|
for (const [name, count] of bySource)
|
||||||
|
console.log(` ${name.padEnd(24)} ${count}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const errors = report.issues.filter((issue) => issue.severity === "error");
|
||||||
|
if (errors.length > 0) {
|
||||||
|
console.log(`\n unresolved errors: ${errors.length}`);
|
||||||
|
for (const issue of errors.slice(0, 20))
|
||||||
|
console.log(` ${issue.classname ?? issue.type}: ${issue.message}`);
|
||||||
|
if (errors.length > 20) console.log(` … and ${errors.length - 20} more`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main(): Promise<number> {
|
||||||
|
const flags = parseCatalogSyncArgs(process.argv.slice(2));
|
||||||
|
try {
|
||||||
|
if (flags.listSources) return await commandListSources(flags);
|
||||||
|
return await commandRun(flags);
|
||||||
|
} catch (error) {
|
||||||
|
console.error(
|
||||||
|
`Catalog sync failed: ${error instanceof Error ? error.message : String(error)}`,
|
||||||
|
);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (isEntryPoint()) {
|
||||||
|
main().then(
|
||||||
|
(code) => {
|
||||||
|
process.exit(code);
|
||||||
|
},
|
||||||
|
(error) => {
|
||||||
|
console.error(error);
|
||||||
|
process.exit(1);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -141,6 +141,11 @@ export interface CatalogAuditOptions {
|
|||||||
repairFurniData?: boolean;
|
repairFurniData?: boolean;
|
||||||
repairStructure?: boolean;
|
repairStructure?: boolean;
|
||||||
checkFurniDataIds?: boolean;
|
checkFurniDataIds?: boolean;
|
||||||
|
/**
|
||||||
|
* Restrict the clone-source comparison to these sources (by id or name).
|
||||||
|
* Empty/omitted keeps every enabled source, which is the default.
|
||||||
|
*/
|
||||||
|
sourceIds?: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function runCatalogAudit(
|
export async function runCatalogAudit(
|
||||||
@@ -838,7 +843,13 @@ export async function runCatalogAudit(
|
|||||||
|
|
||||||
onEvent?.({ type: "progress", message: "Comparing with clone sources…" });
|
onEvent?.({ type: "progress", message: "Comparing with clone sources…" });
|
||||||
|
|
||||||
const sources = await listSources();
|
const allSources = await listSources();
|
||||||
|
// An explicit selection compares against just those sites; leaving it empty
|
||||||
|
// keeps the historical "compare against everything enabled" behaviour.
|
||||||
|
const wantedIds = new Set(options?.sourceIds ?? []);
|
||||||
|
const sources = wantedIds.size
|
||||||
|
? allSources.filter((s) => wantedIds.has(s.id) || wantedIds.has(s.name))
|
||||||
|
: allSources;
|
||||||
const sourceResults = await Promise.all(
|
const sourceResults = await Promise.all(
|
||||||
sources
|
sources
|
||||||
.filter((s) => s.furnidataUrl)
|
.filter((s) => s.furnidataUrl)
|
||||||
|
|||||||
Reference in new issue
Block a user