Per single furni import, patchLocalizedFurniDataEntries ran for every
language and did a full 100k-entry JSON.parse + scan + JSON.stringify on the
main thread. That CPU spike is now offloaded to the translation worker
(init/prepare/finalize reuses the same pure core helpers), so importing
never blocks the event loop. The main thread only does async file I/O and the
network LibreTranslate calls. Falls back to the same helpers on the main
thread if no worker is available.
Also cache readFurniData() by file mtime+size so the 100k-entry JSON is
parsed once per change instead of on every import/fix/reconcile read
(invalidated on write).
The translation build's CPU-heavy work (deep clone of the 100k+ entry
master, the two full scans and JSON.stringify per language) now runs in a
worker thread, so "Alles vertalen" no longer blocks the main event loop.
Network/DB parts (official Habbo fetch, LibreTranslate, file writes) stay
on the main thread. The pure helpers were extracted to furni-data-i18n-core
(no server-only deps) so the worker can import them. Falls back to the same
helpers on the main thread when no worker is available or in tests.
The synchronous convertSwfToNitro / extractIconFromSwf calls (pure CPU,
no DB/network) now run in a small worker pool so the main Node event
loop stays free during imports — the dominant import-time CPU spike is
moved off the request thread. The pool degrades gracefully to a
synchronous fallback if workers can't be created, and is skipped in the
test environment.
- patchLocalizedFurniDataEntries now only touches the wanted classnames
instead of scanning the whole 100k file, reads the master once (was
twice), and uses the batched translator instead of one HTTP call per
text.
- Remove the now-unused serial translateWithLibre.
- buildLocalizedFurniDataFiles yields between the two section scans and
before stringify so a single language no longer blocks the event loop
for the whole pass at once.
Lower LibreTranslate concurrency to 2 and yield the event loop between
languages so the 13x deep-clone + stringify loop no longer pins a core
continuously. The build is also already opt-in (off by default on import).
The maintenance "Fix alles" now has a checkbox to include or skip the
duplicate catalog_pages merge, so admins can choose whether to run it.
removeDuplicates takes an includePages flag and fixEverything threads it
through to the action.
Furni imports can now choose whether to translate (per import) and which
languages to build, instead of always rebuilding all 13 FurnitureData_<lang>
files. The studio header also has a global switch that persists the
furnidata_translate_enabled setting, seeding the per-import default.
The maintenance "Fix alles" now collapses catalog_pages that share the same
caption_save + parent_id into a single survivor (moving catalog_items and
reparenting child pages before deleting duplicates), so the catalog tree
never renders double links. The duplicate-page count is also surfaced in the
health panel.
- Move /admin/import/* tools under /admin/studio/* and add studio nav, layout and tabs
- Add shared catalog maintenance panel plus a /admin/studio/maintenance tab
- Make furnidata reconciliation overwrite conflicting entries with the
DB-authoritative items_base classname/id (surfaced via fixedConflicts)
- Add furnidata_translate_enabled setting to skip the heavy localized
furnidata build during import (manual build-languages still forces it)
- Update staff smoke contract test for the studio hub
Translate deduplicated texts in batches of 48 with 4 parallel workers
instead of one request per string, serve cached hits locally, and
persist results so rebuilds never re-call the API for the same text.
Every write path now sets offer_id = catalog_items.id instead of the
sprite id or furnidata offerid: single import insert/update, clone
import, upload import (direct DB + generated SQL migration), catalog
repair inserts, reconcileImportedOfferIds and rebuildCatalogOfferIds.
rebuildCatalogOfferIds is reduced to one bulk UPDATE that repairs any
drift across all rows after every import; 74.838 legacy rows were
repaired on the live database with this change.
- Change skipFurniDataWrite from true to false in both import routes
- Previously: FurnitureData.json entries were deferred/skipped during import
- Now: Entries are written immediately after each furniture item import
- Result: FurnitureData.json is updated directly, translations work again, offer_id fix preserves correctly set values
- Change condition from to
- Prevents overwriting offer_id that was correctly set to sprite_id during furniture import
- Fixes catalog showing wrong furniture when offer_ids and sprite_ids don't match
- Switch default EMULATOR_MODE from 'rcon' to 'api'
- Remove RCON_HOST/PORT/timeout/maxRetries env vars
- Add EMULATOR_API_URL to .env and .env.example
- Update createEmulatorTransport() to use HTTP API only
- Remove rconHost/port from offline alert context
- All 827 tests pass, TypeScript compiles cleanly
- Add EMULATOR_MODE=api option with HTTP transport mirroring RCON payload
- buildLocalizedFurniDataFiles now falls back to LibreTranslate for customs (incl. pl/ru/ar/ja)
- Remove Organiseer import button from studio and Reorganize Catalog from import page
- Delete organize API route and PUT handler; remove organizeCatalogAll from catalog-repair.ts
- Increase imported-furni list perPage from 50 to 500
tryDownloadCandidates now fires every candidate URL concurrently and keeps
the first valid response, removing the old 15s×retry sequential waits on
slow/unreachable sources that made import speed uneven. Batch concurrency
raised 8 -> 12 to absorb the now-faster downloads.
AddSize32ToBundle downscales the largest existing sprite (usually size 64)
to generate a 32 visualization, so every bundle — including the
source-provided gamedata furniture — carries a 32 frame without needing the
original SWF. AddSize32AllNitros rewrites every existing .nitro across all
nitro directories (incl. /var/www/Gamedata/bundled/furniture) and is
idempotent (bundles that already have 32 are skipped). Expose it via a Studio
button and the background admin endpoint POST /api/admin/import/furni/add-size32.
The SWF-to-Nitro converter previously dropped the size 32 visualization
and any sprite image whose name contained '_32_', so bundles lacked the
small/catalog render size. Keep them so each furniture bundle renders
correctly at every scale the client requests.
- xml-processor: stop skipping visualization size === 32
- index.ts: stop excluding '_32_' sprite assets/images from the sheet
Run the independent icon and swf downloads in parallel via Promise.all
instead of sequentially. The nitro fallback still runs after the swf check
since it depends on the swf result.
Combined with batch concurrency and the single FurnitureData write, this
further cuts per-item import time (roughly halves the download phase).
- Run batch imports with bounded concurrency (IMPORT_BATCH_CONCURRENCY = 8)
instead of fully sequentially, parallelizing network downloads and DB writes.
- Collect FurnitureData.json entries and write the whole file once per batch
(appendFurniEntriesBulk) instead of rewriting it on every single item.
Catalog id allocation stays serialized and FurnitureData writes remain
lock-guarded, so concurrent imports are safe.
organizeCatalogAll no longer pulls every items_base row into the catalog.
It now touches only items whose catalog entry lives on a page under the
Imported Furniture parent (re-homing/re-pricing onto line/category
sub-pages). Items are never newly added. Button renamed to 'Organiseer
import'.
Line pages used caption_save 'imp_line_<slug>' which could exceed the
25-char column limit (e.g. imp_line_pixel_collectible = 26 chars),
causing the INSERT to fail. Cap the slug at 16 chars.
Add a fully automatic "Organize all" action in the Studio that organizes
the entire database into catalog pages:
- Create a dedicated sub-page per furniture line (e.g. weebz, habbox) from
FurnitureData.json, falling back to the auto-detected category.
- Add missing purchasable items to the catalog on their line/category page.
- Re-home and re-price existing catalog entries onto their correct page.
- Pages are created automatically when missing; no manual selection needed.
Includes the organizeCatalogAll service, a POST /api/admin/import/furni/organize
endpoint, and the Studio button.
- Extended fixDatabaseConsistencyAfterImport() to also set cost_credits = 3 where <= 0
- Integrated into both single and batch import routes
- API responses now include costCreditsFixed count
Ensures 100% of catalog_items have cost_credits > 0 after import
- Typecheck, lint, tests all pass
- Pushed to GitLab
- Added fixDatabaseConsistencyAfterImport() function in furni-import.ts
* Fills empty catalog_name from items_base.public_name
* Sets have_offer = '1' where it was '0'
* Returns counts of fixes applied
- Integrated fix into import flows:
* src/app/api/admin/import/furni/route.ts (single & batch POST)
* src/app/api/admin/import/furni/batch/route.ts (SSE batch POST)
- API responses now include:
* catalogNameFixed: number of catalog names filled
* haveOfferFixed: number of have_offer values fixed
- Ensures 100% catalog_name coverage and have_offer=1 after every import
- Typecheck, lint, and all 827 tests pass
Simo
openhands