feat: export Catalog Studio assets and SQL to catalog repository
This commit is contained in:
1 parent
bfbf244141
commit
9ec9ab31ad
21 files changed
+1811
-514
No files matched your search
@@ -0,0 +1,73 @@
|
||||
# Catalog Studio repository export
|
||||
|
||||
Studio mutations automatically queue an export to
|
||||
`https://gitlab.epicnabbo.nl/remco/Epicnabbo-Catalogus-Updated-Daily.git`, branch
|
||||
`Beta-3`. The jobs worker processes pending exports every minute. Streaming
|
||||
imports stay active until their stream completes. Server actions for catalog
|
||||
pages, offers, deletion and maintenance are covered as well.
|
||||
|
||||
## Server setup
|
||||
|
||||
1. Create a **dedicated clean clone** of the repository on `Beta-3`, outside the
|
||||
CMS directory. Configure non-interactive Git push authentication for the
|
||||
worker OS account using its credential helper. Do not put tokens in URLs.
|
||||
2. Set `CATALOG_GIT_CHECKOUT` to that absolute clone directory in the CMS and
|
||||
worker environments. Set `CATALOG_GIT_STATE_DIR` to a persistent, writable
|
||||
directory outside the clone, shared by both processes on the same host.
|
||||
3. Run `pnpm jobs:worker` alongside the CMS under your process supervisor.
|
||||
Both processes must have access to the configured asset directories and DB.
|
||||
The worker command enables the React server condition for server-only modules.
|
||||
4. Restart the CMS after setting the environment. In **Studio → Sync**, use
|
||||
**Export now / retry** for the initial export. Subsequent mutations queue
|
||||
automatically. Status shows pending/active operations and the last commit.
|
||||
|
||||
Leaving `CATALOG_GIT_CHECKOUT` empty disables export. No credentials are shipped.
|
||||
This source change alone does not configure or deploy the production service.
|
||||
|
||||
## Exported content
|
||||
|
||||
| Source | Repository destination |
|
||||
| --- | --- |
|
||||
| Configured Nitro bundles, including furniture, figures, effects and pets | `Gamedata/bundled` |
|
||||
| Configured furni icons | `Gamedata/icons` |
|
||||
| Existing badge/catalog images | `Gamedata/c_images` |
|
||||
| FurnitureData and supported public game-data JSON files | `Gamedata/config` |
|
||||
| Existing localized FurnitureData files | `catalogue version 2 ( Final (Dev)/langs furnidata` |
|
||||
| `items_base`, `catalog_pages`, `catalog_items` | `catalogue version 2 ( Final (Dev)/sqls` |
|
||||
| `catalog_pages_bc`, `catalog_items_bc`, when present | Same SQL directory |
|
||||
|
||||
SQL is read in one consistent, read-only InnoDB transaction. Dumps contain table
|
||||
definitions and deterministic upserts with hexadecimal UTF-8 string literals.
|
||||
Import `items_base.sql`, then `catalog_pages.sql`, then `catalog_items.sql`.
|
||||
Existing schemas are not migrated by these dumps. Rows absent from the source
|
||||
are omitted; importing an upsert dump into another existing database does not
|
||||
delete that database's extra rows. No user, session or credential tables are exported.
|
||||
|
||||
Only existing local assets are exported. Translation generation follows Studio's
|
||||
existing setting; export does not generate missing languages or download assets.
|
||||
Public JSON is explicitly allowlisted so translation caches and private runtime
|
||||
files cannot enter the repository. Invalid JSON or concurrent Studio changes
|
||||
prevent publication of that snapshot.
|
||||
|
||||
## Failure and concurrency behavior
|
||||
|
||||
- Pending events survive process restarts; events added during publication remain
|
||||
pending for the next run. Partial imports are exported as their settled local
|
||||
state, including successful items from a batch containing failures.
|
||||
- A single filesystem lock serializes the worker. Dead local process markers are
|
||||
recovered on the next run. For an unreadable marker or a marker from another
|
||||
host, stop the CMS and worker before repairing the queue directory.
|
||||
- A failed push retains the local commit and pending events for retry. Concurrent
|
||||
upstream commits are rebased; a conflict aborts the rebase and leaves the event
|
||||
pending. Resolve conflicts in the dedicated clone, then retry.
|
||||
- The checkout must be clean before each run. Unrelated files are preserved and
|
||||
no force push is used. Missing local files do not cause remote deletions.
|
||||
- Status errors omit raw Git output to avoid exposing authentication material.
|
||||
|
||||
## Verification
|
||||
|
||||
Run the `catalog-git-*` service tests and `src/lib/catalog-export-api.test.ts` with
|
||||
Vitest. The integration test creates a temporary bare remote and verifies push,
|
||||
failed-push recovery, no-op export and preservation of unrelated files. SQL
|
||||
snapshot tests mock the database; production DB import and production push need
|
||||
verification in the deployed environment.
|
||||
Reference in new issue
Block a user