EpicNext-Cms/docs/operations/backup-and-restore-drill.md

Backup and isolated restore drill

This opt-in operator tool creates one MariaDB logical dump and copies explicitly selected persistent files. It never runs during install, update or CI deployment. The only restore operation is a disposable drill: there is no production restore command, database target, destination directory or overwrite option.

Scope and prerequisites

Use the existing project Node toolchain and Docker CLI/Engine on a Linux host. Creation uses a short-lived mariadb:11.4.5 client on the Docker host network, so 127.0.0.1 means that host. Use a local Docker Engine/context with the same filesystem; remote Docker daemons and Docker Desktop are not supported for creation. The image must already be available or downloadable through the operator's normal image policy. No packages are installed by this tool.

Choose the single application database explicitly. Its tables must use InnoDB; empty databases, system schemas and unsupported engines are rejected. The backup account needs access to every application table, view, trigger, routine and event being exported. Account/grant provisioning belongs to the operator; the tool does not change privileges. Restore compatibility is checked with the pinned MariaDB image, not guaranteed across arbitrary server versions, plugins, collations or external schema dependencies.

Before starting, pause every database/file writer and schema changer for the whole creation command: CMS requests that write, workers, schedulers, emulator processes, MariaDB events, import jobs and other tools using these resources. Keep them paused until the command exits. --writers-quiesced records your acknowledgement; it does not stop services or prove that they are stopped. No DDL may run while dumping. InnoDB's transaction snapshot alone cannot make independently copied files consistent with rows or coordinate other services. The before/after table inventory and second source-file hash pass detect many concurrent changes, but cannot replace quiescing.

The dump explicitly uses --single-transaction --quick --skip-lock-tables --routines --events --triggers --hex-blob --tz-utc. It retains schema/data and named SQL objects while streaming rows. See MariaDB dump snapshot and object options. This is a logical application backup, not point-in-time recovery: binlogs, server accounts/grants, server configuration, Redis state and unrelated databases are outside its scope.

Private configuration

Create a private JSON file outside the clone and every selected source directory, for example /etc/epicnext/backup.json. Use a directory accessible only to the operator and file mode 0600. Edit it with the host's normal private configuration workflow; do not put a password in a shell command or commit the file.

{
  "database": {
    "host": "127.0.0.1",
    "port": 3306,
    "user": "REPLACE_WITH_BACKUP_ACCOUNT",
    "password": "REPLACE_PRIVATELY",
    "database": "REPLACE_WITH_APPLICATION_DATABASE"
  },
  "roots": {
    "storage": "/srv/epicnext/storage",
    "nitro": "/srv/epicnext/public/nitro-assets",
    "swf": "/srv/epicnext/public/swf",
    "gamedata": "/var/www/Gamedata"
  }
}

Replace the example clone path and database settings. storage, nitro and swf are required existing directories, including when empty. gamedata is optional; omit its key only when those files are independently backed up or not used. All paths must be absolute, distinct, non-overlapping and free of .. and symlink/junction components. Links, hardlinked files, special files and secret configuration filenames such as .env, .docker-install and persistent.path inside a selected root cause rejection. The configuration itself may not be inside any source root. Keep writers and directory ownership controlled for the duration; this is not a filesystem snapshot resistant to hostile concurrent renames.

The tool reads only this explicit JSON. It does not source .env, inspect a running application's environment or inherit its secrets into Docker. The MariaDB password is written to a random private temporary directory/file (0700/0600) and read through a read-only container mount with --defaults-file as the first client option. It is absent from process arguments and tool logs; source configuration and absolute source paths are absent from the manifest. Temporary credentials are removed on ordinary success/failure. See MariaDB option-file handling.

Only Docker connection/runtime environment variables are passed to child processes. Do not point DOCKER_HOST/DOCKER_CONTEXT at another host or enable shell tracing around private configuration work.

Create and verify

Provision a private backup parent directory, with adequate free space, outside all source roots. Choose a new absolute artifact directory for each run. After pausing the writers described above, run from the clone root:

node scripts/backup/cli.mjs create \
  --config /etc/epicnext/backup.json \
  --output /srv/epicnext-backups/2026-09-13T200000Z \
  --writers-quiesced

The date is an example; use a new name for the actual run. Existing directories are refused, including earlier incomplete attempts. A successful artifact contains:

manifest.json
 database.sql
 files/storage/...
 files/nitro/...
 files/swf/...
 files/gamedata/...   (only when selected)

manifest.json is written last and marks the format complete. It records each relative file path, byte count and SHA-256, empty directories, selected logical roots, creation time and database inventory. The inventory includes table row counts and MariaDB extended table checksums plus names/types of views, triggers, routines and events. These checksums validate restored table contents; they are not cryptographic signatures or a substitute for application-level checks. See MariaDB CHECKSUM TABLE semantics and version limits.

On a caught failure the newly created artifact is removed; an interrupted process can leave an incomplete directory, which verification refuses. Source roots and existing backup directories are never overwritten. Files are copied and hashed as streams, then source hashes are checked again across the dump interval. This performs multiple full reads of the assets and table data; allow sufficient time and disk capacity during the maintenance window.

After creation you may resume writers. Copy the completed artifact to the designated protected backup location according to the operator's retention/encryption policy. SQL and uploaded files contain application data and may themselves contain sensitive values. Hashes detect corruption against the manifest, not an attacker who can replace both. Keep the manifest and artifact under trusted access control. Secrets, TLS material and deployment configuration excluded from this artifact need their separate recovery procedure.

Isolated restore drill

Run the drill against a trusted completed artifact:

node scripts/backup/cli.mjs drill \
  --artifact /srv/epicnext-backups/2026-09-13T200000Z

The drill first rejects missing, extra, changed or unsafe paths/files. It copies the persistent files into a new private temporary directory and verifies their contents. It creates a randomly named MariaDB container with no network and no published ports, a fresh password supplied by file, and a disposable database volume. SQL is imported through the container's standard input; there is no connection to the source database. The event scheduler stays off. The resulting table counts/checksums and object inventory must match the manifest. This proves dump importability and the recorded contents, not a full CMS/emulator startup or external-service recovery.

On ordinary completion/failure it removes the container, its anonymous volume and temporary files. Cleanup failure makes the drill fail. A host crash or forced process termination can interrupt cleanup; inspect only resources labeled cms.backup-drill=true and private cms-backup-private-* temporary directories from that run, and review their ownership before manual removal. Never substitute an existing database/container into this procedure.

A real production recovery remains a separate, reviewed procedure with its own deployment configuration, credentials, downtime and application checks. This tool deliberately cannot perform it.

Repository verification

pnpm exec vitest run --coverage.enabled=false scripts/backup
pnpm exec vitest run --config vitest.integration.config.ts integration/backup.test.ts

The local suite exercises real file copies, empty directories, SQL/file tampering, manifest traversal, links, secret-file rejection, overwrites, cleanup and changes during backup. The integration suite requires Docker: failure to start MariaDB fails the suite. It uses a real database with Unicode text, large unsigned identifiers, a foreign key, view, trigger, procedure and event; it creates an artifact, restores it to a second disposable server and checks both byte corruption and a SQL content change with a recomputed file hash. A passing local file suite alone is not evidence that the MariaDB drill ran.