# DEPLOY - q4.suby.pl

Quake 4 browser port based on OpenQ4. The public service contains GPL engine code,
SDK-licensed gameplay modules, deterministic self-authored CC0 minimal `pak0`/`pak1`
packs, and an approved CC0 q4web arena addon. For this deployment, the operator has
confirmed authorization to deliver the required retail data. The config-gated gateway
serves a versioned common/per-map PK4 graph for covered multiplayer maps and retains the
exact `pak001.pk4` through `pak022.pk4` plus `zpak_english.pk4` catalog as a full
fallback. Both use SHA-addressed URLs; the bytes remain outside Git, the public tree,
generated content, and the source offer.

| Item | Value |
|---|---|
| Public URL | `https://q4.suby.pl` |
| Direct URL | `https://q4.z6.mm.mk:11443` |
| Gateway | `suby:3280`, `q4.service` |
| Authoritative game | native OpenQ4 UDP `127.0.0.1:28004` |
| Working directory | `/srv/q4` |
| Health | `/healthz` |
| Status | `/api/status` |
| Full retail fallback | `/retail/manifest.json`, exact 23-file allowlist |
| Retail map bundles | `/retail/bundles/manifest.json`, `/retail/bundles/<sha256>-<name>.pk4` |
| Replays | `/api/replays`, newest 300 finalized `.q4sd` files |
| Replay UDP pool | `127.0.0.1:28040-28047`, one isolated process per active replay |

## Normal operation

```bash
curl -s http://127.0.0.1:3280/healthz | python3 -m json.tool
curl -s http://127.0.0.1:3280/api/status | python3 -m json.tool
systemctl status q4
journalctl -u q4 -f

# The only permitted restart path. Exit 2 means it deliberately skipped.
cd /srv/q4 && tools/safe_restart.sh
```

The restart drains clients and retail transfers for up to ten minutes by default, then
waits up to 120 seconds for `"status":"ok"`, `"dedicated":true`, `"retail":true`, and
`"retailBundles":true`. The dedicated flag means OpenQ4 has loaded the authoritative
map, not merely spawned; the two retail flags mean the exact full catalog and its
source-bound bundle graph were validated and enabled. Override the post-restart window
with `Q4_RESTART_READY_TIMEOUT_SECONDS`; drain and polling controls are
`Q4_RESTART_DRAIN_POLLS`, `Q4_RESTART_DRAIN_INTERVAL_SECONDS`, and
`Q4_RESTART_READY_INTERVAL_SECONDS`.

Files under `public/` are served live. Changes to Node code, engine artifacts, native
binaries, maps used by the dedicated server, or the systemd environment require a
drained safe restart. Never run `systemctl restart q4` directly while the service is
already live.

Retail delivery defaults to disabled. Production enables it explicitly with
`Q4_RETAIL_DELIVERY=operator-authorized`, points `Q4_RETAIL_DIR` at the ignored
operator data and `Q4_RETAIL_MANIFEST` at the ignored staged manifest. It also points
`Q4_RETAIL_BUNDLE_DIR` at `runtime/retail-bundles` and
`Q4_RETAIL_BUNDLE_MANIFEST` at `runtime/retail-bundle-manifest.json`. Any other
non-disabled delivery mode is rejected. At gateway startup the bundle manifest must
match the active full retail revision; each advertised file is size/SHA-256 verified
and pinned before its hash route is enabled. Bundle payload routes support bounded
HTTP Range requests and use the same `private, no-store` response policy and download
limits as the full catalog. `Q4_RETAIL_MAX_CONCURRENT` and `Q4_RETAIL_MAX_PER_IP`
bound both kinds of transfer; neither setting widens an allowlist.

The browser stores common and per-map packs in the versioned OPFS revision as a
persistent cache: reopening a map reuses its verified payload, while changing maps
downloads only packs not already present and retains the earlier packs. A missing
bundle manifest (HTTP 404/410) or map not covered by its graph falls back to the exact
23-file catalog. A failed bundle download or hash mismatch is an error, not a fallback
to partially trusted content.

The active complete graph covers all 23 multiplayer maps and all 62 supported
map/mode scenarios. The common pack is 351,993,687 bytes, per-map packs range from
487,469 to 17,636,172 bytes, and retaining every map takes 537,679,229 bytes. A first
map therefore downloads 352,481,156-369,629,859 bytes instead of the
2,779,708,326-byte full retail catalog; each later map downloads only its missing map
pack. `retailBundles:true` reports a valid active graph, while current all-scenario
evidence proves complete coverage.

Every `deploy/install.sh` mode runs `tools/stage-retail-bundles.py --verify` before
installing or publishing and repeats it as the locked `q4` account. Publication,
`tools/safe_restart.sh`, and the Caddy watchdog all require
`"retailBundles":true`; they must fail closed instead of exposing a service with only
an unverified or stale bundle graph.

`POST /api/replays/<id>/watch` allocates or reuses a replay process and returns its
`port` and same-origin `connectUrl`. The HTML client creates a one-use 64-hex capability,
sends it as `X-Q4-Replay-Claim`, and spends it on the same-origin WebSocket as the
`replayClaim` query parameter. Escape before the socket connects sends an idempotent
same-origin `DELETE` to the watch URL with the same header; cancelling one viewer never
removes another viewer's pending claim. A ready process which nobody claims is stopped
after 60 seconds as a fallback. `Q4_REPLAY_PORTS` is a comma-separated pool that must
not overlap `Q4_GAME_PORTS`; production uses `28040-28047`.
`Q4_REPLAY_PLAYBACK=0` disables allocation without hiding downloads. Optional lifecycle
knobs are `Q4_REPLAY_MAX_VIEWERS`, `Q4_REPLAY_START_TIMEOUT_MS`,
`Q4_REPLAY_END_GRACE_MS`, `Q4_REPLAY_IDLE_MS`, and `Q4_REPLAY_HARD_CAP_MS`.
Playback is frozen after process readiness until the first viewer reaches the in-game
state, so browser/WASM startup cannot consume a short replay before connection.

## Reproduce artifacts

Prerequisites include Node 22 or newer, GCC/G++ 13, Meson, Ninja, Emscripten as described by
`port/wasm/emscripten.ini`, SDL/OpenAL development headers, `innoextract`, `zip`, `unzip`,
`Xvfb`, `rsync`, and Google Chrome at `/usr/bin/google-chrome`.

```bash
cd /srv/q4
npm ci
tools/sync-upstream.sh

# Private operator data; output remains ignored and outside the public tree.
tools/import-retail.sh \
  '/mnt/smbgames/games/GOG/setup_quake_iv_1.4.3_lang_update_(63131).exe'

# Hash and validate the exact pak001-pak022 + zpak_english deployment catalog.
# The ignored manifest contains only metadata and SHA-addressed route paths.
node tools/stage-retail-manifest.mjs

# Build deterministic, disjoint common/per-map packs from the source-bound native
# I/O profiles, then independently verify every output PK4 and manifest edge.
python3 tools/stage-retail-bundles.py
python3 tools/stage-retail-bundles.py --verify

tools/build-native.sh --clean --jobs 4
# Generate all 22 retail AAS entries locally. The playerclip-aware compiler is primary;
# exact CRC/SHA-bound Q4Max or SABot bytes are fail-closed fallbacks only.
node tools/stage-retail-botnav.mjs
tools/build-wasm.sh --clean --jobs 4

# Build and ingest the tested CC0 arena without implicit staging or restart, then
# regenerate the audited CC0 runtime exactly once.
python3 tools/build-map-source-pk4.py \
  --source mods/maps/q4web_arena --output /tmp/q4web_arena-source.pk4
node tools/ingest.mjs /tmp/q4web_arena-source.pk4 \
  --license CC0-1.0 --source self-authored --author Oivo \
  --force --no-stage --no-restart
node tools/stage-web-content.mjs
```

`engine/` is a tracked frozen deployment artifact. `runtime/` and `assets-src/` are
private operator state. The full retail files and generated bundle packs are read from
there by exact allowlist routes, but are never copied under `public/`, `content/`,
`engine/`, or the source archive. Bundle inputs default to
`runtime/retail-bundle-profiles/manifest.json`; outputs default to
`runtime/retail-bundles/` and `runtime/retail-bundle-manifest.json`.
Approved Q4Max/SABot input archives remain under `build/native-aas/`; the generated
bot-navigation pack, manifest, and logs remain under `runtime/`. None of these source
or output files is exposed by the gateway or included in the public source offer.
`content/assets/` is rebuilt only from audited `content-src/` inputs and approved
addons; `mods/dist/` and the generated public source archive are generated and never
committed.

## Pre-deploy verification

```bash
npm test
npm run test:zip-cache
python3 tools/audit-public-content.py \
  --public-dir content/assets/baseoq4 \
  --retail-dir assets-src/gog/q4base \
  --allowlist content-src/retail-duplicate-allowlist.json
node vendor/openq4/tools/tests/webgl2_shader_smoke.mjs
python3 tools/test-wasm-network.py
port/native/smoke-dedicated.sh
port/native/smoke-bot-minplayers-reset.sh
port/native/smoke-capacity.sh
port/native/smoke-server-demo-playback.sh
port/native/smoke-server-demo-join.sh
node tools/stage-retail-botnav.mjs --verify
node tools/smoke-retail-botnav.mjs
python3 tools/stage-retail-bundles.py --verify

# Recreate the public source offer from the current checkout, then prove that
# its extracted and patched trees match this release's clean source verification.
source_verify="$(tools/build-wasm.sh --clean --verify-sources-only)"
printf '%s\n' "${source_verify}"
native_source_verify="$(python3 tools/native-source-identity.py --root .)"
printf '%s\n' "${native_source_verify}"
openq4_diff="$(sed -n 's/^openq4_patched_diff_sha256=//p' <<<"${source_verify}")"
gamelibs_diff="$(sed -n 's/^gamelibs_patched_diff_sha256=//p' <<<"${source_verify}")"
patch_manifest="$(sed -n 's/^patch_manifest_sha256=//p' <<<"${source_verify}")"
runtime_source="$(sed -n 's/^free_runtime_source_sha256=//p' <<<"${source_verify}")"
native_patch_manifest="$(sed -n 's/^native_patch_manifest_sha256=//p' <<<"${native_source_verify}")"
native_openq4_diff="$(sed -n 's/^native_openq4_patched_diff_sha256=//p' <<<"${native_source_verify}")"
native_gamelibs_diff="$(sed -n 's/^native_gamelibs_patched_diff_sha256=//p' <<<"${native_source_verify}")"
[[ "${openq4_diff}" =~ ^[0-9a-f]{64}$ \
  && "${gamelibs_diff}" =~ ^[0-9a-f]{64}$ \
  && "${patch_manifest}" =~ ^[0-9a-f]{64}$ \
  && "${runtime_source}" =~ ^[0-9a-f]{64}$ \
  && "${native_patch_manifest}" =~ ^[0-9a-f]{64}$ \
  && "${native_openq4_diff}" =~ ^[0-9a-f]{64}$ \
  && "${native_gamelibs_diff}" =~ ^[0-9a-f]{64}$ ]]
tools/stage-source-offer.sh
Q4_EXPECT_OPENQ4_DIFF_SHA256="${openq4_diff}" \
Q4_EXPECT_GAMELIBS_DIFF_SHA256="${gamelibs_diff}" \
Q4_EXPECT_PATCH_MANIFEST_SHA256="${patch_manifest}" \
Q4_EXPECT_RUNTIME_SOURCE_SHA256="${runtime_source}" \
Q4_EXPECT_NATIVE_PATCH_MANIFEST_SHA256="${native_patch_manifest}" \
Q4_EXPECT_NATIVE_OPENQ4_DIFF_SHA256="${native_openq4_diff}" \
Q4_EXPECT_NATIVE_GAMELIBS_DIFF_SHA256="${native_gamelibs_diff}" \
  tools/test-source-offer.sh

# Start the release locally with publication disabled. This removes an old
# watchdog cron and every dedicated q4.suby.pl route before systemd starts q4.
sudo deploy/install.sh --start-local
for attempt in $(seq 1 60); do
  health=$(curl -fsS http://127.0.0.1:3280/healthz 2>/dev/null || true)
  grep -q '"status":"ok"' <<<"$health" \
    && grep -q '"dedicated":true' <<<"$health" \
    && grep -q '"retail":true' <<<"$health" \
    && grep -q '"retailBundles":true' <<<"$health" \
    && break
  sleep 2
done
health=$(curl -fsS http://127.0.0.1:3280/healthz)
printf '%s\n' "$health" >&2
grep -q '"status":"ok"' <<<"$health"
grep -q '"dedicated":true' <<<"$health"
grep -q '"retail":true' <<<"$health"
grep -q '"retailBundles":true' <<<"$health"

Q4_E2E_URL=http://127.0.0.1:3280 npm run test:e2e:shell
Q4_E2E_URL=http://127.0.0.1:3280 npm run test:e2e:bundle-cache
Q4_E2E_URL=http://127.0.0.1:3280 npm run test:e2e:bundle-maps

# The replay phase requires a finalized API replay lasting at least 90 seconds.
# Set Q4_E2E_REPLAY_ID to the id of a known qualifying replay when automatic
# selection is not appropriate.
(
  full_e2e_profile="$(mktemp -d /tmp/q4-full-e2e.XXXXXX)" || exit 1
  trap 'rm -rf -- "${full_e2e_profile}"' EXIT
  Q4_E2E_PROFILE="${full_e2e_profile}" \
    Q4_E2E_URL=http://127.0.0.1:3280 \
    npm run test:e2e:full
)

# Prove that all three independent evidence files are bound to the exact release,
# bundle-manifest bytes, cache/importer code, and stored OPFS files that are about
# to be published.
node tools/verify-e2e-evidence.mjs
node tools/verify-retail-bundle-cache-evidence.mjs
npm run verify:e2e:bundle-maps

# This is the only publication step. It repeats all three evidence checks, rechecks
# local readiness, conditionally restores the route, and then enables its cron.
sudo deploy/install.sh --publish
```

The release gate additionally requires the full Playwright test to fetch and verify
the enabled server retail catalog into a clean browser OPFS profile before checking a
rendered campaign map, two simultaneous WebAssembly MP clients, bot presence, voice
peer connection, a replay viewer followed by a second late-joining viewer, and
desktop/mobile canvas pixel checks. Its replay phase requires a finalized
`openq4-server-demo` whose duration is at least 90 seconds. `Q4_E2E_REPLAY_ID` may pin
the run to a specific replay, but that replay must satisfy the same completeness and
duration requirements. Save evidence in `.selfcheck/`. Never run `--publish` until
that command has completed successfully.

The all-map bundle E2E is the exhaustive map-closure gate. It fails unless
`runtime/retail-bundle-manifest.json` covers every map exposed by the multiplayer
menu. In a single fresh persistent Chromium profile it starts all 62 map/mode
scenarios through the HTML launcher. Only the first mode of each map may perform the
still-missing bounded Range downloads; every later mode of that map must fetch zero
PK4 payload. One absolute deadline covers each complete scenario. The gate verifies
the accumulated OPFS state, the dedicated server's exact current map/mode, the first
in-game snapshot, a persistent accumulator free of fatal, `InvalidGUI`, and missing
GUI/model/image/cube/animation/sound/material/shader diagnostics, and a non-black
central canvas. It returns to the first scenario with zero PK4 payload, then uses a
separate bounded hashing deadline to enumerate the actual OPFS `q4base` directory and
hash every entry. Extra, missing, or non-file entries fail verification. The gate
writes `.selfcheck/retail-bundle-maps-e2e.json`. Rerun it after any map profile,
bundle graph, launcher, importer, gateway, engine, content, or native release change.

For a focused failure diagnosis, set a comma-separated exact map list, for example:

```bash
Q4_BUNDLE_MAPS_E2E_MAPS=mp/q4xdm14,mp/q4xdm15 npm run test:e2e:bundle-maps
```

This always uses a fresh Chromium/OPFS profile and runs every supported mode of the
selected maps. It writes JSON and screenshots only below `.selfcheck/diagnostics/`.
Its diagnostic schema is deliberately rejected by the publication verifier and never
replaces `.selfcheck/retail-bundle-maps-e2e.json`; publication still requires a fresh
complete 62-scenario run.

The separate bundle-cache E2E starts with a fresh persistent Chromium profile. It
installs `mp/q4web_arena`, reloads it, switches to `mp/q4dm3`, and returns. It must
prove common+map downloads through ordered Range requests on the first map, zero PK4
payload on reload, only the new map pack on switch, retained earlier packs, and no
`/retail/q4base/` fallback. It writes
`.selfcheck/retail-bundle-cache-e2e.json`. Run it after the local gateway reports
`retailBundles:true`. Its schema-3 evidence may be retained only while the complete
`runtime/retail-bundle-manifest.json` bytes, size, revision, and source revision are
identical, the importer/app/gateway/server code identity is unchanged, and the tested
OPFS files still have the recorded SHA-256 values. Any bundle graph or cache-path code
change requires rerunning this E2E. It never substitutes for schema-4 full-E2E
or exhaustive all-map evidence; publication requires all three files.

`--publish` first validates the source archive checksum and its embedded
`SOURCE_MANIFEST.json` against the current frozen `engine/engine-manifest.json`,
`engine/build-info.txt`, browser and native source identities, the native runtime
build-info, and every artifact covered by `engine/SHA256SUMS`. It also hashes every
native patch member named by the source manifest, including patch 0028, before it runs
`tools/verify-e2e-evidence.mjs`,
`tools/verify-retail-bundle-cache-evidence.mjs`, and
`tools/verify-retail-bundle-maps-evidence.mjs`, still before its first health, Caddy,
or cron action. The full evidence verifier requires schema 4, a passed
run which verified the deployed release, and campaign, multiplayer, replay, and mobile
evidence whose individual checks carry the exact current engine/content/retail release
identity. Required phases may be retained across separate E2E invocations when the
artifact identity is unchanged; the most recent run does not need to request all four
phases at once.

The isolated native capacity smoke holds eight bots for 30 seconds, connects two named
native clients, then requires the same eight-bot floor for another 30 seconds. Its four
delimited `serverInfo` snapshots and process logs remain in
`runtime/native-capacity-smoke/`.

Retail bot navigation is generated locally from the operator data and the pinned native
OpenQ4 compiler. Its content-hashed PK4 stays under `runtime/native/baseoq4`; neither the
PK4 nor its manifest is public browser content. The gateway enables bots per map only when
`runtime/retail-botnav-manifest.json` matches the active retail revision and the staged PK4
contains exactly the declared AAS entries with valid ZIP metadata, CRC, SHA-256 and graph
structure, generated by the currently staged native client and multiplayer GameLib. The
schema-3 manifest covers exactly all 22 retail multiplayer maps.

Every map first uses the compiler-only `retail-playerclip-aas32-v1` profile. Patch
`0032` adds the opt-in `playerClipSolid` compiler setting, and the temporary profile
sets it to `1`, so the player-sized AAS treats `CONTENTS_PLAYERCLIP` as solid instead
of compiling paths through invisible player barriers. The same temporary save path
contains a compiler-only `info_player_team` seed used to make team-spawn-only maps
discoverable by `runAAS`; the `000_*_openq4.def` seed and
`001_*_playerclip_*_openq4.def` override load before stock declarations under
OpenQ4's first-declaration-wins ordering. Neither temporary declaration is included
in the navigation PK4.

Source fallback is deliberately fail-closed. It is eligible only when the primary
attempt matches one of the exact approved signatures: a clean, output-free AAS leak,
or the known post-write SIGABRT after a complete parsed AAS was written. The selected
fallback must then be the map-specific Q4Max `q4max-aas-01.zip` or SABot `sabota11.zip`
profile whose outer archive, inner PK4, entry bytes, map CRC, size, and SHA-256 all
match the compiled-in identities. Staging copies those verified AAS bytes under the
`.aas32` name without graph mutation. An unknown compiler failure, missing map
profile, changed source byte, CRC mismatch, or invalid AAS aborts the entire stage.
For `mp/q4xdm13`, staging uses the pinned SABot entry directly because the local
compiler fails nondeterministically both before and after a complete write. This keeps
release bytes independent of allocator corruption while retaining exact archive,
container, entry, CRC, size, and SHA-256 checks.
The source archives stay under `build/native-aas/`; they, the compiler logs, generated
PK4, and manifest are never public routes or browser/source-offer content.

Navigation availability and bot AI capability are separate contracts: the API exposes
`botNavigationReady` plus `botModes`, and currently enables bots only for `DM` on the
17 retail maps which support that mode. Team DM, CTF, Arena CTF, and Tourney remain
human-only even when their map has verified AAS. The install/publish path repeats
manifest validation as the locked `q4` service account. A clean native build removes
the PK4, so staging it again before a restart is mandatory.

The bot-navigation smoke changes through all 22 retail maps and verifies the exact
`Map:` signal plus `DewmAAS 1.08`. On each of the 17 DM maps, patch `0033` extends the
unchanged `botStatus slots=... entities=... active=...` summary with bounded per-bot
navigation telemetry. The smoke requires a healthy, non-spectating,
non-respawning/non-teleporting bot in a valid AAS area, real movement input, and at
least 64 units of XY displacement while `spawnAt` and `minRespawnTime` remain
unchanged. Movement input means `|forward| + |right| >= 16`. Respawn, teleport, or a
discontinuity above 512 units resets the baseline so a spawn/teleport jump cannot
satisfy the gate. Patch `0034` makes real bot clients request normal server respawn
rather than spectator mode, while preserving fake-client replay behavior, so
PM_SPECTATOR camera movement remains a hard failure. The
remaining five retail maps load and validate AAS without adding unsupported bots.

Do not satisfy the source-offer gate with an archive left by an earlier build. The
pre-deploy sequence above regenerates it only after obtaining the current clean
source identity, then requires the extracted archive to reproduce the browser and
native patched-tree hashes plus both patch manifests and the free-runtime hash.
Publication independently rejects an archive whose embedded identity does not match
the frozen browser engine and native runtime source chain.

`tools/stage-web-content.mjs` runs the same audit before publishing. It compares every
public PK4 entry with the private retail corpus by normalized path and content SHA-256;
exact byte duplicates fail whether they keep the same path or use a new name. The committed allowlist is
intentionally empty, and any future exception requires explicit license evidence.

## First installation

Stage and verify artifacts first. The installer creates the locked system identity
`q4`, transfers ownership only for the three runtime write locations, installs the
unit, and reloads systemd. Its default mode intentionally changes neither service nor
publication state:

```bash
cd /srv/q4
sudo deploy/install.sh
systemd-analyze verify /etc/systemd/system/q4.service

# Start a local-only release target before browser E2E. This removes an existing
# /etc/cron.d/q4-caddy-watchdog before enabling/starting q4.service.
sudo deploy/install.sh --start-local
for attempt in $(seq 1 60); do
  health=$(curl -fsS http://127.0.0.1:3280/healthz 2>/dev/null || true)
  grep -q '"status":"ok"' <<<"$health" \
    && grep -q '"dedicated":true' <<<"$health" \
    && grep -q '"retail":true' <<<"$health" \
    && grep -q '"retailBundles":true' <<<"$health" \
    && break
  sleep 2
done
health=$(curl -fsS http://127.0.0.1:3280/healthz)
printf '%s\n' "$health" >&2
grep -q '"status":"ok"' <<<"$health"
grep -q '"dedicated":true' <<<"$health"
grep -q '"retail":true' <<<"$health"
grep -q '"retailBundles":true' <<<"$health"

# Run the complete pre-deploy suite above, including full Playwright E2E. Publish
# only after it passes; the installer rechecks local readiness and delegates route
# creation to the idempotent watchdog before enabling the two-minute cron.
sudo deploy/install.sh --publish
```

`--start-local` and `--publish` are idempotent. The former always removes the cron
and every dedicated `q4.suby.pl` route before `systemctl enable --now q4`; the latter refuses to install it unless the
release-bound E2E evidence passes and local health reports `status=ok`,
`dedicated=true`, `retail=true`, and `retailBundles=true`. Publication never performs a
raw unconditional Caddy `POST`: `tools/caddy_route_watchdog.sh` first exits when the
host route already exists. Re-running `--publish` therefore checks the existing route
and refreshes the same cron file without duplicating it.

Route removal uses Caddy's `ETag`/`If-Match` concurrency contract and fails closed if
the admin API is unavailable, the host is part of a shared route, or concurrent
changes do not settle. The remover and watchdog share a lock; a stale cron process
cannot restore the route after its cron file has been removed.

The service runs as the non-login system user/group `q4` with an empty capability
set, `NoNewPrivileges`, private devices/tmp, and `ProtectSystem=strict`. Only
`replays/`, `runtime/native/home/`, and `runtime/replay-instances/` are writable in
its mount namespace. The full retail catalog and bundle graph are read-only and
reachable only through their manifests and SHA-addressed allowlist routes; retail PK4s, native
binaries/GameLibs, public files, content, and engine artifacts remain root-owned and
read-only. After restaging native or retail files, verify they are
world-readable/traversable for `q4`; do not recursively chown the checkout.

The `q4.suby.pl` Caddy route proxies HTTPS/WSS to `10.0.0.150:3280`. The direct
`q4.z6.mm.mk:11443` route is generated automatically from `/srv/gry/server.js` after
the portal registry contains `{ id: 'q4', port: 3280 }`. Client URLs and sockets are
same-origin, so no special direct-domain engine build is needed.

After the gateway is publicly healthy, add the q4 entry and tile to `/srv/gry`, then
restart the portal only through `/srv/gry/tools/safe_restart.sh`.

## Recovery

1. Restore the tracked `/srv/q4` tree and run `npm ci`.
2. Recreate pinned vendor trees with `tools/sync-upstream.sh`.
3. Re-extract the operator data with `tools/import-retail.sh`, then regenerate the
   exact deployment catalog with `node tools/stage-retail-manifest.mjs`. Restore or
   recapture source-bound bundle profiles, run `python3 tools/stage-retail-bundles.py`,
   and confirm `python3 tools/stage-retail-bundles.py --verify` passes.
4. Restore both pinned navigation source archives as regular files under
   `build/native-aas/` before staging retail bot navigation:
   - `q4max-aas-01.zip`: 25,635,781 bytes,
     SHA-256 `4d1232e6e466d7074538cbfb76bc94e616e2e9fc34a114ceac4080016a6de86f`.
   - `sabota11.zip`: 21,341,561 bytes,
     SHA-256 `624618da8af8be35e9b9e9ff08991dc89a5ed468a94b7492e65045e41a12110f`.
   Do not repack or substitute either archive; the staging tool verifies these exact
   identities before using a source fallback.
5. Rebuild native and WebAssembly artifacts, regenerate and verify retail bot
   navigation, ingest the CC0 arena, stage the audited
   self-authored CC0 content, and stage the source offer using the commands above.
6. Run `deploy/install.sh`, then `deploy/install.sh --start-local`; verify local health
   reports `status=ok`, `dedicated=true`, `retail=true`, and `retailBundles=true`.
7. Ensure `/api/replays` exposes a complete replay lasting at least 90 seconds, or set
   `Q4_E2E_REPLAY_ID` to the id of a specific replay meeting that same requirement.
   Run every pre-deploy and full-browser check, then run
   `deploy/install.sh --publish`. Restore the portal tile last.

Replay recordings are runtime evidence, not required application state. If retained,
restore only complete `.q4sd` plus matching metadata; discard `.part` files. Browser
saves and verified retail PK4s live in each user's origin-private OPFS. Within one
bundle revision, common and previously used map packs remain in the browser's cache;
only missing packs are fetched later. A missing server-catalog revision is downloaded
and verified again on the next game launch; manual-import revisions remain local to
that browser.

## Teardown

```bash
systemctl disable --now q4
rm -f /etc/cron.d/q4-caddy-watchdog
/srv/q4/tools/caddy_route_remove.sh
```

Do not delete `assets-src/gog` unless the operator is prepared to re-extract the
licensed installer. Never copy it into a backup intended for public distribution.
