6 Commits
Author SHA1 Message Date
openhands 8a6d92afd8 fix(cloudflare): cache the gamedata tree at the edge with respect_origin
Gitea Actions Runner Test / test-job (push) Successful in 1s
CI / check (push) Successful in 32s
CI / tests-unit (push) Successful in 1m44s
CI / tests-integration (push) Successful in 1m44s
CI / tests-ui (push) Successful in 2m31s
CI / preflight (push) Skipped
CI / deploy (push) Successful in 2m16s
Icons are plain .png, a cacheable extension by default, so the zone's
"Browser Cache TTL = 1 year" pinned them to max-age=31536000 regardless of
the 300/3600/604800 that nginx sends per class. Extend the edge rule to
/gamedata/ and keep respect_origin, so the nginx header wins and a 404
(notably no-store from the gamedata 404 handler) is never pinned.
2026-10-01 18:00:48 +02:00
openhands dbaccd7cfc fix(gamedata): never cache a missing gamedata file
A missing gamedata file got no Cache-Control at all, because add_header
without `always` only applies to 2xx/3xx. Cloudflare then fell back to the
zone setting "Browser Cache TTL = 1 year", so the 404 came back as
`max-age=31536000` with `cf-cache-status: HIT` — pinned in the visitor's
browser and at the edge. An icon requested while its import was still
running stayed a 404 for the rest of the year, even after the file existed.
That was the "some icons load, some don't" report.

Give every gamedata location a named 404 handler that sends no-store, and
split icons/ out as its own cache class: those files are rewritten under
the same name (repair-icons, reimport), so an hourly must-revalidate keeps
a repaired icon visible within the hour instead of days later.
2026-10-01 18:00:36 +02:00
openhands 4e036b08d5 fix(proxy): drop request rate limiting from gamedata entirely
/gamedata/* is served straight from disk by nginx; no request hits the CMS
backend or a database, so a request-rate limit protects nothing while
costing players their icons. A room load fires hundreds of these files in
one burst, which every limit turned into visible 503s.

Removed the static zone from the gamedata locations. Traefik's
epicnabbo-gamedata router likewise carries no rateLimit middleware.
/client/ and /nitro-client/ keep theirs, and the main route keeps the
30r/s page budget plus the server-wide connection limit.

Measured: 1000 icon requests fired fully in parallel now all return 200,
while 200 parallel requests on / are still rejected.
2026-10-01 17:56:48 +02:00
openhands f0dcf440a7 fix(proxy): split rate limiting into page and static zones
The single server-scope limit_req (30r/s) treated a page load and a room
load as the same thing. Loading a Nitro room fires several hundred gamedata
icons in one burst, which that zone answered with 503s, so icons showed up
late in the client.

Add a separate static zone (1000r/s, burst 1000, nodelay) for the gamedata
and client asset locations, and apply the page-rate zone explicitly on the
main route instead of at server scope. Connection limit stays server-wide.

Measured: 900 icon requests in burst now all return 200, while 200 parallel
requests on / are still rejected.
2026-10-01 17:49:41 +02:00
openhands 4a1211a931 feat(proxy): add per-IP rate and connection limits
The edge had no limit_req/limit_conn at all, so a single client could
flood the Next.js backend and the Nitro client with unbounded parallel
requests. Traefik's logs already showed this: bursts of gamedata icon
requests answered with 429.

Add limit_req (30r/s, burst 60, nodelay) and limit_conn (30) zones keyed
on the real client IP, applied at server scope so both cached assets and
proxied API routes share one budget. The burst is deliberately generous
because the Nitro client fetches gamedata and icons in bursts when
loading a room.
2026-10-01 17:25:49 +02:00
openhands e3c010f383 fix(proxy): raise nginx worker rlimit above worker_connections
nginx inherited systemd's soft LimitNOFILE of 1024, so every start logged
"2048 worker_connections exceed open file resource limit: 1024" and the
worker_connections value could not actually be reached.

Set worker_rlimit_nofile to 65536. Bounded from above by a systemd drop-in
at /etc/systemd/system/nginx.service.d/override.conf (LimitNOFILE=65536),
since the master's hard limit caps what workers may request.
2026-10-01 17:11:04 +02:00
3 changed files with 98 additions and 9 deletions

No files matched your search

+65 -2
View File
@@ -192,6 +192,12 @@ server {
keepalive_timeout 30s;
send_timeout 10s;
# Abuse limits. Deliberately NOT set at server scope: a room load and a page
# load are not the same request profile, so each location picks its own zone.
# /gamedata/* has no request limit at all — it is a disk cache, so limiting
# it only cost players their icons. The page routes carry the budget.
limit_conn cms_conn_per_ip 30;
# Traefik health-check route herstellen
location = /health {
access_log off;
@@ -203,6 +209,7 @@ server {
location ^~ /client/ {
alias /var/www/Octane/dist/;
try_files $uri $uri/ =404;
limit_req zone=cms_static_per_ip burst=1000 nodelay;
location ~* \.(js|json|css|html|wasm|ttf|woff|woff2|gif|webp|png|jpg|jpeg|svg|dat)$ {
add_header Cache-Control "public, max-age=2592000";
@@ -216,6 +223,7 @@ server {
location ^~ /nitro-client/ {
alias /var/www/Octane/dist/;
try_files $uri $uri/ =404;
limit_req zone=cms_static_per_ip burst=1000 nodelay;
location ~* \.(js|json|css|html|wasm|ttf|woff|woff2|gif|webp|png|jpg|jpeg|svg|dat)$ {
add_header Cache-Control "public, max-age=2592000";
@@ -234,7 +242,7 @@ server {
location = /gamedata { return 301 /gamedata/config/; }
location = /gamedata/ { return 301 /gamedata/config/; }
# ─── Gamedata: drie cache-klassen, want niet alles onder /gamedata/ is
# ─── Gamedata: vier cache-klassen, want niet alles onder /gamedata/ is
# even veranderlijk.
#
# Dit pad had één regel voor de hele boom: `max-age=604800` (7 dagen). De
@@ -251,7 +259,13 @@ server {
# dat de bestandsnaam verandert (schalen, repareren), dus
# ook revalideren, maar minder vaak: ze worden veel vaker
# opgehaald dan ze worden geschreven.
# 3. alles wat overblijft (c_images, album*, clothes, …) — content-addressed
# 3. icons/ — `{classname}_icon.png`. Wordt wél herschreven onder
# dezelfde naam (repair-icons.ts, herimport), dus ook
# klasse 4's "nooit herschreven" geldt hier niet. Wel
# minder vaak dan 2: per uur een must-revalidate is één
# 304 per icon per uur, en een gerepareerd icon is zo
# binnen een uur zichtbaar in plaats van dagenlang oud.
# 4. alles wat overblijft (c_images, album*, clothes, …) — content-addressed
# of per item uniek, nooit herschreven onder dezelfde naam. Blijft lang.
location ^~ /gamedata/config/ {
alias /var/www/Gamedata/config/;
@@ -266,6 +280,7 @@ server {
add_header Content-Type "text/plain; charset=utf-8";
return 204;
}
error_page 404 = @gamedata_missing;
}
location ^~ /gamedata/bundled/ {
@@ -281,6 +296,27 @@ server {
add_header Content-Type "text/plain; charset=utf-8";
return 204;
}
error_page 404 = @gamedata_missing;
}
location ^~ /gamedata/icons/ {
alias /var/www/Gamedata/icons/;
add_header Cache-Control "public, max-age=3600, must-revalidate";
access_log off;
add_header Cache-Tag "cms-gamedata";
# Geen limit_req: gamedata is schijf-cache, geen CMS-backend. Een
# kamerladen vuurt honderden bestanden in één burst af en elke limiet
# hier leidde alleen tot zichtbaar gemiste icons.
add_header Access-Control-Allow-Origin $http_origin always;
add_header Access-Control-Allow-Methods "GET, OPTIONS" always;
if ($cors_headers) {
add_header Access-Control-Max-Age 1728000;
add_header Content-Type "text/plain; charset=utf-8";
return 204;
}
error_page 404 = @gamedata_missing;
}
location /gamedata/ {
@@ -296,6 +332,32 @@ server {
add_header Content-Type "text/plain; charset=utf-8";
return 204;
}
error_page 404 = @gamedata_missing;
}
# Een ONTBREKEND gamedata-bestand mag nooit gecacht worden, en daarom
# krijgt elke 404 hier een eigen handler.
#
# Zonder deze handler stuurde nginx op een 404 helemaal geen Cache-Control:
# `add_header` geldt zonder `always` alleen voor 2xx/3xx. Cloudflare vond
# dan geen expliciete cache-instructie en nam de zone-instelling over:
# "Browser Cache TTL = 1 jaar". Gevolg: de 404 kwam terug als
# `cache-control: max-age=31536000` met `cf-cache-status: HIT` — dus
# vastgezet in de browser van de bezoeker én op de edge. Een icon dat één
# keer te vroeg werd opgevraagd (import nog bezig) bleef daarom het hele
# jaar een 404, ook nadat het bestand er wél stond. Dat was de "sommige
# icons laden wel, sommige niet"-klacht.
#
# `no-store` (niet een korte TTL): het bestand kan elk moment verschijnen,
# dus er is geen enkel venster waarin we een 404 willen vasthouden. De
# Cache-Tag blijft meegegeven zodat een al gecachte 404 alsnog te purgen is
# via `scripts/cf-purge.sh cms-gamedata`.
location @gamedata_missing {
add_header Cache-Control "no-store" always;
add_header Cache-Tag "cms-gamedata" always;
add_header Access-Control-Allow-Origin $http_origin always;
add_header Access-Control-Allow-Methods "GET, OPTIONS" always;
return 404;
}
location /camera/ {
@@ -447,6 +509,7 @@ server {
# ─── Hoofd-routering ───
location / {
proxy_pass http://cms_app;
limit_req zone=cms_req_per_ip burst=60 nodelay;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
+18
View File
@@ -12,6 +12,11 @@
user www-data;
worker_processes auto;
# Raise the file-descriptor rlimit for the workers. Must stay <= the master's
# RLIMIT_NOFILE *hard* limit, otherwise nginx refuses to start with
# "setrlimit(RLIMIT_NOFILE) failed". Bounded from above by the systemd drop-in
# /etc/systemd/system/nginx.service.d/override.conf (LimitNOFILE=65536).
worker_rlimit_nofile 65536;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log warn;
@@ -39,6 +44,19 @@ http {
client_header_buffer_size 1k;
large_client_header_buffers 4 8k;
# Rate limiting per client IP.
#
# Two zones, because a room load and a page load are not the same thing.
# Loading a Nitro room fires several hundred gamedata icons in one burst;
# at the page rate that produced 503s on real players. Static assets
# therefore get their own, much higher allowance. These are small immutable
# files, so a request rate is not what protects them anyway — nginx already
# serves them with must-revalidate, and the CMS upstream stays behind
# cms_req_per_ip for the expensive routes.
limit_req_zone $binary_remote_addr zone=cms_req_per_ip:10m rate=30r/s;
limit_req_zone $binary_remote_addr zone=cms_static_per_ip:10m rate=1000r/s;
limit_conn_zone $binary_remote_addr zone=cms_conn_per_ip:10m;
# Blue/green cutover: ci-deploy.sh writes the active upstream here, and
# `proxy_pass http://cms_app` below follows it via graceful nginx -s reload.
upstream cms_app {
+15 -7
View File
@@ -1,6 +1,7 @@
#!/usr/bin/env bash
# Create/update the Cloudflare Cache Rule that stores the CMS public API
# allowlist at the edge (the routes nginx tags with `Cache-Tag: cms-public`).
# allowlist plus the client-facing gamedata tree at the edge (the routes nginx
# tags with `Cache-Tag: cms-public` respectievelijk `cms-gamedata`).
#
# Why a rule is required: Cloudflare only caches a handful of file extensions
# by default; `/api/*` responses are served `cf-cache-status: DYNAMIC` even
@@ -8,12 +9,19 @@
# with "Cache Everything" turns those the other way.
#
# What the rule does:
# - edge_ttl bypass_by_default : edge cachet volgens de s-maxage van nginx;
# zonder (publieke) header (bv. errorresponses) juist NIET cachen.
# - edge_ttl bypass_by_default : edge cachet volgens de max-age/s-maxage van
# nginx; zonder (publieke) header (bv. errorresponses) juist NIET cachen.
# - browser_ttl respect_origin : de zone heeft "Browser Cache TTL = 1 jaar" en
# overschrijft daarmee het max-age dat nginx per klasse stuurt. Deze rule
# herstelt dat voor de publieke API's: browsers krijgen de korte
# max-age van nginx terug (10/60/300s) i.p.v. een jaar stale data.
# herstelt dat voor de publieke API's én /gamedata/: browsers krijgen de
# per-klasse max-age van nginx terug i.p.v. een jaar stale data.
#
# Het /gamedata/-deel is toegevoegd omdat de zone-TTL anders ook de iconen
# (`.png`, een standaard cachebare extensie) op een jaar zette: nginx stuurde
# 300/3600/604800, maar de browser kreeg `max-age=31536000` en een 404 werd als
# `max-age=31536000` + `cf-cache-status: HIT` vastgezet. Een ontbrekend icon dat
# later werd geïmporteerd bleef daardoor een jaar 404. Met `respect_origin` geldt
# de nginx-header, en die stuurt op een 404 juist `no-store`.
#
# Idempotent: vergelijkt de bestaande rule (op description + inhoud) en zet
# alleen bij als die verschilt. Re-running is veilig.
@@ -27,12 +35,12 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="$SCRIPT_DIR/../.env"
BASE="https://api.cloudflare.com/client/v4"
PHASE="http_request_cache_settings"
DESCRIPTION="EpicNabbo CMS public API edge cache (cms-public)"
DESCRIPTION="EpicNabbo CMS public API + gamedata edge cache (cms-public, cms-gamedata)"
# Cache de allowlist exact zoals nginx hem tagt (deployment/proxy/nginx-cms.conf).
# Geen regex: `matches` vereist Business; vrije operators zijn `in` en
# `starts_with()`.
EXPRESSION='(http.request.method eq "GET") and (http.request.uri.path in { "/api/staff" "/api/teams" "/api/guilds" "/api/photos" "/api/leaderboard" "/api/online" "/api/online/count" "/api/shop" "/api/shop/categories" "/api/values" "/api/values/categories" "/api/radio/current-dj" "/api/radio/points/leaderboard" } or starts_with(http.request.uri.path, "/api/values/"))'
EXPRESSION='(http.request.method eq "GET") and (http.request.uri.path in { "/api/staff" "/api/teams" "/api/guilds" "/api/photos" "/api/leaderboard" "/api/online" "/api/online/count" "/api/shop" "/api/shop/categories" "/api/values" "/api/values/categories" "/api/radio/current-dj" "/api/radio/points/leaderboard" } or starts_with(http.request.uri.path, "/api/values/") or starts_with(http.request.uri.path, "/gamedata/"))'
load_env() {
local name="$1"