feat(security): ops alerts, shared backoff, atomic quota and daily stats for CrowdSec
Gitea Actions Runner Test / test-job (push) Successful in 1s
CI / check (push) Successful in 29s
CI / tests-integration (push) Successful in 1m36s
CI / tests-unit (push) Successful in 1m40s
CI / tests-ui (push) Successful in 2m28s
CI / preflight (push) Skipped
CI / deploy (push) Successful in 2m3s
Gitea Actions Runner Test / test-job (push) Successful in 1s
CI / check (push) Successful in 29s
CI / tests-integration (push) Successful in 1m36s
CI / tests-unit (push) Successful in 1m40s
CI / tests-ui (push) Successful in 2m28s
CI / preflight (push) Skipped
CI / deploy (push) Successful in 2m3s
Add an alerting/stats layer over the existing CrowdSec integration: - New crowdsec-alerts.ts: cooldown-gated ops alerts (Redis NX lock, TTL from HEALTH_ALERT_COOLDOWN_MIN) fanning out through the app's sendAlert service. Raised for daily quota exhaustion, block bursts (5-min window past CROWDSEC_ALERT_BLOCK_BURST), and signal-push failures. - New crowdsec-stats.ts: daily counters (lookups/blocks/reports/report_fail) in Redis with a 14-day reader for the admin panel. - Shared 403/429 backoff: the pause marker now lives in Redis (crowdsec:backoff-until) so every instance honours it, not just the process that hit the limit. - Atomic quota reservation: INCR-before-call with self-rollback on overshoot, so concurrent instances can never slip calls past the daily ceiling. - Admin anti-DDoS page gains a last-14-days activity table next to the quota bar.
This commit is contained in:
1 parent
5e4fc9ab59
commit
301edd2c9a
9 files changed
+620
-28
No files matched your search
@@ -0,0 +1,66 @@
|
||||
import "server-only";
|
||||
|
||||
import { env } from "@/env";
|
||||
import { logger } from "@/lib/logger";
|
||||
import { redis } from "@/lib/redis";
|
||||
import { type SendAlertInput, sendAlert } from "@/lib/services/alert";
|
||||
|
||||
// === CrowdSec operational alerts ===========================================
|
||||
//
|
||||
// Thin, fire-and-forget wrapper around the app's alert service for the
|
||||
// reputation pipeline. Every raise is cooldown-gated through a Redis NX lock
|
||||
// (key crowdsec:alert:{key}, TTL = HEALTH_ALERT_COOLDOWN_MIN), so N instances
|
||||
// and flapping conditions surface exactly one alert per window instead of
|
||||
// spamming Discord/email/alert_logs. Falls back to alerting anyway when Redis
|
||||
// is unreachable — a silent quota blowout is worse than one duplicate alert.
|
||||
|
||||
const ALERT_PREFIX = "crowdsec:alert:";
|
||||
|
||||
function cooldownSeconds(): number {
|
||||
const raw = Number(env.HEALTH_ALERT_COOLDOWN_MIN ?? 15);
|
||||
return Math.ceil((Number.isFinite(raw) && raw > 0 ? raw : 15) * 60);
|
||||
}
|
||||
|
||||
/**
|
||||
* Raise an alert unless the cooldown window is still active. Returns the
|
||||
* sendAlert promise when the alert was actually raised, or false when it was
|
||||
* suppressed. Never throws; the caller may `void` the result on hot paths.
|
||||
*/
|
||||
export async function raiseCrowdsecAlert(
|
||||
key: string,
|
||||
input: {
|
||||
type?: string;
|
||||
severity: SendAlertInput["severity"];
|
||||
message: string;
|
||||
context?: SendAlertInput["context"];
|
||||
},
|
||||
): Promise<false | Awaited<ReturnType<typeof sendAlert>>> {
|
||||
if (redis) {
|
||||
try {
|
||||
const acquired = await redis.set(
|
||||
`${ALERT_PREFIX}${key}`,
|
||||
String(Date.now()),
|
||||
"EX",
|
||||
cooldownSeconds(),
|
||||
"NX",
|
||||
);
|
||||
if (acquired !== "OK") return false;
|
||||
} catch {
|
||||
// Cooldown bookkeeping failed — alert anyway rather than silently drop.
|
||||
}
|
||||
}
|
||||
try {
|
||||
return await sendAlert({
|
||||
type: "ddos",
|
||||
severity: input.severity,
|
||||
message: input.message,
|
||||
context: input.context,
|
||||
});
|
||||
} catch (error) {
|
||||
logger.error("[crowdsec-alert] sendAlert raised an unexpected error", {
|
||||
key,
|
||||
err: error,
|
||||
});
|
||||
return false;
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user