openhands
301edd2c9a
Gitea Actions Runner Test / test-job (push) Successful in 1s
CI / check (push) Successful in 29s
CI / tests-integration (push) Successful in 1m36s
CI / tests-unit (push) Successful in 1m40s
CI / tests-ui (push) Successful in 2m28s
CI / preflight (push) Skipped
CI / deploy (push) Successful in 2m3s
Add an alerting/stats layer over the existing CrowdSec integration: - New crowdsec-alerts.ts: cooldown-gated ops alerts (Redis NX lock, TTL from HEALTH_ALERT_COOLDOWN_MIN) fanning out through the app's sendAlert service. Raised for daily quota exhaustion, block bursts (5-min window past CROWDSEC_ALERT_BLOCK_BURST), and signal-push failures. - New crowdsec-stats.ts: daily counters (lookups/blocks/reports/report_fail) in Redis with a 14-day reader for the admin panel. - Shared 403/429 backoff: the pause marker now lives in Redis (crowdsec:backoff-until) so every instance honours it, not just the process that hit the limit. - Atomic quota reservation: INCR-before-call with self-rollback on overshoot, so concurrent instances can never slip calls past the daily ceiling. - Admin anti-DDoS page gains a last-14-days activity table next to the quota bar.
67 lines
2.0 KiB
TypeScript
67 lines
2.0 KiB
TypeScript
import "server-only";
|
|
|
|
import { env } from "@/env";
|
|
import { logger } from "@/lib/logger";
|
|
import { redis } from "@/lib/redis";
|
|
import { type SendAlertInput, sendAlert } from "@/lib/services/alert";
|
|
|
|
// === CrowdSec operational alerts ===========================================
|
|
//
|
|
// Thin, fire-and-forget wrapper around the app's alert service for the
|
|
// reputation pipeline. Every raise is cooldown-gated through a Redis NX lock
|
|
// (key crowdsec:alert:{key}, TTL = HEALTH_ALERT_COOLDOWN_MIN), so N instances
|
|
// and flapping conditions surface exactly one alert per window instead of
|
|
// spamming Discord/email/alert_logs. Falls back to alerting anyway when Redis
|
|
// is unreachable — a silent quota blowout is worse than one duplicate alert.
|
|
|
|
const ALERT_PREFIX = "crowdsec:alert:";
|
|
|
|
function cooldownSeconds(): number {
|
|
const raw = Number(env.HEALTH_ALERT_COOLDOWN_MIN ?? 15);
|
|
return Math.ceil((Number.isFinite(raw) && raw > 0 ? raw : 15) * 60);
|
|
}
|
|
|
|
/**
|
|
* Raise an alert unless the cooldown window is still active. Returns the
|
|
* sendAlert promise when the alert was actually raised, or false when it was
|
|
* suppressed. Never throws; the caller may `void` the result on hot paths.
|
|
*/
|
|
export async function raiseCrowdsecAlert(
|
|
key: string,
|
|
input: {
|
|
type?: string;
|
|
severity: SendAlertInput["severity"];
|
|
message: string;
|
|
context?: SendAlertInput["context"];
|
|
},
|
|
): Promise<false | Awaited<ReturnType<typeof sendAlert>>> {
|
|
if (redis) {
|
|
try {
|
|
const acquired = await redis.set(
|
|
`${ALERT_PREFIX}${key}`,
|
|
String(Date.now()),
|
|
"EX",
|
|
cooldownSeconds(),
|
|
"NX",
|
|
);
|
|
if (acquired !== "OK") return false;
|
|
} catch {
|
|
// Cooldown bookkeeping failed — alert anyway rather than silently drop.
|
|
}
|
|
}
|
|
try {
|
|
return await sendAlert({
|
|
type: "ddos",
|
|
severity: input.severity,
|
|
message: input.message,
|
|
context: input.context,
|
|
});
|
|
} catch (error) {
|
|
logger.error("[crowdsec-alert] sendAlert raised an unexpected error", {
|
|
key,
|
|
err: error,
|
|
});
|
|
return false;
|
|
}
|
|
}
|