Alerting

Alert Rules

Defined in Prometheus (prometheus/alerts/), and — until the Phase C migration finishes — mirrored for about 12 of them in Grafana Unified Alerting.

The tables below carry the rationale for each threshold, which is the part that cannot be extracted from config. For the authoritative list of rules, conditions and severities, see the auto-generated catalog:

Alert	Condition	For	Severity	Source
AlertmanagerNotificationsFailing	increase(alertmanager_notifications_failed_total{integrat...	2m	warning	alertmanager.rules.yml
HealthCheckFailing	Probe returns failure	2m	critical	application.rules.yml
NginxHigh5xxRate	5xx error rate > 5%	5m	warning	application.rules.yml
CeleryQueueBacklog	Queue length > 50	10m	warning	application.rules.yml
CeleryWorkerDown	Celery worker down	2m	critical	application.rules.yml
CeleryHighFailRate	Task failure rate > 10%	5m	warning	application.rules.yml
ContainerRestarted	Container restarted	—	warning	containers.rules.yml
ContainerDown	time() - max by(name) (container_last_seen{name=~"alethei...	2m	critical	containers.rules.yml
ContainerVanished	(count(group by(name) (container_last_seen{name=~"alethei...	2m	critical	containers.rules.yml
ContainerHighMemory	Container memory > 90% of its limit	5m	warning	containers.rules.yml
ContainerHighMemoryUnlimited	Unlimited container over 1.5 GiB (resident + swap)	5m	warning	containers.rules.yml
ContainerHighCpu	Container CPU > 150% (1.5 cores)	5m	warning	containers.rules.yml
HeliosContainerRestarting	Restarted more than 2 times in the window	—	warning	helios.rules.yml
HeliosHealthCheckFailing	Probe returns failure	5m	critical	helios.rules.yml
HeliosSSLExpiringSoon	SSL cert expires in < 14 days	1h	warning	helios.rules.yml
HighCpuUsage	CPU usage > 80%	5m	warning	node.rules.yml
HighMemoryUsage	Memory usage > 85%	5m	warning	node.rules.yml
HostSwapThrashing	Host swapping out > 500 pages/s	15m	warning	node.rules.yml
HostSwapExhausted	Host swap > 75% used	30m	critical	node.rules.yml
DiskSpaceLow	Disk free < 15%	5m	critical	node.rules.yml
SystemdUnitFailed	node_systemd_unit_state{state="failed"} == 1	2m	critical	node.rules.yml
PostgresConnectionsHigh	Connections > 80% of max	5m	warning	postgres.rules.yml
PostgresDown	PostgreSQL is down	1m	critical	postgres.rules.yml
SslCertExpiringSoon	SSL cert expires in < 14 days	5m	warning	ssl.rules.yml
Watchdog	vector(1)	—	none	watchdog.rules.yml

Node (Infrastructure)

Alert	Condition	Severity	Rationale
HighCpuUsage	> 80% for 5 min	warning	Sustained high CPU impacts all containers. 80% leaves headroom before saturation.
HighMemoryUsage	> 85% for 5 min	warning	OOM kills start above 90%. 85% gives time to react.
DiskSpaceLow	< 15% free for 5 min	critical	Below 15% risks database corruption (WAL writes fail).

Containers

Alert	Condition	Severity	Rationale
ContainerRestarted	New restart detected	warning	Catches crash loops and OOM kills early. Scoped to the allow-list below, not all containers — aletheia-local-* / -dev-* stop routinely during development.
ContainerDown	Monitored container stopped reporting > 2 min	critical	Core services — immediate attention required. Names the container, but needs the series to still exist, so it cannot see a container removed outright.
ContainerVanished	Fewer monitored containers reporting than the allow-list expects	critical	The vanish-proof half of ContainerDown: catches a container removed entirely, or cAdvisor itself dying and taking every series with it. Cannot name which one went.
ContainerHighMemory	> 90% of memory limit for 5 min	warning	Approaching OOM kill territory. Measures working set, not container_memory_usage_bytes — the latter includes reclaimable page cache, which the kernel evicts rather than OOM-killing on, and overstated Grafana by 15 points (91% vs 76%). Denominator guarded with > 0: containers with no mem_limit report a limit of 0, and x/0 = +Inf made this fire unconditionally for all 8 of them.
ContainerHighMemoryUnlimited	Container with no mem_limit past the absolute threshold below	warning	The other half of the rule above. The > 0 denominator guard removes limit-less containers from ContainerHighMemory entirely, which left 8 of them with no memory alert at all — shared_postgres most importantly, where a runaway query was silent until the host OOM killer took down something unrelated. Scored on absolute bytes because these containers have no limit to be a percentage of, and counts resident + swap: working set alone falls as the kernel pages a container out, so a swap-heavy runaway shrinks out of view exactly when it matters.
HostSwapThrashing	Sustained page-out (see thresholds below)	warning	Nothing else here reads a swap metric, and HighMemoryUsage cannot: MemAvailable excludes swap, so pressure relieved by swapping was invisible stack-wide. Over one 15-day window the host bottomed at 2.21 GiB available (~81%) without firing, while swap climbed 0 → 7.6 GiB unremarked.
HostSwapExhausted	Swap nearly full (see thresholds below)	critical	Swap is the buffer hiding memory pressure from every other rule; when it fills, the next allocation failure goes to the OOM killer, which picks its own victim regardless of oom_score_adj.
ContainerHighCpu	> 150% (1.5 cores) for 5 min	warning	Catches single-container runaways (e.g. self-queueing Celery loops) that HighCpuUsage misses because they don't move the host-wide average.

ContainerRestarted, ContainerDown and ContainerVanished share one allow-list. The resource alerts (ContainerHighMemory, ContainerHighMemoryUnlimited, ContainerHighCpu) deliberately do not use it — they exclude only -local- / -dev- containers, because a resource alert should catch the runaway nobody thought to enumerate:

The container alert allow-list covers 19 containers:

• aletheia-prod-beat
• aletheia-prod-celery
• aletheia-prod-celery-heavy
• aletheia-prod-web
• aletheia-staging-beat
• aletheia-staging-celery
• aletheia-staging-celery-heavy
• aletheia-staging-web
• helios-prod-web
• helios-staging-web
• monitoring_alloy
• monitoring_blackbox_exporter
• monitoring_cadvisor
• monitoring_loki
• monitoring_prometheus
• nginx-proxy
• shared_postgres
• shared_redis
• umami

Memory and swap thresholds, read from the rules themselves:

Alert	Fires above	Sustained for
ContainerHighMemory	90% of the container's mem_limit	5m
ContainerHighMemoryUnlimited	1.5 GiB resident + swap	5m
HostSwapThrashing	500 pages/s sustained page-out	15m
HostSwapExhausted	75% of total swap in use	30m

The two container-memory rules partition the containers rather than overlapping: one with a mem_limit is scored by ratio (ContainerHighMemory), one without is scored by absolute bytes (ContainerHighMemoryUnlimited). Each series is matched to its own container's limit (on(name, id)), and a container is treated as unlimited unless a real positive limit is visible — so a missing container_spec_memory_limit_bytes over-alerts rather than falling between the two rules.

That last point is the whole design constraint, and it was got wrong on the first attempt. The original join asked "is there a limit reading that says zero", which meant a missing reading silenced both rules; and it compared each container against the highest limit seen under its name, so a recreated container answered for its own ghost and left a ~10-minute window covered by neither. Both were reproduced with promtool and fixed on 2026-07-22.

make test-alerts asserts the partition on the specific cases listed in tests/regression.test.yml — including the two above and the ghost-series cases. It does not prove coverage for an arbitrary container, so treat it as a regression guard on known failure modes rather than a proof of the invariant. A gap here is invisible by nature: its symptom is that nothing fires.

PostgreSQL

Alert	Condition	Severity	Rationale
PostgresConnectionsHigh	> 80% of max_connections for 5 min	warning	Connection exhaustion causes new requests to fail.
PostgresDown	Exporter reports down	critical	Database outage — all apps affected.

Application

Alert	Condition	Severity	Rationale
HealthCheckFailing	Any blackbox-health (Aletheia-only) probe non-200 for 2 min	critical	Aletheia backend is unreachable, or its API surface (/api/v1/websites/.../config/) is broken. Frontend roots are a separate job — not this alert.
NginxHigh5xxRate	> 5% of requests are 5xx for 5 min	warning	Signals backend errors or overload.
CeleryQueueBacklog	> 50 pending messages for 10 min	warning	Tasks piling up — worker may be stuck or down.
CeleryWorkerDown	Worker offline > 2 min	critical	Background tasks stop processing entirely.
CeleryHighFailRate	> 10% failure rate over 5 min	warning	Systematic task failures (bad data, external service down).

SSL

Alert	Condition	Severity	Rationale
SslCertExpiringSoon	Expires in < 14 days	warning	Certbot should auto-renew at 30 days. If it hasn't by 14 days, renewal is broken.

Helios

Alert	Condition	Severity	Rationale
HeliosContainerRestarting	> 2 restarts in 15 min	warning	Crash loop — investigate logs.
HeliosHealthCheckFailing	Practice site root probe non-200 > 5 min	critical	Frontend reachability only (DNS/cert/process/nginx). Does not detect Aletheia outages — see the soft-404 caveat below.
HeliosSSLExpiringSoon	Practice domain SSL < 14 days to expiry	warning	Practice site will show browser warning.

Notification Routing

Two systems are delivering right now — expect duplicate emails

Since 2026-07-21 Alertmanager delivers the Prometheus rules, while Grafana still delivers its own copy of about 12 of them. For a mirrored rule you will receive two emails for one problem, titled differently by each system. This parallel run is deliberate: the new path is being proven before the old one is deleted. It ends when the Grafana rules are removed.

Path	Delivers	Channel	Config
Alertmanager	The Prometheus rules in prometheus/alerts/	Email (relay and addresses under Email Deliverability below, generated from the config)	monitoring/alertmanager/alertmanager.yml
Grafana	Its own ~12 mirrored rules	Email + Teams	grafana/provisioning/alerting/

Alertmanager is email-only by design: Teams cards do not produce notifications anyone reliably reads, and a channel nobody reads is false comfort rather than redundancy.

Timing (both paths use the same values, so a duplicate arrives on the same cadence and is recognisable as a duplicate rather than a second incident):

• Group wait: 30 seconds (batch initial alerts)
• Group interval: 5 minutes (batch follow-up updates)
• Repeat interval: 4 hours (re-send if still unresolved)

Severity-based routing and inhibition are deliberately not configured yet — you only learn what is actually noisy once delivery works, and building routing rules speculatively bakes in guesses about alert volume nobody has observed.

The notifier is watched — by a dead-man's switch

No alert can report that Alertmanager is down, because that alert would have to travel through Alertmanager. The same is true of Grafana, and of cAdvisor. This is closed by the always-firing Watchdog rule (vector(1)), routed — and only it — to an external heartbeat service (healthchecks.io) via an Alertmanager webhook on a 1-minute repeat. healthchecks.io alerts on the absence of that ping, over a channel that shares no infrastructure with our stack, so total stack death (Prometheus, Alertmanager, or the host) is caught from outside. It does not exercise the email/Brevo leg — that is watched separately by AlertmanagerNotificationsFailing. See the Watchdog runbook.

Email Deliverability

Email is the sole notification channel, so its failure modes are the whole delivery risk. The path:

Property	Value
SMTP relay	smtp-relay.brevo.com:587
Sender (From:)	noreply@groupe-suffren.com
Sender domain governing SPF/DKIM/DMARC	groupe-suffren.com
Recipients	alerts@groupe-suffren.com
Transport security	required (STARTTLS)

Note the sender is a third-party relay (Brevo) sending as a domain whose mail is otherwise received by Microsoft 365, and the same Brevo account is shared with Grafana and with aletheia's application mail. That sharing is worth knowing during an incident: quota exhaustion or a reputation hit caused by application mail lands on the alerting path too.

What each guard actually proves — and where they all stop:

Guard	Proves	Blind to
make test-alerts	The rule fires on the series it watches	Delivery entirely
docs-check check 4	The metric the rule reads exists	Delivery entirely
AlertmanagerNotificationsFailing	The SMTP handoff is not erroring	Anything after a 250 OK
Watchdog → healthchecks.io	The stack is alive and routing	The email leg — the ping never touches SMTP
docs-check check 5	SPF/DKIM/DMARC authorize the configured relay	Whether a message reached a mailbox

They stack up to "the message was handed to Brevo", and stop there. A 250 OK means Brevo took the message, not that anyone received it — the same caveat the retired Teams webhook's ok carried. The gap that leaves is a real one and is named rather than waved off: Alertmanager healthy, Watchdog pinging green, and the relay silently dropping or quarantining every message. Nothing on this side can see it.

Check 5 closes the most likely cause of that gap rather than the gap itself. The usual reason a relay accepts mail nobody reads is authentication — a receiver quarantining a message whose SPF/DKIM/DMARC posture looks wrong — so the check derives the relay and sender from alertmanager.yml and asserts the sender domain's DNS actually authorizes them. Run it with make docs-check.

SPF does not authorize the relay — and that turns out not to matter

The sender domain's SPF record authorizes Microsoft 365 and ends in -all (hard fail); the relay is not in it. That looks alarming, and was treated as an open risk until it was measured.

A delivered alert's Authentication-Results header came back spf=pass on 2026-07-22. That is impossible against a record which does not authorize the relay — so the relay is rewriting the SMTP envelope sender to its own bounce domain, and SPF is evaluated against the envelope sender, not the From: header. The sender domain's record is not consulted for mail on this path at all.

So do not "fix" this by adding include:spf.brevo.com. It would change nothing. DMARC alignment compares the From: domain against the domain SPF actually authenticated, and with the envelope rewritten to the relay's own domain, SPF can never align here regardless of what we publish.

Delivery therefore rests entirely on DKIM, which is configured and aligned (brevo1/brevo2 keys, verified live). Treat a DKIM failure as an outage of the alerting channel itself — it is the only thing holding DMARC up. Check 5 scores DKIM and reports SPF as informational, so the SPF line stays visible on every run: if the relay ever stops rewriting the envelope, the -all goes live again and it becomes a real finding.

Silencing an Alert

The Alertmanager UI is at https://monitoring.groupe-suffren.com/alertmanager/ (exposed 2026-07-22), behind the shared .htpasswd — Alertmanager has no authentication of its own, and anyone who reaches it can silence every alert on the stack, which looks exactly like an outage going unreported.

Use it to silence a known-noisy or in-progress alert instead of editing the rule: Silences → New Silence, or the Silence button on a firing alert. A silence needs an end time, so it expires on its own; a commented-out rule does not, which is how alerting quietly rots.

A silence hides the symptom, not the cause

Silence for the length of the work, not "until someone remembers". If an alert is chronically noisy the threshold is wrong — fix the rule (see below) rather than renewing the silence, and record why in the rule comment.

Two things the UI is not:

• It is not a second delivery channel. Alertmanager still emails; the UI is a console onto the same state.
• Silencing Watchdog would stop the heartbeat pings and page you via healthchecks.io ~10 minutes later, which is the dead-man's switch working correctly. Don't.

Notification emails link back here (--web.external-url), so the silence button is one click from the alert that prompted it. Alertmanager itself keeps serving at the root internally — nginx strips the /alertmanager/ prefix — so Prometheus's alerting: stanza and make deploy's reload are unaffected by the exposure; the rationale for that split is in nginx/conf.d/monitoring.conf.full.

Blackbox Probes

External monitoring via HTTP health checks and SSL certificate probes:

SSL certificate monitoring:

• https://aletheia.groupe-suffren.com
• https://aletheia-staging.groupe-suffren.com
• https://aletheia-dev.groupe-suffren.com
• https://monitoring.groupe-suffren.com
• https://analytics.groupe-suffren.com
• https://cosmos-staging.groupe-suffren.com

Aletheia backend health monitoring (HealthCheckFailing):

• https://aletheia.groupe-suffren.com/health/
• https://aletheia-staging.groupe-suffren.com/health/
• https://aletheia-dev.groupe-suffren.com/health/
• https://aletheia-staging.groupe-suffren.com/api/v1/websites/sites/cda/config/

Helios masks backend outages — the authoritative Aletheia signal is the /health/ + API config probes

Helios (the Next.js frontend) returns HTTP 200 with a generic soft-404 error UI when the Aletheia API throws on its home/blog/team pages. So a 200 from a Helios practice-site root (cabinet-dentaire-aubagne.fr, le-canet.chirurgiens-dentistes.fr) is a false green during an Aletheia outage. Those root probes are therefore treated as frontend-reachability only (they catch a fully-down site — DNS/cert/process/nginx — e.g. a probe value of 0/timeout). They live in a dedicated blackbox-helios-frontend scrape job, separate from the Aletheia blackbox-health job, so they drive HeliosHealthCheckFailing alone and never the backend HealthCheckFailing.

The independent, authoritative backend signal is the blackbox-health job's Aletheia targets:

• …/health/ — deep-checks DB + Redis + Celery (200/503).
• …/api/v1/websites/sites/<domain>/config/ — exercises the apps/websites DRF view/serializer layer end-to-end. /health/ stays 200 if only that API layer breaks, so this probe closes the gap. The https_2xx module accepts only HTTP 200, so any non-200 (404/500/503) trips probe_success → HealthCheckFailing (critical).

Coverage caveat (2026-06-03): the API config probe runs against staging only — staging is seeded with the 4 practice SiteConfigs. Prod has no SiteConfig seeded yet (practice sites unlaunched; helios-prod in maintenance), so the prod config probe is committed but commented out in prometheus.yml; enable it at practice-site launch. Staging runs the same image, so it still catches API-layer regressions before they reach prod.

Modifying Alert Rules

1. Edit the relevant file in monitoring/prometheus/alerts/
2. Add or update a test in monitoring/prometheus/tests/ — see below
3. Run make test-alerts — asserts the rules actually fire on the series they claim to watch
4. Run make deploy, then make apply, then make status to confirm it is live
5. For Grafana-side rules (until they are removed): edit monitoring/grafana/provisioning/alerting/rules.yml. Grafana provisions at startup, so this needs a container restart — make apply does it.

The pre-commit hook runs both make docs-check and make test-alerts when anything under prometheus/alerts/ or prometheus/tests/ changes.

A rule that looks green is not a rule that works

Four rules in this repo have been found green-but-dead, in four distinct classes, and every one was caught by a human reading the expression rather than by tooling:

Class	Example	Why it looked fine
Inverted evaluator	probe_success == 0 as a Grafana query with a gt 0 threshold	PromQL == filters, it does not return a boolean, so the evaluator asked 0 > 0
Missing metric	celery_tasks_total{state="PENDING"} — never emitted by celery-exporter	NoData + noDataState: OK renders green
Dead conjunction	increase(container_last_seen[1m]) == 0 and …	The left side is never 0 for a running container
Never stops firing	usage / spec_limit where the limit is 0	x/0 = +Inf, and +Inf > 90 is always true

A fifth shape is worth knowing even though it is not a silent rule: a rule that fires correctly but measures the wrong quantity. ContainerHighMemory read container_memory_usage_bytes, which counts reclaimable page cache the kernel simply evicts — so it reported real-looking pressure that did not exist. Tests catch this only if they feed both candidate metrics in one case, with values that disagree; a test feeding one metric passes either way.

The first cost a 5-day undetected staging outage (Jul 15-20 2026). The fourth was invisible until Alertmanager gave it somewhere to deliver.

This is why make test-alerts exists. It asserts firing behaviour ("given these series, this alert fires") rather than inspecting expression shape, which is the only form of check that catches all four classes. When you add a rule, add both a fires-when-it-should and a stays-silent-when-it -shouldn't case — a test that only asserts firing would pass on a rule hardwired to fire forever, which is exactly class four.