Runbooks

Common monitoring scenarios and how to handle them.

Reading this page

Two systems are sending you alerts right now. Since 2026-07-21 Alertmanager delivers the Prometheus rules, alongside Grafana which still delivers its own copy of about 12 of them. This parallel run is deliberate — the new path is being proven before the old one is removed — and it has one visible consequence: for mirrored rules you will get two emails for one problem, one titled with each system's name for it. That is expected, not a bug, and it ends when the Grafana rules are deleted.

Each section lists both names, because they differ and either may be the one in your inbox:

• Alert — the Prometheus rule name, delivered by Alertmanager to alerts@groupe-suffren.com. This is now a live delivery path.
• Grafana rule — the string on the Grafana-sent email or Teams card. Removed once Grafana drops to dashboards.

A section marked none — Alertmanager only has no Grafana mirror, so it does not produce a duplicate: you get a single email, under the Prometheus name, delivered by Alertmanager. (Before 2026-07-21 these had no delivery path at all — that phrasing is gone from this page because it is no longer true.)

Container Down / Restarting

Alert: ContainerDown, ContainerVanished, ContainerRestarted, HeliosContainerRestarting Grafana rule: "Container Down", "Container Restarted"

ContainerDown and ContainerVanished detect different failures — they are not duplicates, and which one fired tells you something:

	What it means	Names the container?
ContainerDown	The container is still known to cAdvisor but stopped reporting for 2 min. Usual case: it crashed or was stopped.	Yes — see {{ $labels.name }}
ContainerVanished	A monitored container disappeared from cAdvisor entirely — removed (docker rm), or cAdvisor itself died and took every series with it.	No — only the count

ContainerVanished firing alone, with no accompanying ContainerDown, most often means cAdvisor is the problem rather than the container. Check it first:

docker ps --filter name=monitoring_cadvisor
docker logs --tail 50 monitoring_cadvisor

If the summary reads 0 of N, that is the whole monitored set gone at once — almost certainly cAdvisor, not N simultaneous outages. (The count comes from the allow-list in containers.rules.yml; see Alerting for the current list.)

1. Check which container is affected:

   docker ps -a --filter "status=exited" --format "{{.Names}}\t{{.Status}}"
   

2. Check container logs for the crash reason:

   docker logs --tail 100 <container_name>
   

3. Check if OOM killed:

   docker inspect <container_name> | grep -A5 "State"
   

4. Restart if needed:

   # For app containers — use the app's Makefile
   cd /opt/docker/aletheia/repo && make restart ENV=prod
   
   # For infra containers — use Aether
   cd /opt/docker/aether/repo && make restart-monitoring
   

Systemd Unit Failed

Alert: SystemdUnitFailed Grafana rule: none — Alertmanager only (no Grafana mirror; a single email under the Prometheus name)

A systemd unit is in the failed state. This catches silent service failures like the Apr 2026 iptables outage where the firewall service failed for 10 days undetected.

1. Identify which unit failed (the name label on the alert):

   sudo systemctl --failed
   

2. Get the failure reason:

   sudo systemctl status <unit-name> --no-pager
   

3. Check recent logs for context:

   sudo journalctl -u <unit-name> --since "1 hour ago" --no-pager
   

4. Common causes:

   • ExecStart path no longer exists (like the iptables incident) — check the unit file
   • Dependency unavailable — e.g. service needs docker but docker is down
   • Config syntax error — unit file itself is malformed
   • Resource limit hit — memory, tasks, or time limits

5. Once fixed, restart the unit to clear the failed state:

   sudo systemctl restart <unit-name>
   sudo systemctl status <unit-name>
   

6. Verify no other units are failed: sudo systemctl --failed should show zero.

High CPU Usage

Alert: HighCpuUsage, ContainerHighCpu Grafana rule: "High CPU Usage", "Container High CPU"

1. Check which process/container is consuming CPU:

   docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}" | sort -k2 -rh
   

2. Check the Node Exporter Grafana dashboard for CPU breakdown (user, system, iowait)
3. If iowait is high: disk I/O bottleneck — check for large database queries or backup running
4. If user CPU is high: identify the container and check its logs for processing-heavy operations
5. For Celery workers: check for CPU-intensive tasks in the Celery dashboard
6. ContainerHighCpu specifically: a single container is pinned >1.5 cores while the host is otherwise fine. Almost always a self-feeding task loop or stuck process — check the alerting container's logs for the same task name / endpoint repeating. The 2026-05-13 MediaFile signal loop is the canonical example: process_media_variants re-firing post_save indefinitely.

High Memory Usage

Alert: HighMemoryUsage, ContainerHighMemory, ContainerHighMemoryUnlimited Grafana rule: "High Memory Usage"

1. Check per-container memory in the Docker Grafana dashboard
2. Identify the offending container:

   docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"
   

3. For Celery workers: check for memory leaks in long-running tasks
4. For PostgreSQL: check work_mem and active query count in the PostgreSQL dashboard

Which of the three fired matters

They watch different things, and the one that fired tells you where to look:

• ContainerHighMemory — a container is near its own mem_limit. The container is the problem and Docker will OOM-kill it; the host is probably fine. docker stats shows the percentage directly.
• ContainerHighMemoryUnlimited — a container with no mem_limit has grown past the absolute threshold, counting resident memory and swap. Nothing will stop it: there is no cgroup ceiling, so it grows until the host runs out and the kernel OOM killer picks a victim — which may well be a different, innocent container. Its value is that it names the container.
• HighMemoryUsage — the host itself is above its threshold. This one cannot name a culprit; if it fires alone, work down docker stats by absolute usage. Note it reads MemAvailable, which excludes swap — it can stay green through real pressure that swap is absorbing, so do not treat its silence as proof the host is healthy. HostSwapThrashing covers that case.
• HostSwapThrashing / HostSwapExhausted — the host is short of RAM and paying for it in page I/O, or has nearly used up the buffer that was hiding that shortage. See the swap section below.

Do not assume ContainerHighMemoryUnlimited gives you a head start on HighMemoryUsage. It can fire second, or alone, or not at all while the host suffers — the two thresholds land at roughly the same point on this host, and HighMemoryUsage has gone a full 15-day window without firing while the host bottomed out at ~81% used, because swap absorbed the pressure. Treat each alert as evidence about the thing it measures, not as a stage in a sequence.

Current thresholds, read from the rules:

Alert	Fires above	Sustained for
ContainerHighMemory	90% of the container's mem_limit	5m
ContainerHighMemoryUnlimited	1.5 GiB resident + swap	5m
HostSwapThrashing	500 pages/s sustained page-out	15m
HostSwapExhausted	75% of total swap in use	30m

ContainerHighMemoryUnlimited has no allow-list — it watches anything without a mem_limit, deliberately, so it can catch a container nobody enumerated. The set that qualifies today is shared_postgres, shared_redis, nginx-proxy and certbot (see below); check the alert's own name label rather than assuming it is one of those, because a newly added limit-less container joins the rule automatically. In practice it has meant shared_postgres:

1. Find the query holding the memory — a single runaway sort or a leaked connection pool is the usual cause:

   docker exec shared_postgres psql -U postgres -c \
     "SELECT pid, usename, datname, state, now()-query_start AS runtime, left(query,120) \
      FROM pg_stat_activity WHERE state <> 'idle' ORDER BY runtime DESC LIMIT 10;"
   

2. Cross-check the top query shapes in the PostgreSQL dashboard (pg_stat_statements is preloaded) before killing anything.
3. Terminate only if it is genuinely runaway: SELECT pg_terminate_backend(<pid>);
4. If usage is legitimate and simply growing, the fix is work_mem / shared_buffers tuning, not a mem_limit — see the note below.

Do not 'fix' this by adding a mem_limit

It is the obvious response and it is wrong for these containers. shared_postgres carries oom_score_adj: -800 and nginx-proxy -600, which tell the host OOM killer to kill something else first. A mem_limit creates a cgroup-local OOM kill that ignores oom_score_adj entirely — so adding one converts the two containers we most wanted to protect into the first ones killed. The limits are absent on purpose, and that is exactly why this alert exists.

Host Swapping

Alert: HostSwapThrashing, HostSwapExhausted Grafana rule: none — Alertmanager only (no Grafana mirror)

These exist because no other alert on this stack can see swap. HighMemoryUsage reads MemAvailable, which excludes it by construction, and the container memory rules read working set, which falls as the kernel pages a container out. So the stack could be entirely green while the host shed several gigabytes to disk — and was: over the 15-day window when this gap was found, minimum available memory hit 2.21 GiB (~81% used) without HighMemoryUsage firing, while swap in use climbed from 0 to 7.6 GiB.

Which one fired matters.

• HostSwapThrashing — happening now. The host is short of RAM and paying disk latency for it; everything on the box is slower, and it will not show up as a memory alert. This is a latency incident.

• HostSwapExhausted — building up. The buffer that has been quietly absorbing memory pressure is nearly gone. When swap fills, the next allocation failure goes to the kernel OOM killer, which picks its own victim.

• Confirm the shape — is it filling, draining, or churning?

  free -m
  vmstat 5 5          # si/so columns: sustained non-zero `so` is the alert
  

• Find who is swapped out. Per-container metrics do not show this well, so go to the processes:

  for f in /proc/[0-9]*/status; do \
    awk '/^Name:/{n=$2}/^VmSwap:/{if($2>50000)print $2, n}' $f 2>/dev/null; done \
    | sort -rn | head -15
  

• Map the worst offenders back to containers:

  docker stats --no-stream --format '{{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}'
  

• Decide which problem you have:

• A container is genuinely growing → it is the same investigation as High Memory Usage above. Start there; ContainerHighMemoryUnlimited counts swap, so it should be naming the container.
• Many workers each modestly oversized → the usual cause here is Celery and gunicorn worker counts, not a leak. Reducing worker concurrency reclaims more than tuning any single process.
• Nothing is growing and swap is simply old → pages swapped out during a past spike are not faulted back in until touched. If so is ~0 and only HostSwapExhausted fired, this is stale swap, not live pressure. It still needs clearing (swapoff -a && swapon -a, only with enough free RAM to hold the difference — check free -m first, it will OOM the box otherwise).

Why swap is not simply disabled

Swap is what keeps a transient overshoot from becoming an instant OOM kill, and shared_postgres / nginx-proxy carry negative oom_score_adj specifically so the host kills something else first. Removing swap removes that grace period. The goal is to notice swapping, not to eliminate it — which is what these two alerts are for.

Disk Space Low

Alert: DiskSpaceLow Grafana rule: "Disk Space Low"

1. Check disk usage:

   df -h /
   du -sh /opt/docker/*/  | sort -rh | head -20
   

2. Common space consumers:

3. Docker images: docker system df → docker system prune
4. Loki logs: check /opt/docker/monitoring/loki/ volume
5. PostgreSQL WAL: check /var/lib/postgresql/18/docker/pg_wal/

6. Backup archives: check /opt/docker/backups/

7. Clean Docker resources:

   docker system prune -f          # dangling images, stopped containers
   docker image prune -a -f        # unused images (use with caution)
   

PostgreSQL Issues

Alert: PostgresConnectionsHigh, PostgresDown Grafana rule: "PostgreSQL Connections High"

1. Check connection count:

   docker exec shared_postgres psql -U admin -c \
     "SELECT datname, count(*) FROM pg_stat_activity GROUP BY datname;"
   

2. Find long-running queries:

   docker exec shared_postgres psql -U admin -c \
     "SELECT pid, now() - pg_stat_activity.query_start AS duration, query
      FROM pg_stat_activity
      WHERE state != 'idle'
      ORDER BY duration DESC
      LIMIT 10;"
   

3. Kill a stuck query (last resort):

   docker exec shared_postgres psql -U admin -c "SELECT pg_terminate_backend(<pid>);"
   

Celery Queue Backlog

Alert: CeleryQueueBacklog, CeleryWorkerDown, CeleryHighFailRate Grafana rule: "Celery Task Queue Backlog"

1. Check queue depth in the Celery Grafana dashboard
2. Check worker status:

   cd /opt/docker/aletheia/repo && make celery-logs ENV=prod
   

3. Check for stuck tasks: look for tasks running longer than expected in the dashboard (p99 runtime panel)
4. Restart workers if needed:

   cd /opt/docker/aletheia/repo && make restart ENV=prod
   

SSL Certificate Expiring

Alert: SslCertExpiringSoon, HeliosSSLExpiringSoon Grafana rule: "SSL Certificate Expiring Soon"

Certificates should auto-renew via certbot. If they're not:

1. Check certbot logs:

   docker logs certbot --tail 50
   

2. Force renewal:

   docker exec certbot certbot renew --force-renewal
   

3. Reload nginx after renewal:

   docker exec nginx-proxy nginx -s reload
   

Health Check Failing

Alert: HealthCheckFailing, HeliosHealthCheckFailing Grafana rule: "Application Health Check Failed"

First decide which signal fired — they mean different things:

• HealthCheckFailing (any blackbox-health target non-200) is the authoritative backend signal. That job is Aletheia-only: /health/ (DB + Redis + Celery) and the API config probe …/api/v1/websites/sites/<domain>/config/ (the apps/websites DRF layer). A 200 from /health/ with a failing config probe means DB/Redis/Celery are fine but the websites view/serializer layer is broken. The Helios frontend roots are not in this job, so this alert never fires on a frontend-only outage.

• HeliosHealthCheckFailing is frontend reachability only (its own blackbox-helios-frontend job). Helios returns HTTP 200 with a soft-404 error UI when Aletheia is down, so this alert does not detect an Aletheia outage — it only fires when the practice site is fully unreachable (DNS/cert/process/nginx). For backend health, look at HealthCheckFailing, not this.

• Check from the server itself:

  curl -sI https://aletheia.groupe-suffren.com/health/                                              # backend deep health (200/503)
  curl -s  -o /dev/null -w '%{http_code}\n' \
    https://aletheia-staging.groupe-suffren.com/api/v1/websites/sites/cabinet-dentaire-aubagne.fr/config/   # API/DRF layer (expect 200)
  curl -sI https://cabinet-dentaire-aubagne.fr/                                                     # Helios frontend (200 even if backend is down!)
  

• If /health/ is 200 but the config probe is non-200: the DRF layer is broken (view/serializer/migration) — check Aletheia web logs, not the DB. The live config probe targets staging (aletheia-staging…), so a 404 there is a real regression — the probed SiteConfig.domain (Aubagne) was unseeded/deleted on staging; reseed it (restore_website_seed). (A 404 is only expected for the prod config probe, which is why that target stays commented out in prometheus.yml until practice-site launch.)

• If the app responds locally but not externally: check nginx config and DNS
• If the app doesn't respond locally: check container status and logs (see "Container Down" above)
• Check the blackbox targets in Prometheus UI (http://localhost:9090/targets) for specific probe failures

Helios Practice Site Unreachable (frontend)

Alert: HeliosHealthCheckFailing Grafana rule: "Helios Practice Site Unreachable (frontend)"

A practice site is fully unreachable — DNS, cert, the Next.js process, or the nginx vhost. This is not a backend alert: Helios returns HTTP 200 with a soft-404 error UI when Aletheia is down, so an Aletheia outage does not trigger this. For backend health see "Health Check Failing" above.

This rule is dormant until the first .fr cutover

The blackbox-helios-frontend job has no targets yet, so the rule is held at noDataState: OK — a job monitoring nothing would otherwise alert permanently. make helios-golive prints a step 3b reminding you to flip it to Alerting on the first real cutover. If this alert fires, a live practice site is down.

1. Confirm the outage is real and find which layer broke:

   curl -sI https://<practice-domain>/                     # expect 200
   getent hosts <practice-domain>                          # DNS resolves to this server?
   echo | openssl s_client -connect <practice-domain>:443 -servername <practice-domain> 2>/dev/null \
     | openssl x509 -noout -dates                          # cert valid + not expired?
   

2. If DNS or cert is the problem, the site was likely cut over incorrectly — check the vhost exists and is activated, not merely deployed:

   cd /opt/docker/aether/repo && make status               # flags deployed-but-not-live vhosts
   ls /opt/docker/nginx/conf.d/helios-fr-*.conf
   

3. If DNS and cert are fine, check the Helios container for that environment:

   docker ps --filter name=helios- --format '{{.Names}}\t{{.Status}}'
   docker logs --tail 100 helios-prod-web
   

4. Check the probe itself before assuming the site is down — this rule fires on probe failure, and a dead prober looks identical from here:

   # blackbox target health
   docker exec monitoring_prometheus wget -qO- 'http://localhost:9090/api/v1/query?query=probe_success{job="blackbox-helios-frontend"}'
   

High 5xx Error Rate

Alert: NginxHigh5xxRate Grafana rule: "High Nginx 5xx Error Rate"

1. Check nginx error log for the failing upstream:

   docker logs --tail 200 nginx-proxy 2>&1 | grep -E "5[0-9]{2}|error|upstream"
   

2. Identify which backend is returning errors:

   # Check per-service in Grafana → Nginx dashboard, or:
   docker logs --tail 100 nginx-proxy 2>&1 | grep " 50[0-9] " | awk '{print $7}' | sort | uniq -c | sort -rn
   

3. If a specific app is failing: check that app's container logs and health

4. If all backends are failing: check shared services (PostgreSQL, Redis) — a database outage causes 500s across all apps
5. If nginx itself is the issue: docker exec nginx-proxy nginx -t to validate config

Redis Issues

Alert: RedisDown (if configured), or detected via app errors

1. Check Redis container status:

   docker ps --filter name=shared_redis
   docker logs --tail 50 shared_redis
   

2. Test connectivity:

   docker exec shared_redis redis-cli ping
   # Expected: PONG
   

3. Check memory usage:

   docker exec shared_redis redis-cli info memory | grep used_memory_human
   

4. Check per-database key counts (prod=0/1, staging=2/3, dev=4/5):

   docker exec shared_redis redis-cli info keyspace
   

5. If Redis is unresponsive, restart:

   cd /opt/docker/aether/repo && make restart-shared
   

Note

Redis is used for Celery broker and Django cache. A Redis outage will cause Celery tasks to stop processing and may degrade app response times.

Nginx Routing Issues

Symptom: 502 Bad Gateway, 504 Gateway Timeout, or requests reaching the wrong service

1. Test nginx config syntax:

   docker exec nginx-proxy nginx -t
   

2. Check which config is active (.conf.full = HTTPS, .conf.temp = maintenance):

   ls -la /opt/docker/nginx/conf.d/*.conf
   

3. Check nginx error log for upstream failures:

   docker logs --tail 100 nginx-proxy 2>&1 | grep -E "error|upstream"
   

4. Verify the upstream container is running and on the correct network:

   # Check the container exists and is on the backend network
   docker inspect <container_name> --format '{{range $net,$conf := .NetworkSettings.Networks}}{{$net}} {{end}}'
   

5. After config changes, reload (not restart) nginx:

   docker exec nginx-proxy nginx -s reload
   

Monitoring Stack Down

Symptom: Grafana unreachable, no alerts firing, Prometheus targets showing as down

1. Check all monitoring containers:

   docker ps -a --filter "name=monitoring_" --format "table {{.Names}}\t{{.Status}}"
   

2. If Prometheus is down:

   docker logs --tail 50 monitoring_prometheus
   # Common cause: bad config syntax after editing alert rules
   cd /opt/docker/aether/repo && make restart-monitoring
   

3. If Loki is down (log gap risk):

   docker logs --tail 50 monitoring_loki
   # Check disk space — Loki will stop ingesting if disk is full
   df -h /
   du -sh /opt/docker/monitoring/loki/
   

4. If Alloy (log collector) is down:

   docker logs --tail 50 monitoring_alloy
   # Alloy needs access to the Docker socket
   ls -la /var/run/docker.sock
   

5. Restart the full monitoring stack:

   cd /opt/docker/aether/repo && make restart-monitoring
   

Warning

While the monitoring stack is down, no alerts will fire. Check containers manually with docker ps -a until monitoring is restored.

Alertmanager Notifications Failing

Alert: AlertmanagerNotificationsFailing

Symptom: Alertmanager is up and evaluating, but its email leg to Brevo is dropping notifications. This is partial failure — the notifier works, delivery does not. (Total Alertmanager failure is not caught here — it is caught by the external Watchdog heartbeat, which pages via healthchecks.io when the stack stops pinging. See the Watchdog runbook below.)

This alert delivers over the path it is reporting on

If email is totally down, this alert's own email will not arrive either. It reaches you on degraded email (intermittent failures, single drops), and — while the Grafana parallel run lasts — through Grafana's independent channel. Do not treat its silence as proof email is healthy.

1. Confirm the failure and see why:

   docker exec monitoring_prometheus wget -qO- \
     'http://localhost:9090/api/v1/query?query=alertmanager_notifications_failed_total%7Bintegration%3D%22email%22%7D'
   # the `reason` label (clientError / serverError / contextDeadlineExceeded …)
   # tells you whether Brevo rejected it (auth/quota) or the connection timed out
   

2. Check Alertmanager's own logs for the SMTP error text:

   docker logs --tail 50 monitoring_alertmanager | grep -i "notify\|smtp\|email"
   

3. Most common cause — the derived SMTP password is empty or stale:

   sudo test -s /opt/docker/monitoring/alertmanager/smtp_password && echo "present" || echo "EMPTY"
   # if empty: the SMTP_PASSWORD key is missing from monitoring/.env.
   # `make decrypt` then `make deploy` re-derives it (deploy now aborts rather
   # than blanking it, but an already-blanked file needs this to refill).
   

4. If the credential is fine, check Brevo status (quota, IP allow-list, key rotation) — serverError / clientError reasons point there rather than at the config.

Alert Emails Are Not Arriving

Alert: none — and that is the whole point of this section.

Symptom: You suspect alerts are not reaching the inbox, or you want to prove they are. Nothing on this stack can page you about this, because every guard stops at the moment the relay says 250 OK: AlertmanagerNotificationsFailing sees SMTP errors but not silent drops, and the Watchdog heartbeat never touches SMTP at all, so it keeps healthchecks.io green through a total email outage. Email is the sole channel, so this is the one failure that can be complete and invisible at the same time.

This is therefore a procedure you run, not an alert you wait for.

1. Establish where the message stopped. Alertmanager counts what it handed over:

   docker exec monitoring_alertmanager wget -qO- http://localhost:9093/metrics \
     | grep -E '^alertmanager_notifications_(total|failed_total)\{integration="email"'
   

   total rising with failed_total at 0 means Alertmanager and the relay are fine and the loss is downstream — go to step 3. If failed_total is rising, this is the AlertmanagerNotificationsFailing runbook above instead.

Note these counters reset when the container restarts, so total 0 on a recently restarted Alertmanager means "nothing sent since restart", not "nothing ever sent".

1. Send a real test alert end-to-end. It routes to the email receiver like any other (only alertname="Watchdog" is diverted), so this exercises the exact production path:

   NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
   END=$(date -u -d '+2 minutes' +%Y-%m-%dT%H:%M:%SZ)
   docker exec monitoring_alertmanager sh -c "wget -qO- \
     --post-data='[{\"labels\":{\"alertname\":\"DeliverabilityAudit\",\"severity\":\"none\"},
       \"annotations\":{\"summary\":\"synthetic test - ignore\"},
       \"startsAt\":\"$NOW\",\"endsAt\":\"$END\"}]' \
     --header='Content-Type: application/json' \
     http://localhost:9093/api/v2/alerts"
   

   It flushes after group_wait (30s); re-run step 1 and expect total to have gone up by one. The alert self-resolves after two minutes.

2. Read the message in the destination mailbox — including its headers. This is the step no tooling here can do for you, and it is the only thing that distinguishes "delivered" from "accepted by the relay". Check the spam and quarantine folders, not just the inbox.

In the raw message source, find Authentication-Results. Expected, and measured on 2026-07-22:

• dkim=pass with a signing domain matching the sender domain. This is the one that matters. DKIM is the only thing making DMARC pass on this path (see alerting.md for why SPF structurally cannot), so dkim=fail is an outage of the alerting channel itself, even while mail appears to send fine.
• spf=pass — expected, and not about our SPF record: the relay rewrites the envelope sender to its own domain, so the pass belongs to the relay. Our record is not consulted.
• spf=fail / softfail — this would mean the relay has stopped rewriting the envelope sender, at which point the sender domain's -all goes live and starts hard-failing every alert. Re-score spf-authorizes-relay in check_email_deliverability.py and publish include:spf.brevo.com. Not the situation today.

If anything here differs from the above, record the new reading in the roadmap entry — it is expensive to re-derive and cheap to write down.

1. If the relay accepted it and the mailbox never got it, the loss is at the relay or the receiver. Check, in this order: the relay's own transactional log for that message ID (bounce, blocked, suppression list); whether the recipient address landed on a suppression list after an earlier bounce; and the receiving tenant's quarantine.

2. Check the posture that makes silent quarantine likely in the first place:

   make docs-check   # check 5 — SPF/DKIM/DMARC vs the configured relay
   

The relay account is shared

The same relay account and sender address are used by Grafana and by aletheia's application mail. A quota exhausted or a reputation damaged by application mail degrades the alerting path with it, and neither is visible from the monitoring stack. If alerts stopped arriving with no config change, check whether application mail volume changed.

Watchdog / Monitoring Heartbeat

Alert: Watchdog

This alert is different from every other one on this page. It is designed to fire constantly (expr: vector(1)) and is routed only to healthchecks.io, never to email. You will never be paged by it. You get paged when it stops — healthchecks.io alerts on the absence of the heartbeat ping, over a channel (its own email/SMS/etc.) that shares no infrastructure with our stack.

So a healthchecks.io "down" notification means the monitoring stack itself is not delivering — Prometheus is not evaluating, Alertmanager is down, or the host/network is gone. It is the one signal that can catch total stack death, because it is checked from outside the stack. (It does not prove alert emails work — the ping path never touches Brevo/SMTP; that leg is watched separately by AlertmanagerNotificationsFailing.)

When healthchecks.io reports the heartbeat missing:

1. Is the whole box down? Check from another host that the server responds at all before assuming it is a monitoring-only fault.

2. Is Prometheus evaluating?

   docker ps --filter name=monitoring_prometheus
   docker exec monitoring_prometheus wget -qO- 'http://localhost:9090/api/v1/query?query=ALERTS%7Balertname%3D%22Watchdog%22%7D'
   # a firing Watchdog here but no ping arriving => the break is downstream (Alertmanager / webhook / egress)
   

3. Is Alertmanager up and routing it?

   docker ps --filter name=monitoring_alertmanager
   docker logs --tail 50 monitoring_alertmanager | grep -i "watchdog\|webhook\|healthchecks"
   

4. Can the container reach healthchecks.io, and is the ping URL present?

   sudo test -s /opt/docker/monitoring/alertmanager/healthchecks_url && echo "present" || echo "EMPTY"
   # if empty: HEALTHCHECKS_PING_URL is missing from monitoring/.env.
   # `make decrypt` then `make deploy` re-derives it (deploy aborts rather than
   # blanking it, but an already-blanked file needs this to refill).
   docker exec monitoring_alertmanager wget -qO- "$(sudo cat /opt/docker/monitoring/alertmanager/healthchecks_url)" && echo "ping OK"
   

5. Restart the stack if a container is wedged (make restart-monitoring). A successful ping resolves the healthchecks.io alert automatically.

Tuning the detection window

healthchecks.io fires after Period + Grace with no ping. Those are set on the healthchecks.io side and must stay comfortably larger than Alertmanager's repeat_interval for the Watchdog route (1m) so a single missed flush is not a false alarm. Current values: Period 5m, Grace 5m → detects a real stall in roughly 10 minutes. Change the two together, never one alone.

Grafana Access Issues

If Grafana is unreachable at monitoring.groupe-suffren.com:

1. Check the container:

   docker ps --filter name=monitoring_grafana
   

2. Check nginx proxy config:

   docker exec nginx-proxy nginx -t
   cat /opt/docker/nginx/conf.d/monitoring.conf
   

3. Check Grafana logs:

   docker logs monitoring_grafana --tail 50
   

Alertmanager UI Unreachable

https://monitoring.groupe-suffren.com/alertmanager/ is the silence/ack path. It being down does not stop notifications — Alertmanager can deliver email perfectly while its UI is unreachable — so this is an access problem, not an alerting outage. (If notifications have also stopped, that is the Watchdog's job; see the healthchecks.io runbook above.)

1. 401 with a password prompt = working as designed. The credentials are the shared /opt/docker/nginx/.htpasswd, the same ones staging uses.

2. 404 or the Grafana login page instead of Alertmanager → the trailing slash or the location block. Check the deployed config actually has it — the server reads monitoring.conf, not monitoring.conf.full:

   grep -A2 alertmanager /opt/docker/nginx/conf.d/monitoring.conf
   docker exec nginx-proxy nginx -t
   

   If the block is missing, the config was deployed but never activated: make apply, then make status.

3. 502 → nginx cannot resolve or reach the container. It is reached by name over the web network, so a container recreated without that network is the usual cause:

   docker ps --filter name=monitoring_alertmanager
   docker inspect -f '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}' monitoring_alertmanager
   # expect: monitoring web
   docker exec nginx-proxy wget -qO- http://monitoring_alertmanager:9093/-/healthy
   

   Fix with make restart-monitoring (the compose file declares both networks).