Deployment

Overview

Aether manages infrastructure configs in git. The live server configs live at /opt/docker/{nginx,shared,monitoring,umami,backups}/. The Makefile syncs between them.

Git repo (aether)  ──make deploy──▸  Server (/opt/docker/)
Server (/opt/docker/)  ──make pull──▸  Git repo (aether)

Core Workflow

1. Edit configs in the repo

Make changes in the aether repo — nginx configs, Prometheus rules, Grafana dashboards, Docker Compose files, etc.

2. Review with diff

Always diff before deploying:

cd /opt/docker/aether/repo
make diff

This compares every tracked file in the repo against its server counterpart. Output shows CHANGED: <file> with a unified diff for each difference. Everything in sync means no drift.

in sync does not mean live

make diff compares files. It cannot tell you whether the running system has loaded them, and two services routinely have not:

• nginx includes *.conf, but deploy writes *.conf.full / *.conf.temp — files nginx never reads. Activation is a separate step (see make apply).
• Grafana provisions alert rules at startup, so a mounted-but-unapplied config is invisible to any file comparison.

Use make status to ask the different question: is what was deployed actually live?

3. Deploy to server

make deploy

This copies config files to their server locations and automatically:

• Runs nginx -t + nginx -s reload (validates config before reloading)
• Sends a reload signal to Prometheus (/-/reload)

Deploy uses install -C, not cp, so it writes only files whose content actually differs and prints exactly those. Empty output means nothing changed — the output is a changelog, not a list of every path.

This matters beyond tidiness: plain cp rewrote every file unconditionally, bumping mtime on byte-identical content. That made "is this config newer than its container?" useless as a signal and produced false positives in make status, which now compares content hashes instead.

4. Apply — make the deployed config live

make status             # what is deployed but not yet live?
make apply              # activate + reload
make apply SITE=helios-prod   # also take that one site OUT of maintenance

apply activates each deployed .conf.full into the bare .conf nginx reads, restarts the services that only read config at startup (Grafana, Loki, Alloy), and reloads those that support it (Prometheus, Blackbox).

It is deliberately not folded into deploy. .conf.temp is maintenance mode, so an auto-activating deploy would silently pull a site out of maintenance mid-incident. For the same reason apply refuses to activate a site that is currently on .temp, or to publish one that has never been activated, unless you name it with SITE=.

Restart decisions are made on config content hashes, not mtime — deploy copies unconditionally and bumps mtime on identical files, so an mtime check would restart services after every deploy for no reason.

5. Restart if needed

make deploy reloads nginx and Prometheus, but Docker Compose changes (new containers, image updates, volume changes) require a restart:

make restart            # All infra: shared, monitoring, umami, nginx

Per-Service Operations

Target	What it restarts	When to use
make restart-nginx	Nginx proxy	After docker-compose.yml or volume changes
make restart-monitoring	Prometheus, Grafana, Loki, Alloy, exporters	After docker-compose.yml changes
make restart-umami	Umami analytics	After docker-compose.yml changes
make restart-shared	PostgreSQL + Redis	After docker-compose.yml or postgres config changes

Warning

make restart-shared restarts PostgreSQL. All app connections will drop momentarily. Apps with connection pooling will reconnect automatically, but active queries will fail.

Pulling Server Changes

If you edited a config directly on the server (quick fix, debugging), pull it into git:

cd /opt/docker/aether/repo
make pull         # Server → repo
git diff          # Review what changed
git add <files> && git commit -m "Pull server config changes"

Security Config Deployment

Security configs (SSH, iptables, fail2ban, sysctl) live outside /opt/docker/ and have a separate deploy target:

make security-deploy

This copies configs to their system locations (/etc/ssh/, /etc/fail2ban/, etc.) but does not restart services automatically. After deploying:

sudo systemctl daemon-reload
sudo systemctl restart sshd fail2ban iptables-firewall

Secret Management

Secrets follow a different workflow — they're encrypted with SOPS + age.

Editing a secret on the server

# 1. Edit the plaintext file
sudo nano /opt/docker/aletheia/envs/.env.prod

# 2. Restart the affected service
cd /opt/docker/aletheia/repo && make restart ENV=prod

# 3. Verify the service is healthy

# 4. Encrypt back into the repo
cd /opt/docker/aether/repo && make encrypt

# 5. Commit the updated .enc file
git add -A && git commit -m "Update prod secrets"

make encrypt only rewrites .enc files whose decrypted content actually changed, so the diff in step 5 should contain exactly the files you edited. If it shows more than that, something else drifted — read it before committing. Scope a run explicitly with make encrypt FILES="envs/aletheia/.env.prod", or force a full re-encrypt (new data keys for every file) with make encrypt FORCE=1.

Comments are not round-tripped

SOPS's dotenv store keeps only key/value pairs. Comments and blank lines in a server .env file are dropped from the repo .enc copy, so make decrypt onto a fresh server will not restore them.

Decrypting secrets to the server

make decrypt    # All .enc files → server locations

See recurring tasks — how to rotate secrets for the full password rotation procedure.

Rollback

There is no automated rollback. To revert a config change:

# 1. Revert to the previous commit
git log --oneline -5          # find the commit to revert to
git checkout <commit> -- <file>

# 2. Re-deploy
make deploy

# 3. Restart if the change was in docker-compose.yml
make restart-nginx            # or whichever service

Safety Checklist

• [ ] make diff before make deploy — review all changes
• [ ] nginx -t passes (deploy does this automatically, but verify if unsure)
• [ ] Restart only the affected service, not everything
• [ ] After secret changes: encrypt → commit → push
• [ ] Test the service after deployment (health check, Grafana, etc.)