drinktracker/deploy

Deploying drinktracker

./deploy/deploy.sh              # typecheck, push, build, swap, restart, verify
./deploy/deploy.sh --yes        # skip the confirmation prompt
./deploy/deploy.sh --rollback   # repoint at the previous release (~3s, no rebuild)

The app runs natively under systemd — there is no Docker on this host.

Where things live

Production LXC	192.168.2.169 (hostname drinktracker, Proxmox vmid 100)
Deploy access	ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169
App	/opt/drinktracker/{repo,releases/<sha>,current}, service user drinktracker
Secrets	/etc/drinktracker/drinktracker.env (root:drinktracker 0640)
Services	drinktracker, minio, postgresql@16-main
Database	drinkman, user drinkdb, 127.0.0.1:5432
Object storage	MinIO at 127.0.0.1:9000, bucket drink-images
App port	3000 on 0.0.0.0; unauthenticated requests return 307 → /login
Backups	/var/backups/drinktracker, nightly 03:25, 14-day retention

Traps

drinktracker.tenseconddelay.net resolves to the reverse proxy (192.168.2.172), not the container. NEXTAUTH_URL points at the public name, so resolving it and SSHing there fails in a way that looks like a rejected key but is simply a different machine.

next.config.mjs evaluates rewrites() at BUILD time. The /minio-images/:path* destination is baked into routes-manifest.json as http://localhost:9000/drink-images/:path*. Changing MINIO_ENDPOINT in the env file appears to work and does nothing — images break only after the next rebuild, so the change and the breakage are separated in time. MinIO must stay on localhost:9000 serving drink-images.

The image proxy is an unauthenticated GET. Next rewrites don't sign requests, so the bucket's anonymous-download policy is load-bearing. Verify with mc anonymous get local/drink-images → must say download.

prisma db push, not migrate deploy. Production has no _prisma_migrations table, so migrate deploy would try to apply the init migration against populated tables and fail. --accept-data-loss means removing a field from schema.prisma drops the column and its data — the deploy script dumps first for this reason.

How a deploy works

Build happens in /opt/drinktracker/repo and is assembled out-of-place into releases/<sha>, so the ~3 minute build runs while the current release keeps serving. Only the symlink swap and systemctl restart cause downtime, about 3 seconds. Rollback is the same swap in reverse and needs no rebuild.

The build deliberately does not source the env file — same as the Dockerfile — because of the build-time rewrite baking described above.

Docker still works, just not here

Dockerfile and docker-compose.prod.yml are maintained for running this app elsewhere (a VPS). docker compose build && docker compose up -d works on any normal Docker host.

It does not work on this LXC, which is why the app runs natively. Two independent problems, both from unprivileged-LXC nesting:

1. AppArmor — the container can't load the docker-default profile (apparmor_parser: Access denied), so any container without --security-opt apparmor=unconfined fails.
2. runc can't set net.ipv4.ip_unprivileged_port_start, so containers only work with --network host.

docker build is impossible regardless: Docker 29 removed the classic builder and BuildKit spawns a builder container without the unconfined option. Docker is installed but disabled; the old containers and volumes are retained until 2026-09-07 as a rollback path.

Verifying by hand

ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169
systemctl is-active postgresql@16-main minio drinktracker
sudo journalctl -u drinktracker -n 50
sudo journalctl -u drinktracker | grep '\[switchboard\]'    # AI routing + cost per call
readlink /opt/drinktracker/current                          # which release is live

Each AI call logs one [switchboard] feature=… model=… cost=… latency_ms=… line. FAILOVER warnings are expected — the gateway's :batch model variants currently fail on every request and fall back. CONTEXT OVERFLOW is worth investigating.

Backups

drinktracker-backup.timer runs nightly at 03:25: pg_dump + a tar of /var/lib/minio/data + a copy of the env file, into /var/backups/drinktracker, retained 14 days (~1 GB).

TODO — the chain has no off-site leg. /usr/local/bin/drinktracker-backup supports OFFSITE_DEST (an scp target) and OFFSITE_KEY; set them via Environment= in drinktracker-backup.service. Until then everything lives on one LXC on one Proxmox host, and the nightly job prints a warning saying so.

Restore:

sudo systemctl stop drinktracker
zcat /var/backups/drinktracker/pg-<stamp>.sql.gz | sudo -u postgres psql -d drinkman
sudo systemctl stop minio
sudo tar -C /var/lib/minio -xzf /var/backups/drinktracker/minio-<stamp>.tgz
sudo chown -R minio:minio /var/lib/minio/data
sudo systemctl start minio drinktracker

Notes

• The deploy needs unattended sudo on the LXC (/etc/sudoers.d/drinkadmin-deploy, currently NOPASSWD: ALL). A narrowly-scoped allowlist was considered and rejected: the script calls root for install, tee, systemctl, journalctl and sudo -u postgres pg_dump, so an allowlist would need updating every time the script changes and would fail in confusing ways when it drifted. Be aware this buys little security either way — drinkadmin is in the sudo group, so full root is a password away regardless. What it grants is unattended root.
• nodejs is apt-mark hold-ed at v20.20.0 so the native runtime can't drift from the Dockerfile's pinned node:20-alpine.
• Postgres was created with en_US.UTF-8; the Ubuntu default of C/SQL_ASCII would re-sort every drink list and mangle names like Grüner Veltliner. Preserve the locale if the cluster is ever rebuilt.
• unattended-upgrades is not installed, so nothing will surprise-restart these services — but postgres will never auto-patch either. A deliberate open question.