# Deploying drinktracker ```bash ./deploy/deploy.sh # typecheck, push, build, swap, restart, verify ./deploy/deploy.sh --yes # skip the confirmation prompt ./deploy/deploy.sh --rollback # repoint at the previous release (~3s, no rebuild) ``` The app runs **natively under systemd — there is no Docker on this host.** ## Where things live | | | |---|---| | Production LXC | `192.168.2.169` (hostname `drinktracker`, Proxmox vmid 100) | | Deploy access | `ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169` | | App | `/opt/drinktracker/{repo,releases/,current}`, service user `drinktracker` | | Secrets | `/etc/drinktracker/drinktracker.env` (`root:drinktracker` `0640`) | | Services | `drinktracker`, `minio`, `postgresql@16-main` | | Database | `drinkman`, user `drinkdb`, `127.0.0.1:5432` | | Object storage | MinIO at `127.0.0.1:9000`, bucket `drink-images` | | App port | 3000 on `0.0.0.0`; unauthenticated requests return 307 → `/login` | | Backups | `/var/backups/drinktracker`, nightly 03:25, 14-day retention | ## Traps **`drinktracker.tenseconddelay.net` resolves to the reverse proxy (192.168.2.172), not the container.** `NEXTAUTH_URL` points at the public name, so resolving it and SSHing there fails in a way that looks like a rejected key but is simply a different machine. **`next.config.mjs` evaluates `rewrites()` at BUILD time.** The `/minio-images/:path*` destination is baked into `routes-manifest.json` as `http://localhost:9000/drink-images/:path*`. Changing `MINIO_ENDPOINT` in the env file appears to work and does nothing — images break only after the *next* rebuild, so the change and the breakage are separated in time. MinIO must stay on `localhost:9000` serving `drink-images`. **The image proxy is an unauthenticated GET.** Next rewrites don't sign requests, so the bucket's anonymous-download policy is load-bearing. Verify with `mc anonymous get local/drink-images` → must say `download`. **`prisma db push`, not `migrate deploy`.** Production has no `_prisma_migrations` table, so `migrate deploy` would try to apply the init migration against populated tables and fail. `--accept-data-loss` means removing a field from `schema.prisma` **drops the column and its data** — the deploy script dumps first for this reason. ## How a deploy works Build happens in `/opt/drinktracker/repo` and is assembled **out-of-place** into `releases/`, so the ~3 minute build runs while the current release keeps serving. Only the symlink swap and `systemctl restart` cause downtime, about 3 seconds. Rollback is the same swap in reverse and needs no rebuild. The build deliberately does **not** source the env file — same as the Dockerfile — because of the build-time rewrite baking described above. ## Docker still works, just not here `Dockerfile` and `docker-compose.prod.yml` are maintained for running this app elsewhere (a VPS). `docker compose build && docker compose up -d` works on any normal Docker host. It does **not** work on this LXC, which is why the app runs natively. Two independent problems, both from unprivileged-LXC nesting: 1. AppArmor — the container can't load the `docker-default` profile (`apparmor_parser: Access denied`), so any container without `--security-opt apparmor=unconfined` fails. 2. runc can't set `net.ipv4.ip_unprivileged_port_start`, so containers only work with `--network host`. `docker build` is impossible regardless: Docker 29 removed the classic builder and BuildKit spawns a builder container without the unconfined option. Docker is installed but disabled; the old containers and volumes are retained until 2026-09-07 as a rollback path. ## Verifying by hand ```bash ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169 systemctl is-active postgresql@16-main minio drinktracker sudo journalctl -u drinktracker -n 50 sudo journalctl -u drinktracker | grep '\[switchboard\]' # AI routing + cost per call readlink /opt/drinktracker/current # which release is live ``` Each AI call logs one `[switchboard] feature=… model=… cost=… latency_ms=…` line. `FAILOVER` warnings are expected — the gateway's `:batch` model variants currently fail on every request and fall back. `CONTEXT OVERFLOW` is worth investigating. ## Backups `drinktracker-backup.timer` runs nightly at 03:25: `pg_dump` + a tar of `/var/lib/minio/data` + a copy of the env file, into `/var/backups/drinktracker`, retained 14 days (~1 GB). **TODO — the chain has no off-site leg.** `/usr/local/bin/drinktracker-backup` supports `OFFSITE_DEST` (an scp target) and `OFFSITE_KEY`; set them via `Environment=` in `drinktracker-backup.service`. Until then everything lives on one LXC on one Proxmox host, and the nightly job prints a warning saying so. Restore: ```bash sudo systemctl stop drinktracker zcat /var/backups/drinktracker/pg-.sql.gz | sudo -u postgres psql -d drinkman sudo systemctl stop minio sudo tar -C /var/lib/minio -xzf /var/backups/drinktracker/minio-.tgz sudo chown -R minio:minio /var/lib/minio/data sudo systemctl start minio drinktracker ``` ## Notes - The deploy needs unattended sudo on the LXC (`/etc/sudoers.d/drinkadmin-deploy`, currently `NOPASSWD: ALL`). A narrowly-scoped allowlist was considered and rejected: the script calls root for `install`, `tee`, `systemctl`, `journalctl` and `sudo -u postgres pg_dump`, so an allowlist would need updating every time the script changes and would fail in confusing ways when it drifted. Be aware this buys little security either way — `drinkadmin` is in the `sudo` group, so full root is a password away regardless. What it grants is *unattended* root. - `nodejs` is `apt-mark hold`-ed at v20.20.0 so the native runtime can't drift from the Dockerfile's pinned `node:20-alpine`. - Postgres was created with `en_US.UTF-8`; the Ubuntu default of `C`/`SQL_ASCII` would re-sort every drink list and mangle names like `Grüner Veltliner`. Preserve the locale if the cluster is ever rebuilt. - `unattended-upgrades` is not installed, so nothing will surprise-restart these services — but postgres will never auto-patch either. A deliberate open question.