# Deploying drinktracker ```bash ./deploy/deploy.sh # typecheck, push, build, swap, restart, verify ./deploy/deploy.sh --yes # skip the confirmation prompt ./deploy/deploy.sh --rollback # repoint at the previous release (~3s, no rebuild) ``` The app runs **natively under systemd — there is no Docker on this host.** ## Where things live | | | |---|---| | Production LXC | `192.168.2.169` (hostname `drinktracker`, Proxmox vmid 100) | | Deploy access | `ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169` | | App | `/opt/drinktracker/{repo,releases/,current}`, service user `drinktracker` | | Secrets | `/etc/drinktracker/drinktracker.env` (`root:drinktracker` `0640`) | | Services | `drinktracker`, `minio`, `postgresql@16-main` | | Database | `drinkman`, user `drinkdb`, `127.0.0.1:5432` | | Object storage | MinIO at `127.0.0.1:9000`, bucket `drink-images` | | App port | 3000 on `0.0.0.0`; unauthenticated requests return 307 → `/login` | | Backups | `/var/backups/drinktracker`, nightly 03:25, 14-day retention | ## Traps **`drinktracker.tenseconddelay.net` resolves to the reverse proxy (192.168.2.172), not the container.** `NEXTAUTH_URL` points at the public name, so resolving it and SSHing there fails in a way that looks like a rejected key but is simply a different machine. **Images are served by an authenticated route, not a rewrite.** `next.config.mjs` has no `rewrites()` block at all. `/minio-images/:path*` is a route handler (`src/app/minio-images/[...key]/route.ts`) that requires a session, rejects `..` in the key, and enforces per-user ownership by key prefix (`/` or `scans//`). It reads `MINIO_ENDPOINT` at request time, so changing it takes effect on restart with no rebuild. > This replaced an unauthenticated rewrite that, combined with an anonymous-download bucket policy, exposed every stored image to the public internet. **The bucket must not have an anonymous policy, and MinIO must stay bound to loopback** (`127.0.0.1:9000`, verify with `ss -lntp | grep 9000`). `mc anonymous get local/drink-images` is not a usable check here — the `local` alias on this host has no working credentials and returns `Access Denied` regardless. Check from outside instead: `curl -si https://drinktracker.tenseconddelay.net/minio-images?list-type=2` must **not** return a `200` with an XML key listing. **Never use `next/image` for stored images.** The optimizer fetches them server-side without the session cookie, so every image would 401. All renders are plain ``, and there is deliberately no `images.remotePatterns` config. **`NEXTAUTH_URL` must exactly equal the public origin, with no trailing slash.** Beyond login redirects it now drives the OAuth `issuer` and the MCP `resource` identifier, both of which clients compare byte-for-byte. A mismatch does not error — connectors just fail to complete with an unhelpful "couldn't reach the server". **`prisma db push`, not `migrate deploy`.** Production has no `_prisma_migrations` table, so `migrate deploy` would try to apply the init migration against populated tables and fail. `--accept-data-loss` means removing a field from `schema.prisma` **drops the column and its data** — the deploy script dumps first for this reason. ## How a deploy works Build happens in `/opt/drinktracker/repo` and is assembled **out-of-place** into `releases/`, so the ~3 minute build runs while the current release keeps serving. Only the symlink swap and `systemctl restart` cause downtime, about 3 seconds. Rollback is the same swap in reverse and needs no rebuild. The build deliberately does **not** source the env file, same as the Dockerfile. Nothing in this app needs configuration baked into the bundle: `NEXTAUTH_URL`, `MINIO_ENDPOINT` and the rest are read at request time by dynamic routes, so the build stays identical regardless of which host it runs on. (This originally guarded against a `rewrites()` block that baked a MinIO URL into `routes-manifest.json`; that block is gone, but building without secrets is still the right default.) ## Docker still works, just not here `Dockerfile` and `docker-compose.prod.yml` are maintained for running this app elsewhere (a VPS). `docker compose build && docker compose up -d` works on any normal Docker host. It does **not** work on this LXC, which is why the app runs natively. Two independent problems, both from running a container runtime nested inside an LXC: > An earlier version of this note blamed *unprivileged*-LXC nesting. That is wrong: `/proc/self/uid_map` reads `0 0 4294967295`, an identity mapping, so **this container is privileged**. The two failures below were reproduced and are real; the root cause was never actually established, so don't treat "because unprivileged" as a diagnosis if you revisit this. 1. AppArmor — the container can't load the `docker-default` profile (`apparmor_parser: Access denied`), so any container without `--security-opt apparmor=unconfined` fails. 2. runc can't set `net.ipv4.ip_unprivileged_port_start`, so containers only work with `--network host`. `docker build` is impossible regardless: Docker 29 removed the classic builder and BuildKit spawns a builder container without the unconfined option. Docker is installed but disabled; the old containers and volumes are retained until 2026-09-07 as a rollback path. ## Verifying by hand ```bash ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169 systemctl is-active postgresql@16-main minio drinktracker sudo journalctl -u drinktracker -n 50 sudo journalctl -u drinktracker | grep '\[switchboard\]' # AI routing + cost per call sudo journalctl -u drinktracker | grep '\[mcp\]' # one line per MCP tool call sudo journalctl -u drinktracker | grep '\[oauth\]' # connector grants and token issuance readlink /opt/drinktracker/current # which release is live ``` Each AI call logs one `[switchboard] feature=… model=… cost=… latency_ms=…` line. `FAILOVER` warnings are expected — the gateway's `:batch` model variants currently fail on every request and fall back. `CONTEXT OVERFLOW` is worth investigating. MCP tool calls log `[mcp] user=… tool=… ok=… ms=…`, and the same events are rows in `McpAuditLog` (arguments are deliberately never recorded). `[oauth]` lines cover consent grants and token issuance; a `PKCE mismatch` or `refresh token replay` warning there means a grant was revoked defensively and is worth looking at. Discovery can be checked without credentials: ```bash curl -s https://drinktracker.tenseconddelay.net/.well-known/oauth-protected-resource | jq curl -si https://drinktracker.tenseconddelay.net/api/mcp -X POST \ -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \ -H 'MCP-Protocol-Version: 2026-07-28' -H 'Mcp-Method: tools/list' \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{"_meta":{"io.modelcontextprotocol/protocolVersion":"2026-07-28","io.modelcontextprotocol/clientCapabilities":{}}}}' ``` The second must return **401** with a `WWW-Authenticate: Bearer … resource_metadata="…"` header. A `307` there means `/.well-known` fell out of `PUBLIC_ROUTES` in `src/lib/auth.ts` and discovery is broken. ## Backups `drinktracker-backup.timer` runs nightly at 03:25: `pg_dump` + a tar of `/var/lib/minio/data` + a copy of the env file, into `/var/backups/drinktracker`, retained 14 days (~1 GB). Each run also copies the dump and the MinIO tarball off-site to TrueNAS: | | | |---|---| | Destination | `jpscott84@192.168.2.196:/mnt/Beefcake/backup/drinktracker` (dataset `Beefcake/backup`) | | Transport | `scp` over SSH — **not** NFS, so there is no mount to go stale and silently swallow a night's backup into the LXC's own disk | | Key | `/root/.ssh/drinktracker_backup_ed25519`, a dedicated backup-only keypair | | Config | `Environment=` lines in `/etc/systemd/system/drinktracker-backup.service.d/offsite.conf` | Set as a **drop-in** rather than an edit to the unit, so re-installing the service cannot silently drop the off-site leg and return to local-only backups. A successful run logs `offsite copy ok -> …`. If it ever logs `NOTE: OFFSITE_DEST unset` again, the drop-in has gone missing. A failed copy logs `WARNING: offsite copy FAILED` and **keeps the local copy** — the job still exits 0, so watch the log rather than the exit status. ```bash sudo systemctl start drinktracker-backup.service # run it now sudo journalctl -u drinktracker-backup -n 20 # look for "offsite copy ok" ``` Verify what landed is actually restorable, not merely present: ```bash sudo ssh -i /root/.ssh/drinktracker_backup_ed25519 jpscott84@192.168.2.196 \ 'ls -lh /mnt/Beefcake/backup/drinktracker && gzip -t /mnt/Beefcake/backup/drinktracker/pg-*.sql.gz && echo OK' ``` **Two gaps that remain.** The dumps are **unencrypted** and contain every user's data plus bcrypt password hashes — acceptable writing to a NAS on the same LAN, but encrypt before this ever leaves the network. And TrueNAS is in the same building, so this survives losing the LXC or the Proxmox host, not fire or theft. > If SSH key auth to TrueNAS ever breaks: sshd refuses `authorized_keys` when the user's **home directory** is group- or world-writable, which TrueNAS datasets often are by default. `/mnt/local/jpscott84` must not be `777`. The symptom is indistinguishable from a missing key — the key is offered and silently rejected — so check `journalctl -u ssh | grep 'bad ownership'` on the NAS first. Restore: ```bash sudo systemctl stop drinktracker zcat /var/backups/drinktracker/pg-.sql.gz | sudo -u postgres psql -d drinkman sudo systemctl stop minio sudo tar -C /var/lib/minio -xzf /var/backups/drinktracker/minio-.tgz sudo chown -R minio:minio /var/lib/minio/data sudo systemctl start minio drinktracker ``` ## Notes - The deploy needs unattended sudo on the LXC (`/etc/sudoers.d/drinkadmin-deploy`, currently `NOPASSWD: ALL`). A narrowly-scoped allowlist was considered and rejected: the script calls root for `install`, `tee`, `systemctl`, `journalctl` and `sudo -u postgres pg_dump`, so an allowlist would need updating every time the script changes and would fail in confusing ways when it drifted. Be aware this buys little security either way — `drinkadmin` is in the `sudo` group, so full root is a password away regardless. What it grants is *unattended* root. - `nodejs` is `apt-mark hold`-ed at v20.20.0 so the native runtime can't drift from the Dockerfile's pinned `node:20-alpine`. - Postgres was created with `en_US.UTF-8`; the Ubuntu default of `C`/`SQL_ASCII` would re-sort every drink list and mangle names like `Grüner Veltliner`. Preserve the locale if the cluster is ever rebuilt. - `unattended-upgrades` is not installed, so nothing will surprise-restart these services — but postgres will never auto-patch either. A deliberate open question.