Run natively under systemd; Docker was unworkable in this LXC

Docker in this unprivileged Proxmox LXC was broken two independent ways: AppArmor could not load the docker-default profile, so the daemon could not start any new container, and runc could not set the net.ipv4.ip_unprivileged_port_start sysctl, which is why every service needed network_mode: host. `docker build` was impossible outright since Docker 29 removed the classic builder. The stack only survived because the containers predated the breakage - a reboot would have left the app down, and nothing could be redeployed. Postgres, MinIO and the app now run natively under systemd. Deploys build out-of-place into releases/<sha> and swap a symlink, so the build happens while the old release keeps serving and downtime is the ~3s restart rather than the ~3min build. Rollback is the same swap in reverse with no rebuild. Dockerfile and docker-compose.prod.yml are unchanged and still work on a normal host; the compose app service gains `build: .` so it can come up on a VPS that cannot reach the LAN-only Gitea registry. Documents three traps found during the migration: rewrites() in next.config.mjs is evaluated at build time so MINIO_ENDPOINT changes silently do nothing, prisma migrate deploy would fail because production has no _prisma_migrations table, and the image proxy relies on the bucket's anonymous-download policy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 17:58:34 +00:00
parent f3865c7901
commit 87f951bf94
5 changed files with 229 additions and 152 deletions
--- a/.claude/skills/deploy-drinktracker/SKILL.md
+++ b/.claude/skills/deploy-drinktracker/SKILL.md
@@ -5,65 +5,83 @@ description: Deploy drinktracker to the production LXC and verify it came back h

 # Deploying drinktracker

-Run `./deploy/deploy.sh` from the repo root. It typechecks, pushes to Gitea, rebuilds the
-image on the LXC, restarts the stack and polls until the app answers. Use `--yes` to skip
-the confirmation prompt when the user has already approved the deploy.
+Run `./deploy/deploy.sh` from the repo root (`--yes` to skip the prompt when the user has
+already approved). It typechecks, pushes to Gitea, builds natively on the LXC into
+`releases/<sha>`, swaps the `current` symlink, restarts, and polls until healthy.
+`./deploy/deploy.sh --rollback` repoints at the previous release in ~3 seconds with no rebuild.

-`deploy/README.md` has the full topology table, the one-time sudoers setup, and rollback
-commands. Read it before improvising — the layout is unusual in ways that waste time.
+`deploy/README.md` has the full topology, backup/restore, and manual verification commands.

-## Facts that are not discoverable from the repo
+**The app runs natively under systemd. There is no Docker on this host.**

- **The LXC is `192.168.2.169`.** `drinktracker.tenseconddelay.net` (and `NEXTAUTH_URL`)
-  resolve to the reverse proxy at `192.168.2.172`. SSHing there fails with what looks like
-  a rejected key but is simply the wrong machine.
- **The live stack is `/root/drinktracker`.** The checkouts under `/home/drinkadmin/`
-  (`drinktracker`, `drinktracker-latest`) are stale June copies. Confirm with
-  `docker inspect drinktracker-app-1 --format '{{index .Config.Labels "com.docker.compose.project.working_dir"}}'`.
- **There is no CI.** No Gitea Actions, no GitHub workflows. `git push` updates the repo
-  and nothing else; the image must be built and the stack restarted separately. If someone
-  reports "I pushed but nothing changed", this is why.
+## Facts not discoverable from the repo
+
+- **The LXC is `192.168.2.169`.** `drinktracker.tenseconddelay.net` and `NEXTAUTH_URL` resolve
+  to the reverse proxy at `192.168.2.172`. SSHing there fails in a way that looks like a
+  rejected key but is a different machine entirely.
 - **Access:** `ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169`. Root SSH is
-  disabled. `drinkadmin` needs the sudoers drop-in from `deploy/README.md` for the script
-  to work unattended.
+  disabled. Confirm what's live with `readlink /opt/drinktracker/current`.
+- **Layout:** app `/opt/drinktracker/{repo,releases/<sha>,current}`, service user
+  `drinktracker`, secrets `/etc/drinktracker/drinktracker.env`, units `drinktracker`,
+  `minio`, `postgresql@16-main`.
+- **Database is `drinkman`, user `drinkdb`** — not `drinktracker`/`drinktracker`.
+- **No CI.** `git push` updates the Gitea repo and nothing else. If someone reports "I pushed
+  but nothing changed", this is why.
+
+## Three things that will bite you
+
+**`next.config.mjs` evaluates `rewrites()` at build time.** The `/minio-images/:path*`
+destination is baked into `routes-manifest.json` as `http://localhost:9000/drink-images/:path*`.
+Changing `MINIO_ENDPOINT` in the env file looks like it works and does nothing; images break
+only after the next rebuild. MinIO must stay on `localhost:9000`. For the same reason the
+deploy builds *without* sourcing the env file, exactly as the Dockerfile does — don't "fix" that.
+
+**`prisma db push`, never `migrate deploy`.** Production has no `_prisma_migrations` table, so
+`migrate deploy` would try to apply `20260228220749_init` against 16 populated tables and fail.
+This is the obvious-looking modernization that breaks production.
+
+**`--accept-data-loss` drops columns.** Removing a field from `schema.prisma` deletes the column
+and its data on the next deploy, in both directions — so rolling back past a schema change is
+destructive too. The deploy script dumps to `/var/backups/drinktracker/pre-<sha>.sql.gz` first.
+This is also why `UserPreference.defaultProvider` is still in the schema despite being unused.
+
+Manual dump: `sudo -u postgres pg_dump drinkman > ~/dt-$(date +%F).sql`

 ## Verifying

-An unauthenticated request to port 3000 returns **307** (redirect to `/login`) — that is a
-healthy response, not a failure. The script accepts 200, 302 and 307.
-
-Each AI call emits one line:
+An unauthenticated request to port 3000 returns **307** (redirect to `/login`) — healthy, not a
+failure. The script accepts 200, 302 and 307.

 ```
-sudo docker logs drinktracker-app-1 2>&1 | grep '\[switchboard\]'
+sudo journalctl -u drinktracker | grep '\[switchboard\]'
 ```

-`FAILOVER` lines are expected and harmless: the gateway's `:batch` model variants fail on
-every request and fall back. `CONTEXT OVERFLOW` is worth investigating — it means a request
-was too large for any model's window and may have been truncated.
-
-## Before deploying schema changes
-
-The `migrate` service runs `prisma db push --skip-generate --accept-data-loss` on every
-start. Removing a field from `schema.prisma` therefore **drops the column and its data** on
-the next deploy, in both directions — so a rollback past a schema change is destructive too.
-Dump the database first when the schema changed:
-
-```
-sudo docker exec drinktracker-db-1 pg_dump -U drinktracker drinktracker > ~/dt-$(date +%F).sql
-```
-
-This is also why `UserPreference.defaultProvider` is still in the schema despite being
-unused: deleting it would drop the column and break restoring older backups.
+Each AI call emits one line with the model and cost. `FAILOVER` lines are expected and harmless:
+the gateway's `:batch` model variants fail on every request and fall back. `CONTEXT OVERFLOW`
+means a request exceeded every model's window and may have been truncated — worth investigating.

 ## If the app does not come back

 ```
-sudo docker compose -f /root/drinktracker/docker-compose.prod.yml ps
-sudo docker logs drinktracker-app-1 --tail 100
+systemctl status drinktracker
+sudo journalctl -u drinktracker -n 100
+systemctl is-active postgresql@16-main minio
+readlink /opt/drinktracker/current
 ```

-Most likely causes, in order: the build failed and compose restarted the previous image;
-`.env.production` is missing a variable the new code reads (compare against `.env.example`);
-or the `migrate` service exited non-zero and `app` never started, which `ps` will show as
-`migrate` in a non-zero exited state.
+Most likely, in order: the build failed and the symlink still points at the old release (so the
+app is fine, the deploy just didn't take); `/etc/drinktracker/drinktracker.env` is missing a
+variable the new code reads (compare against `.env.example`); or postgres/minio didn't start,
+which `is-active` shows immediately.
+
+Note the systemd unit deliberately omits `MemoryDenyWriteExecute` (it breaks the V8 JIT) and
+includes `AF_NETLINK` in `RestrictAddressFamilies` (Next needs it at startup). If you harden
+that unit further and the app stops booting, those are the two to suspect.
+
+## Docker
+
+`Dockerfile` and `docker-compose.prod.yml` are kept for running this app on a VPS and still work
+on a normal Docker host. They cannot work on this LXC: unprivileged-LXC nesting breaks AppArmor
+profile loading and the `net.ipv4.ip_unprivileged_port_start` sysctl, and `docker build` is
+impossible since Docker 29 removed the classic builder. Docker is installed but disabled; old
+containers and volumes are retained until 2026-09-07 as a rollback path.
--- a/deploy/README.md
+++ b/deploy/README.md
@@ -1,71 +1,91 @@
 # Deploying drinktracker

 ```bash
-./deploy/deploy.sh          # typecheck, push, build on the LXC, restart, verify
-./deploy/deploy.sh --yes    # skip the confirmation prompt
-./deploy/deploy.sh --push   # also publish the image to the Gitea registry
+./deploy/deploy.sh              # typecheck, push, build, swap, restart, verify
+./deploy/deploy.sh --yes        # skip the confirmation prompt
+./deploy/deploy.sh --rollback   # repoint at the previous release (~3s, no rebuild)
 ```

-## Where things actually live
+The app runs **natively under systemd — there is no Docker on this host.**
+
+## Where things live

 | | |
 |---|---|
-| Production LXC | `192.168.2.169` (hostname `drinktracker`) |
-| Deploy user | `drinkadmin`, key `~/.ssh/drinktracker_ed25519` |
-| Live compose dir | `/root/drinktracker` |
-| Image tag | `192.168.2.140:3000/jpscott84/drinktracker:latest` |
-| App port | 3000 (host network), unauthenticated requests return 307 → `/login` |
+| Production LXC | `192.168.2.169` (hostname `drinktracker`, Proxmox vmid 100) |
+| Deploy access | `ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169` |
+| App | `/opt/drinktracker/{repo,releases/<sha>,current}`, service user `drinktracker` |
+| Secrets | `/etc/drinktracker/drinktracker.env` (`root:drinktracker` `0640`) |
+| Services | `drinktracker`, `minio`, `postgresql@16-main` |
+| Database | `drinkman`, user `drinkdb`, `127.0.0.1:5432` |
+| Object storage | MinIO at `127.0.0.1:9000`, bucket `drink-images` |
+| App port | 3000 on `0.0.0.0`; unauthenticated requests return 307 → `/login` |
+| Backups | `/var/backups/drinktracker`, nightly 03:25, 14-day retention |

-Two traps worth knowing:
+## Traps

-**`drinktracker.tenseconddelay.net` is the reverse proxy (192.168.2.172), not the container.** `NEXTAUTH_URL` points at the public name, so resolving it and SSHing there fails confusingly — the key looks rejected because it's a different machine.
+**`drinktracker.tenseconddelay.net` resolves to the reverse proxy (192.168.2.172), not the container.** `NEXTAUTH_URL` points at the public name, so resolving it and SSHing there fails in a way that looks like a rejected key but is simply a different machine.

-**The two checkouts under `/home/drinkadmin/` are stale.** Only `/root/drinktracker` is live; that's what the running container's compose labels point to.
+**`next.config.mjs` evaluates `rewrites()` at BUILD time.** The `/minio-images/:path*` destination is baked into `routes-manifest.json` as `http://localhost:9000/drink-images/:path*`. Changing `MINIO_ENDPOINT` in the env file appears to work and does nothing — images break only after the *next* rebuild, so the change and the breakage are separated in time. MinIO must stay on `localhost:9000` serving `drink-images`.

-## How it works
+**The image proxy is an unauthenticated GET.** Next rewrites don't sign requests, so the bucket's anonymous-download policy is load-bearing. Verify with `mc anonymous get local/drink-images` → must say `download`.

-The image is built **on the LXC** and tagged with the registry name that `docker-compose.prod.yml` already references. Compose finds it locally and never pulls, so no registry credentials are needed on either machine. `--push` publishes it as well, which is only useful if something else pulls that tag.
+**`prisma db push`, not `migrate deploy`.** Production has no `_prisma_migrations` table, so `migrate deploy` would try to apply the init migration against populated tables and fail. `--accept-data-loss` means removing a field from `schema.prisma` **drops the column and its data** — the deploy script dumps first for this reason.

-There is no CI, so **`git push` on its own deploys nothing** — it only updates the Gitea repo. The build and restart have to happen separately, which is what this script is for.
+## How a deploy works

-## One-time setup
+Build happens in `/opt/drinktracker/repo` and is assembled **out-of-place** into `releases/<sha>`, so the ~3 minute build runs while the current release keeps serving. Only the symlink swap and `systemctl restart` cause downtime, about 3 seconds. Rollback is the same swap in reverse and needs no rebuild.

-The script needs passwordless sudo on the LXC, because the deployment lives under `/root`. On the container as root:
+The build deliberately does **not** source the env file — same as the Dockerfile — because of the build-time rewrite baking described above.

-```bash
-echo 'drinkadmin ALL=(root) NOPASSWD: ALL' > /etc/sudoers.d/drinkadmin-deploy
-chmod 440 /etc/sudoers.d/drinkadmin-deploy
-visudo -c
-```
+## Docker still works, just not here

-This grants less than it appears to: `drinkadmin` is already in the `docker` group, and anyone who can talk to the Docker socket can mount the host filesystem into a privileged container, so they are root-equivalent already. If you'd rather not, drop the sudoers file and run the deploy steps by hand from a root shell.
+`Dockerfile` and `docker-compose.prod.yml` are maintained for running this app elsewhere (a VPS). `docker compose build && docker compose up -d` works on any normal Docker host.

-**Alternative worth considering:** move the deployment out of `/root` into a `drinkadmin`-owned directory and skip sudo entirely. Keep the directory basename `drinktracker` so the compose project name — and therefore the existing container and volume names — stay the same. `.env.production` must be copied across intact or Postgres will reject the app's credentials, since the volume was initialised with the current password.
+It does **not** work on this LXC, which is why the app runs natively. Two independent problems, both from unprivileged-LXC nesting:
+
+1. AppArmor — the container can't load the `docker-default` profile (`apparmor_parser: Access denied`), so any container without `--security-opt apparmor=unconfined` fails.
+2. runc can't set `net.ipv4.ip_unprivileged_port_start`, so containers only work with `--network host`.
+
+`docker build` is impossible regardless: Docker 29 removed the classic builder and BuildKit spawns a builder container without the unconfined option. Docker is installed but disabled; the old containers and volumes are retained until 2026-09-07 as a rollback path.

 ## Verifying by hand

 ```bash
 ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169
-sudo docker compose -f /root/drinktracker/docker-compose.prod.yml ps
-sudo docker logs drinktracker-app-1 --tail 50
-sudo docker logs drinktracker-app-1 2>&1 | grep '\[switchboard\]'   # AI routing + cost per call
+systemctl is-active postgresql@16-main minio drinktracker
+sudo journalctl -u drinktracker -n 50
+sudo journalctl -u drinktracker | grep '\[switchboard\]'    # AI routing + cost per call
+readlink /opt/drinktracker/current                          # which release is live
 ```

-Each AI call logs one `[switchboard] feature=… model=… cost=… latency_ms=…` line. `FAILOVER` warnings are expected — the gateway's `:batch` model variants currently fail on every request and fall back.
+Each AI call logs one `[switchboard] feature=… model=… cost=… latency_ms=…` line. `FAILOVER` warnings are expected — the gateway's `:batch` model variants currently fail on every request and fall back. `CONTEXT OVERFLOW` is worth investigating.

-## Rollback
+## Backups
+
+`drinktracker-backup.timer` runs nightly at 03:25: `pg_dump` + a tar of `/var/lib/minio/data` + a copy of the env file, into `/var/backups/drinktracker`, retained 14 days (~1 GB).
+
+**TODO — the chain has no off-site leg.** `/usr/local/bin/drinktracker-backup` supports `OFFSITE_DEST` (an scp target) and `OFFSITE_KEY`; set them via `Environment=` in `drinktracker-backup.service`. Until then everything lives on one LXC on one Proxmox host, and the nightly job prints a warning saying so.
+
+Restore:

 ```bash
-git revert <bad-sha> && ./deploy/deploy.sh
+sudo systemctl stop drinktracker
+zcat /var/backups/drinktracker/pg-<stamp>.sql.gz | sudo -u postgres psql -d drinkman
+sudo systemctl stop minio
+sudo tar -C /var/lib/minio -xzf /var/backups/drinktracker/minio-<stamp>.tgz
+sudo chown -R minio:minio /var/lib/minio/data
+sudo systemctl start minio drinktracker
 ```

-Or directly on the LXC, without touching git history:
+## Notes

-```bash
-sudo git -C /root/drinktracker reset --hard <previous-sha>
-cd /root/drinktracker
-sudo docker build -t 192.168.2.140:3000/jpscott84/drinktracker:latest .
-sudo docker compose -f docker-compose.prod.yml up -d --no-build
-```
-
-Note the `migrate` service runs `prisma db push --accept-data-loss` on every start, so a rollback that removes a column will drop it and its data. Take a database dump first if the schema changed.
+- The deploy needs unattended sudo on the LXC (`/etc/sudoers.d/drinkadmin-deploy`, currently
+  `NOPASSWD: ALL`). A narrowly-scoped allowlist was considered and rejected: the script calls
+  root for `install`, `tee`, `systemctl`, `journalctl` and `sudo -u postgres pg_dump`, so an
+  allowlist would need updating every time the script changes and would fail in confusing ways
+  when it drifted. Be aware this buys little security either way — `drinkadmin` is in the `sudo`
+  group, so full root is a password away regardless. What it grants is *unattended* root.
+- `nodejs` is `apt-mark hold`-ed at v20.20.0 so the native runtime can't drift from the Dockerfile's pinned `node:20-alpine`.
+- Postgres was created with `en_US.UTF-8`; the Ubuntu default of `C`/`SQL_ASCII` would re-sort every drink list and mangle names like `Grüner Veltliner`. Preserve the locale if the cluster is ever rebuilt.
+- `unattended-upgrades` is not installed, so nothing will surprise-restart these services — but postgres will never auto-patch either. A deliberate open question.
--- a/deploy/deploy.sh
+++ b/deploy/deploy.sh
@@ -2,28 +2,27 @@
 #
 # Deploy drinktracker to the production LXC.
 #
-# The image is built ON the LXC and tagged with the registry name that
-# docker-compose.prod.yml expects. Compose then finds it locally and never pulls, so
-# this needs no registry credentials on either machine. Pass --push to additionally
-# publish the image to the Gitea registry (requires `docker login` on the LXC).
+# The app runs natively under systemd - no Docker. Each deploy builds into
+# releases/<sha> out-of-place and swaps a symlink, so the ~3 minute build happens
+# while the old release keeps serving and downtime is only the restart.
 #
-# Usage:  ./deploy/deploy.sh [--push] [--no-build] [--yes]
+# Usage:  ./deploy/deploy.sh [--yes] [--no-build] [--rollback]
 #
 set -euo pipefail

 HOST="${DT_HOST:-drinkadmin@192.168.2.169}"
 SSH_KEY="${DT_SSH_KEY:-$HOME/.ssh/drinktracker_ed25519}"
-REMOTE_DIR="${DT_REMOTE_DIR:-/root/drinktracker}"
-IMAGE="${DT_IMAGE:-192.168.2.140:3000/jpscott84/drinktracker:latest}"
-COMPOSE_FILE="docker-compose.prod.yml"
+APP_DIR="${DT_APP_DIR:-/opt/drinktracker}"
+SVC_USER="${DT_SVC_USER:-drinktracker}"
+KEEP_RELEASES=5
 HEALTH_URL="http://localhost:3000/"

-PUSH=0; BUILD=1; ASSUME_YES=0
+ASSUME_YES=0; BUILD=1; ROLLBACK=0
 for arg in "$@"; do
  case "$arg" in
-    --push) PUSH=1 ;;
-    --no-build) BUILD=0 ;;
    --yes|-y) ASSUME_YES=1 ;;
+    --no-build) BUILD=0 ;;
+    --rollback) ROLLBACK=1 ;;
    *) echo "unknown option: $arg" >&2; exit 2 ;;
  esac
 done
@@ -32,87 +31,118 @@ SSH=(ssh -o BatchMode=yes -o ConnectTimeout=10 -i "$SSH_KEY" "$HOST")
 say() { printf '\n\033[1;36m==> %s\033[0m\n' "$*"; }
 die() { printf '\n\033[1;31mFAILED: %s\033[0m\n' "$*" >&2; exit 1; }

+health_poll() {
+  for i in $(seq 1 30); do
+    code=$("${SSH[@]}" "curl -s -o /dev/null -w '%{http_code}' -m 5 $HEALTH_URL" || echo 000)
+    # 307 is the unauthenticated redirect to /login - healthy.
+    case "$code" in 200|302|307) echo "  healthy (HTTP $code) after $((i*2))s"; return 0 ;; esac
+    sleep 2
+  done
+  return 1
+}
+
+# ─── Rollback: repoint the symlink, no rebuild ───────────────────────
+if [ "$ROLLBACK" -eq 1 ]; then
+  say "Available releases (newest last)"
+  "${SSH[@]}" "ls -1tr $APP_DIR/releases"
+  CURRENT=$("${SSH[@]}" "readlink $APP_DIR/current | xargs basename")
+  PREV=$("${SSH[@]}" "ls -1t $APP_DIR/releases | grep -v '^$CURRENT\$' | head -1")
+  [ -n "$PREV" ] || die "no other release to roll back to"
+  echo "  current=$CURRENT  ->  rolling back to $PREV"
+  if [ "$ASSUME_YES" -ne 1 ]; then
+    read -r -p "  proceed? [y/N] " r; [[ "$r" =~ ^[Yy]$ ]] || { echo aborted; exit 1; }
+  fi
+  "${SSH[@]}" "sudo -n -u $SVC_USER bash -c 'ln -sfn $APP_DIR/releases/$PREV $APP_DIR/current.new && mv -Tf $APP_DIR/current.new $APP_DIR/current'"
+  "${SSH[@]}" "sudo -n systemctl restart drinktracker"
+  health_poll || die "app unhealthy after rollback - journalctl -u drinktracker"
+  say "Rolled back to $PREV"
+  exit 0
+fi
+
 # ─── Preflight (local) ───────────────────────────────────────────────
 say "Preflight"
 cd "$(dirname "$0")/.."
-
 [ -n "$(git status --porcelain)" ] && die "working tree is dirty - commit or stash first"
 BRANCH=$(git rev-parse --abbrev-ref HEAD)
 [ "$BRANCH" = "main" ] || die "on branch '$BRANCH', expected main"
-
 npx tsc --noEmit || die "typecheck failed"
 echo "  typecheck ok, tree clean, on main"

-# Push first so the server pulls exactly what was verified here.
 git -c credential.helper=store push origin main
-LOCAL_SHA=$(git rev-parse HEAD)
-echo "  pushed $(git rev-parse --short HEAD)"
+SHA=$(git rev-parse HEAD); SHORT=${SHA:0:7}
+echo "  pushed $SHORT"

 # ─── Preflight (remote) ──────────────────────────────────────────────
 say "Checking $HOST"
 "${SSH[@]}" true || die "cannot reach $HOST with key $SSH_KEY"
-"${SSH[@]}" 'sudo -n true' 2>/dev/null \
-  || die "passwordless sudo required on the LXC (needed for $REMOTE_DIR). See deploy/README.md"
-
-# A dirty server checkout means someone edited production by hand; clobbering that
-# silently would destroy the only copy of the change.
-if ! "${SSH[@]}" "sudo -n git -C $REMOTE_DIR diff --quiet && sudo -n git -C $REMOTE_DIR diff --cached --quiet"; then
-  die "$REMOTE_DIR has uncommitted changes - inspect before deploying"
-fi
+for unit in postgresql@16-main minio drinktracker; do
+  st=$("${SSH[@]}" "systemctl is-active $unit" || true)
+  echo "  $unit: $st"
+  [ "$st" = "active" ] || die "$unit is not active - fix before deploying"
+done

 if [ "$ASSUME_YES" -ne 1 ]; then
-  REMOTE_SHA=$("${SSH[@]}" "sudo -n git -C $REMOTE_DIR rev-parse --short HEAD")
-  echo "  remote is at $REMOTE_SHA, deploying ${LOCAL_SHA:0:7}"
-  read -r -p "  proceed? [y/N] " reply
-  [[ "$reply" =~ ^[Yy]$ ]] || { echo "aborted"; exit 1; }
+  CURRENT=$("${SSH[@]}" "readlink $APP_DIR/current | xargs basename")
+  echo "  current release $CURRENT, deploying $SHORT"
+  read -r -p "  proceed? [y/N] " r; [[ "$r" =~ ^[Yy]$ ]] || { echo aborted; exit 1; }
 fi

-# ─── Deploy ──────────────────────────────────────────────────────────
-say "Syncing $REMOTE_DIR to $LOCAL_SHA"
-"${SSH[@]}" "sudo -n git -C $REMOTE_DIR fetch origin main --quiet && sudo -n git -C $REMOTE_DIR reset --hard $LOCAL_SHA --quiet && sudo -n git -C $REMOTE_DIR log --oneline -1"
-
-# SWITCHBOARD_BASE_URL arrived with the gateway migration and will be missing from
-# any .env.production written before it. Compose supplies a default, but making it
-# explicit keeps the file honest about what the app reads.
-say "Checking .env.production for new variables"
-"${SSH[@]}" "sudo -n grep -q '^SWITCHBOARD_BASE_URL=' $REMOTE_DIR/.env.production" \
-  && echo "  SWITCHBOARD_BASE_URL present" \
-  || echo "  NOTE: SWITCHBOARD_BASE_URL absent; compose default (http://192.168.2.11:8787/v1) applies"
+# ─── Build (out-of-place; the running release keeps serving) ─────────
+say "Syncing and building on $HOST"
+"${SSH[@]}" "sudo -n -u $SVC_USER git -C $APP_DIR/repo fetch --quiet origin main && sudo -n -u $SVC_USER git -C $APP_DIR/repo reset --hard $SHA --quiet && sudo -n -u $SVC_USER git -C $APP_DIR/repo log --oneline -1"

 if [ "$BUILD" -eq 1 ]; then
-  say "Building image on the LXC (this takes a few minutes)"
-  # Pass the build context as a path rather than cd-ing in: the shell runs as the
-  # deploy user, which cannot enter /root even though sudo can build there.
-  "${SSH[@]}" "sudo -n docker build -t $IMAGE $REMOTE_DIR" || die "image build failed"
+  # Built WITHOUT the env file, deliberately - same as the Dockerfile. next.config.mjs
+  # evaluates rewrites() at build time, so sourcing the env here would bake a different
+  # /minio-images destination into routes-manifest.json.
+  "${SSH[@]}" "sudo -n -u $SVC_USER bash -lc '
+    set -e
+    cd $APP_DIR/repo
+    export npm_config_cache=$APP_DIR/.npm NODE_OPTIONS=--max-old-space-size=3072
+    npm ci --prefer-offline --no-audit --fund=false
+    npx prisma generate
+    npm run build
+  '" || die "build failed"
 fi

-if [ "$PUSH" -eq 1 ]; then
-  say "Pushing image to registry"
-  "${SSH[@]}" "sudo -n docker push $IMAGE" || die "registry push failed (is docker login done on the LXC?)"
-fi
+say "Assembling release $SHORT"
+# Mirrors Dockerfile:24-30 - standalone bundle plus static and public copied in.
+"${SSH[@]}" "sudo -n -u $SVC_USER bash -lc '
+  set -e
+  REL=$APP_DIR/releases/$SHORT
+  rm -rf \$REL && mkdir -p \$REL/.next/cache
+  cp -a $APP_DIR/repo/.next/standalone/. \$REL/
+  cp -a $APP_DIR/repo/.next/static \$REL/.next/static
+  cp -a $APP_DIR/repo/public \$REL/public
+  du -sh \$REL
+'" || die "release assembly failed"

-say "Restarting stack"
-# --no-build: compose must use the image we just built, not rebuild or pull it.
-# Compose takes the project directory from the compose file's location, which keeps
-# the project name "drinktracker" and so reuses the existing containers and volumes.
-"${SSH[@]}" "sudo -n docker compose -f $REMOTE_DIR/$COMPOSE_FILE up -d --no-build"
+# ─── Schema ──────────────────────────────────────────────────────────
+say "Backing up database before schema push"
+"${SSH[@]}" "sudo -n install -d -m 0750 /var/backups/drinktracker && sudo -n -u postgres pg_dump drinkman | gzip -9 | sudo -n tee /var/backups/drinktracker/pre-$SHORT.sql.gz >/dev/null && sudo -n ls -lh /var/backups/drinktracker/pre-$SHORT.sql.gz"
+
+say "Applying schema"
+# db push, NOT migrate deploy: production has no _prisma_migrations table, so
+# migrate deploy would try to apply the init migration against populated tables.
+# --accept-data-loss means removing a field from schema.prisma DROPS the column.
+"${SSH[@]}" "sudo -n -u $SVC_USER bash -lc '
+  set -a; . /etc/drinktracker/drinktracker.env; set +a
+  cd $APP_DIR/repo && npx prisma db push --skip-generate --accept-data-loss
+'" || die "prisma db push failed"
+
+# ─── Swap and restart ────────────────────────────────────────────────
+say "Swapping to $SHORT and restarting"
+"${SSH[@]}" "sudo -n -u $SVC_USER bash -c 'ln -sfn $APP_DIR/releases/$SHORT $APP_DIR/current.new && mv -Tf $APP_DIR/current.new $APP_DIR/current'"
+"${SSH[@]}" "sudo -n systemctl restart drinktracker"
+health_poll || die "app unhealthy - check: ${SSH[*]} 'journalctl -u drinktracker -n 50'"

 # ─── Verify ──────────────────────────────────────────────────────────
 say "Verifying"
-"${SSH[@]}" "sudo -n docker compose -f $REMOTE_DIR/$COMPOSE_FILE ps --format 'table {{.Name}}\t{{.Status}}'" || true
+"${SSH[@]}" "curl -s -o /dev/null -w '  public=%{http_code}\n' -m 15 https://drinktracker.tenseconddelay.net/ || true"
+"${SSH[@]}" "sudo -n journalctl -u drinktracker --since '2 min ago' | grep '\[switchboard\]' | tail -3 || echo '  (no AI calls yet)'"

-ok=0
-for i in $(seq 1 30); do
-  code=$("${SSH[@]}" "curl -s -o /dev/null -w '%{http_code}' -m 5 $HEALTH_URL" || echo 000)
-  # 307 is the unauthenticated redirect to /login - a healthy response here.
-  case "$code" in 200|307|302) echo "  app responding (HTTP $code) after ${i}0s"; ok=1; break ;; esac
-  sleep 10
-done
-[ "$ok" -eq 1 ] || die "app did not become healthy - check: ${SSH[*]} 'sudo docker logs drinktracker-app-1 --tail 50'"
+say "Pruning old releases (keeping $KEEP_RELEASES)"
+"${SSH[@]}" "sudo -n -u $SVC_USER bash -c 'cd $APP_DIR/releases && ls -1t | tail -n +$((KEEP_RELEASES+1)) | xargs -r rm -rf; ls -1t'"

-say "Recent gateway activity"
-"${SSH[@]}" "sudo -n docker logs drinktracker-app-1 --tail 200 2>&1 | grep '\[switchboard\]' | tail -5 || echo '  (no AI calls yet)'"
-
-say "Deployed ${LOCAL_SHA:0:7}"
-echo "To roll back:  ./deploy/deploy.sh after 'git revert', or on the LXC:"
-echo "  sudo git -C $REMOTE_DIR reset --hard <previous-sha> && cd $REMOTE_DIR && sudo docker build -t $IMAGE . && sudo docker compose -f $COMPOSE_FILE up -d --no-build"
+say "Deployed $SHORT"
+echo "Roll back with:  ./deploy/deploy.sh --rollback"
--- a/docker-compose.prod.yml
+++ b/docker-compose.prod.yml
@@ -75,7 +75,10 @@ services:
    restart: "no"

  app:
+    # `image:` names the tag this produces; `build:` lets it come up on a host that
+    # can't reach the LAN-only Gitea registry (e.g. a VPS) via `docker compose build`.
    image: 192.168.2.140:3000/jpscott84/drinktracker:latest
+    build: .
    restart: unless-stopped
    network_mode: host
    security_opt:
--- a/install.sh
+++ b/install.sh
@@ -2,7 +2,13 @@
 set -euo pipefail

 # ─────────────────────────────────────────────────────────────
-# DrinkTracker — Production Install Script
+# DrinkTracker — Production Install Script (Docker)
+#
+# For a fresh Docker-based install on a normal host or VPS.
+#
+# NOTE: the production LXC (192.168.2.169) no longer uses this — it runs the app
+# natively under systemd because Docker can't work in an unprivileged LXC. To
+# deploy there, use ./deploy/deploy.sh. See deploy/README.md.
 # ─────────────────────────────────────────────────────────────

 # Colors