diff --git a/.claude/skills/deploy-drinktracker/SKILL.md b/.claude/skills/deploy-drinktracker/SKILL.md index f529690..d3a89ed 100644 --- a/.claude/skills/deploy-drinktracker/SKILL.md +++ b/.claude/skills/deploy-drinktracker/SKILL.md @@ -5,65 +5,83 @@ description: Deploy drinktracker to the production LXC and verify it came back h # Deploying drinktracker -Run `./deploy/deploy.sh` from the repo root. It typechecks, pushes to Gitea, rebuilds the -image on the LXC, restarts the stack and polls until the app answers. Use `--yes` to skip -the confirmation prompt when the user has already approved the deploy. +Run `./deploy/deploy.sh` from the repo root (`--yes` to skip the prompt when the user has +already approved). It typechecks, pushes to Gitea, builds natively on the LXC into +`releases/`, swaps the `current` symlink, restarts, and polls until healthy. +`./deploy/deploy.sh --rollback` repoints at the previous release in ~3 seconds with no rebuild. -`deploy/README.md` has the full topology table, the one-time sudoers setup, and rollback -commands. Read it before improvising — the layout is unusual in ways that waste time. +`deploy/README.md` has the full topology, backup/restore, and manual verification commands. -## Facts that are not discoverable from the repo +**The app runs natively under systemd. There is no Docker on this host.** -- **The LXC is `192.168.2.169`.** `drinktracker.tenseconddelay.net` (and `NEXTAUTH_URL`) - resolve to the reverse proxy at `192.168.2.172`. SSHing there fails with what looks like - a rejected key but is simply the wrong machine. -- **The live stack is `/root/drinktracker`.** The checkouts under `/home/drinkadmin/` - (`drinktracker`, `drinktracker-latest`) are stale June copies. Confirm with - `docker inspect drinktracker-app-1 --format '{{index .Config.Labels "com.docker.compose.project.working_dir"}}'`. -- **There is no CI.** No Gitea Actions, no GitHub workflows. `git push` updates the repo - and nothing else; the image must be built and the stack restarted separately. If someone - reports "I pushed but nothing changed", this is why. +## Facts not discoverable from the repo + +- **The LXC is `192.168.2.169`.** `drinktracker.tenseconddelay.net` and `NEXTAUTH_URL` resolve + to the reverse proxy at `192.168.2.172`. SSHing there fails in a way that looks like a + rejected key but is a different machine entirely. - **Access:** `ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169`. Root SSH is - disabled. `drinkadmin` needs the sudoers drop-in from `deploy/README.md` for the script - to work unattended. + disabled. Confirm what's live with `readlink /opt/drinktracker/current`. +- **Layout:** app `/opt/drinktracker/{repo,releases/,current}`, service user + `drinktracker`, secrets `/etc/drinktracker/drinktracker.env`, units `drinktracker`, + `minio`, `postgresql@16-main`. +- **Database is `drinkman`, user `drinkdb`** — not `drinktracker`/`drinktracker`. +- **No CI.** `git push` updates the Gitea repo and nothing else. If someone reports "I pushed + but nothing changed", this is why. + +## Three things that will bite you + +**`next.config.mjs` evaluates `rewrites()` at build time.** The `/minio-images/:path*` +destination is baked into `routes-manifest.json` as `http://localhost:9000/drink-images/:path*`. +Changing `MINIO_ENDPOINT` in the env file looks like it works and does nothing; images break +only after the next rebuild. MinIO must stay on `localhost:9000`. For the same reason the +deploy builds *without* sourcing the env file, exactly as the Dockerfile does — don't "fix" that. + +**`prisma db push`, never `migrate deploy`.** Production has no `_prisma_migrations` table, so +`migrate deploy` would try to apply `20260228220749_init` against 16 populated tables and fail. +This is the obvious-looking modernization that breaks production. + +**`--accept-data-loss` drops columns.** Removing a field from `schema.prisma` deletes the column +and its data on the next deploy, in both directions — so rolling back past a schema change is +destructive too. The deploy script dumps to `/var/backups/drinktracker/pre-.sql.gz` first. +This is also why `UserPreference.defaultProvider` is still in the schema despite being unused. + +Manual dump: `sudo -u postgres pg_dump drinkman > ~/dt-$(date +%F).sql` ## Verifying -An unauthenticated request to port 3000 returns **307** (redirect to `/login`) — that is a -healthy response, not a failure. The script accepts 200, 302 and 307. - -Each AI call emits one line: +An unauthenticated request to port 3000 returns **307** (redirect to `/login`) — healthy, not a +failure. The script accepts 200, 302 and 307. ``` -sudo docker logs drinktracker-app-1 2>&1 | grep '\[switchboard\]' +sudo journalctl -u drinktracker | grep '\[switchboard\]' ``` -`FAILOVER` lines are expected and harmless: the gateway's `:batch` model variants fail on -every request and fall back. `CONTEXT OVERFLOW` is worth investigating — it means a request -was too large for any model's window and may have been truncated. - -## Before deploying schema changes - -The `migrate` service runs `prisma db push --skip-generate --accept-data-loss` on every -start. Removing a field from `schema.prisma` therefore **drops the column and its data** on -the next deploy, in both directions — so a rollback past a schema change is destructive too. -Dump the database first when the schema changed: - -``` -sudo docker exec drinktracker-db-1 pg_dump -U drinktracker drinktracker > ~/dt-$(date +%F).sql -``` - -This is also why `UserPreference.defaultProvider` is still in the schema despite being -unused: deleting it would drop the column and break restoring older backups. +Each AI call emits one line with the model and cost. `FAILOVER` lines are expected and harmless: +the gateway's `:batch` model variants fail on every request and fall back. `CONTEXT OVERFLOW` +means a request exceeded every model's window and may have been truncated — worth investigating. ## If the app does not come back ``` -sudo docker compose -f /root/drinktracker/docker-compose.prod.yml ps -sudo docker logs drinktracker-app-1 --tail 100 +systemctl status drinktracker +sudo journalctl -u drinktracker -n 100 +systemctl is-active postgresql@16-main minio +readlink /opt/drinktracker/current ``` -Most likely causes, in order: the build failed and compose restarted the previous image; -`.env.production` is missing a variable the new code reads (compare against `.env.example`); -or the `migrate` service exited non-zero and `app` never started, which `ps` will show as -`migrate` in a non-zero exited state. +Most likely, in order: the build failed and the symlink still points at the old release (so the +app is fine, the deploy just didn't take); `/etc/drinktracker/drinktracker.env` is missing a +variable the new code reads (compare against `.env.example`); or postgres/minio didn't start, +which `is-active` shows immediately. + +Note the systemd unit deliberately omits `MemoryDenyWriteExecute` (it breaks the V8 JIT) and +includes `AF_NETLINK` in `RestrictAddressFamilies` (Next needs it at startup). If you harden +that unit further and the app stops booting, those are the two to suspect. + +## Docker + +`Dockerfile` and `docker-compose.prod.yml` are kept for running this app on a VPS and still work +on a normal Docker host. They cannot work on this LXC: unprivileged-LXC nesting breaks AppArmor +profile loading and the `net.ipv4.ip_unprivileged_port_start` sysctl, and `docker build` is +impossible since Docker 29 removed the classic builder. Docker is installed but disabled; old +containers and volumes are retained until 2026-09-07 as a rollback path. diff --git a/deploy/README.md b/deploy/README.md index deb1345..f1aa844 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,71 +1,91 @@ # Deploying drinktracker ```bash -./deploy/deploy.sh # typecheck, push, build on the LXC, restart, verify -./deploy/deploy.sh --yes # skip the confirmation prompt -./deploy/deploy.sh --push # also publish the image to the Gitea registry +./deploy/deploy.sh # typecheck, push, build, swap, restart, verify +./deploy/deploy.sh --yes # skip the confirmation prompt +./deploy/deploy.sh --rollback # repoint at the previous release (~3s, no rebuild) ``` -## Where things actually live +The app runs **natively under systemd — there is no Docker on this host.** + +## Where things live | | | |---|---| -| Production LXC | `192.168.2.169` (hostname `drinktracker`) | -| Deploy user | `drinkadmin`, key `~/.ssh/drinktracker_ed25519` | -| Live compose dir | `/root/drinktracker` | -| Image tag | `192.168.2.140:3000/jpscott84/drinktracker:latest` | -| App port | 3000 (host network), unauthenticated requests return 307 → `/login` | +| Production LXC | `192.168.2.169` (hostname `drinktracker`, Proxmox vmid 100) | +| Deploy access | `ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169` | +| App | `/opt/drinktracker/{repo,releases/,current}`, service user `drinktracker` | +| Secrets | `/etc/drinktracker/drinktracker.env` (`root:drinktracker` `0640`) | +| Services | `drinktracker`, `minio`, `postgresql@16-main` | +| Database | `drinkman`, user `drinkdb`, `127.0.0.1:5432` | +| Object storage | MinIO at `127.0.0.1:9000`, bucket `drink-images` | +| App port | 3000 on `0.0.0.0`; unauthenticated requests return 307 → `/login` | +| Backups | `/var/backups/drinktracker`, nightly 03:25, 14-day retention | -Two traps worth knowing: +## Traps -**`drinktracker.tenseconddelay.net` is the reverse proxy (192.168.2.172), not the container.** `NEXTAUTH_URL` points at the public name, so resolving it and SSHing there fails confusingly — the key looks rejected because it's a different machine. +**`drinktracker.tenseconddelay.net` resolves to the reverse proxy (192.168.2.172), not the container.** `NEXTAUTH_URL` points at the public name, so resolving it and SSHing there fails in a way that looks like a rejected key but is simply a different machine. -**The two checkouts under `/home/drinkadmin/` are stale.** Only `/root/drinktracker` is live; that's what the running container's compose labels point to. +**`next.config.mjs` evaluates `rewrites()` at BUILD time.** The `/minio-images/:path*` destination is baked into `routes-manifest.json` as `http://localhost:9000/drink-images/:path*`. Changing `MINIO_ENDPOINT` in the env file appears to work and does nothing — images break only after the *next* rebuild, so the change and the breakage are separated in time. MinIO must stay on `localhost:9000` serving `drink-images`. -## How it works +**The image proxy is an unauthenticated GET.** Next rewrites don't sign requests, so the bucket's anonymous-download policy is load-bearing. Verify with `mc anonymous get local/drink-images` → must say `download`. -The image is built **on the LXC** and tagged with the registry name that `docker-compose.prod.yml` already references. Compose finds it locally and never pulls, so no registry credentials are needed on either machine. `--push` publishes it as well, which is only useful if something else pulls that tag. +**`prisma db push`, not `migrate deploy`.** Production has no `_prisma_migrations` table, so `migrate deploy` would try to apply the init migration against populated tables and fail. `--accept-data-loss` means removing a field from `schema.prisma` **drops the column and its data** — the deploy script dumps first for this reason. -There is no CI, so **`git push` on its own deploys nothing** — it only updates the Gitea repo. The build and restart have to happen separately, which is what this script is for. +## How a deploy works -## One-time setup +Build happens in `/opt/drinktracker/repo` and is assembled **out-of-place** into `releases/`, so the ~3 minute build runs while the current release keeps serving. Only the symlink swap and `systemctl restart` cause downtime, about 3 seconds. Rollback is the same swap in reverse and needs no rebuild. -The script needs passwordless sudo on the LXC, because the deployment lives under `/root`. On the container as root: +The build deliberately does **not** source the env file — same as the Dockerfile — because of the build-time rewrite baking described above. -```bash -echo 'drinkadmin ALL=(root) NOPASSWD: ALL' > /etc/sudoers.d/drinkadmin-deploy -chmod 440 /etc/sudoers.d/drinkadmin-deploy -visudo -c -``` +## Docker still works, just not here -This grants less than it appears to: `drinkadmin` is already in the `docker` group, and anyone who can talk to the Docker socket can mount the host filesystem into a privileged container, so they are root-equivalent already. If you'd rather not, drop the sudoers file and run the deploy steps by hand from a root shell. +`Dockerfile` and `docker-compose.prod.yml` are maintained for running this app elsewhere (a VPS). `docker compose build && docker compose up -d` works on any normal Docker host. -**Alternative worth considering:** move the deployment out of `/root` into a `drinkadmin`-owned directory and skip sudo entirely. Keep the directory basename `drinktracker` so the compose project name — and therefore the existing container and volume names — stay the same. `.env.production` must be copied across intact or Postgres will reject the app's credentials, since the volume was initialised with the current password. +It does **not** work on this LXC, which is why the app runs natively. Two independent problems, both from unprivileged-LXC nesting: + +1. AppArmor — the container can't load the `docker-default` profile (`apparmor_parser: Access denied`), so any container without `--security-opt apparmor=unconfined` fails. +2. runc can't set `net.ipv4.ip_unprivileged_port_start`, so containers only work with `--network host`. + +`docker build` is impossible regardless: Docker 29 removed the classic builder and BuildKit spawns a builder container without the unconfined option. Docker is installed but disabled; the old containers and volumes are retained until 2026-09-07 as a rollback path. ## Verifying by hand ```bash ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169 -sudo docker compose -f /root/drinktracker/docker-compose.prod.yml ps -sudo docker logs drinktracker-app-1 --tail 50 -sudo docker logs drinktracker-app-1 2>&1 | grep '\[switchboard\]' # AI routing + cost per call +systemctl is-active postgresql@16-main minio drinktracker +sudo journalctl -u drinktracker -n 50 +sudo journalctl -u drinktracker | grep '\[switchboard\]' # AI routing + cost per call +readlink /opt/drinktracker/current # which release is live ``` -Each AI call logs one `[switchboard] feature=… model=… cost=… latency_ms=…` line. `FAILOVER` warnings are expected — the gateway's `:batch` model variants currently fail on every request and fall back. +Each AI call logs one `[switchboard] feature=… model=… cost=… latency_ms=…` line. `FAILOVER` warnings are expected — the gateway's `:batch` model variants currently fail on every request and fall back. `CONTEXT OVERFLOW` is worth investigating. -## Rollback +## Backups + +`drinktracker-backup.timer` runs nightly at 03:25: `pg_dump` + a tar of `/var/lib/minio/data` + a copy of the env file, into `/var/backups/drinktracker`, retained 14 days (~1 GB). + +**TODO — the chain has no off-site leg.** `/usr/local/bin/drinktracker-backup` supports `OFFSITE_DEST` (an scp target) and `OFFSITE_KEY`; set them via `Environment=` in `drinktracker-backup.service`. Until then everything lives on one LXC on one Proxmox host, and the nightly job prints a warning saying so. + +Restore: ```bash -git revert && ./deploy/deploy.sh +sudo systemctl stop drinktracker +zcat /var/backups/drinktracker/pg-.sql.gz | sudo -u postgres psql -d drinkman +sudo systemctl stop minio +sudo tar -C /var/lib/minio -xzf /var/backups/drinktracker/minio-.tgz +sudo chown -R minio:minio /var/lib/minio/data +sudo systemctl start minio drinktracker ``` -Or directly on the LXC, without touching git history: +## Notes -```bash -sudo git -C /root/drinktracker reset --hard -cd /root/drinktracker -sudo docker build -t 192.168.2.140:3000/jpscott84/drinktracker:latest . -sudo docker compose -f docker-compose.prod.yml up -d --no-build -``` - -Note the `migrate` service runs `prisma db push --accept-data-loss` on every start, so a rollback that removes a column will drop it and its data. Take a database dump first if the schema changed. +- The deploy needs unattended sudo on the LXC (`/etc/sudoers.d/drinkadmin-deploy`, currently + `NOPASSWD: ALL`). A narrowly-scoped allowlist was considered and rejected: the script calls + root for `install`, `tee`, `systemctl`, `journalctl` and `sudo -u postgres pg_dump`, so an + allowlist would need updating every time the script changes and would fail in confusing ways + when it drifted. Be aware this buys little security either way — `drinkadmin` is in the `sudo` + group, so full root is a password away regardless. What it grants is *unattended* root. +- `nodejs` is `apt-mark hold`-ed at v20.20.0 so the native runtime can't drift from the Dockerfile's pinned `node:20-alpine`. +- Postgres was created with `en_US.UTF-8`; the Ubuntu default of `C`/`SQL_ASCII` would re-sort every drink list and mangle names like `Grüner Veltliner`. Preserve the locale if the cluster is ever rebuilt. +- `unattended-upgrades` is not installed, so nothing will surprise-restart these services — but postgres will never auto-patch either. A deliberate open question. diff --git a/deploy/deploy.sh b/deploy/deploy.sh index b012abd..3bef7e3 100755 --- a/deploy/deploy.sh +++ b/deploy/deploy.sh @@ -2,28 +2,27 @@ # # Deploy drinktracker to the production LXC. # -# The image is built ON the LXC and tagged with the registry name that -# docker-compose.prod.yml expects. Compose then finds it locally and never pulls, so -# this needs no registry credentials on either machine. Pass --push to additionally -# publish the image to the Gitea registry (requires `docker login` on the LXC). +# The app runs natively under systemd - no Docker. Each deploy builds into +# releases/ out-of-place and swaps a symlink, so the ~3 minute build happens +# while the old release keeps serving and downtime is only the restart. # -# Usage: ./deploy/deploy.sh [--push] [--no-build] [--yes] +# Usage: ./deploy/deploy.sh [--yes] [--no-build] [--rollback] # set -euo pipefail HOST="${DT_HOST:-drinkadmin@192.168.2.169}" SSH_KEY="${DT_SSH_KEY:-$HOME/.ssh/drinktracker_ed25519}" -REMOTE_DIR="${DT_REMOTE_DIR:-/root/drinktracker}" -IMAGE="${DT_IMAGE:-192.168.2.140:3000/jpscott84/drinktracker:latest}" -COMPOSE_FILE="docker-compose.prod.yml" +APP_DIR="${DT_APP_DIR:-/opt/drinktracker}" +SVC_USER="${DT_SVC_USER:-drinktracker}" +KEEP_RELEASES=5 HEALTH_URL="http://localhost:3000/" -PUSH=0; BUILD=1; ASSUME_YES=0 +ASSUME_YES=0; BUILD=1; ROLLBACK=0 for arg in "$@"; do case "$arg" in - --push) PUSH=1 ;; - --no-build) BUILD=0 ;; --yes|-y) ASSUME_YES=1 ;; + --no-build) BUILD=0 ;; + --rollback) ROLLBACK=1 ;; *) echo "unknown option: $arg" >&2; exit 2 ;; esac done @@ -32,87 +31,118 @@ SSH=(ssh -o BatchMode=yes -o ConnectTimeout=10 -i "$SSH_KEY" "$HOST") say() { printf '\n\033[1;36m==> %s\033[0m\n' "$*"; } die() { printf '\n\033[1;31mFAILED: %s\033[0m\n' "$*" >&2; exit 1; } +health_poll() { + for i in $(seq 1 30); do + code=$("${SSH[@]}" "curl -s -o /dev/null -w '%{http_code}' -m 5 $HEALTH_URL" || echo 000) + # 307 is the unauthenticated redirect to /login - healthy. + case "$code" in 200|302|307) echo " healthy (HTTP $code) after $((i*2))s"; return 0 ;; esac + sleep 2 + done + return 1 +} + +# ─── Rollback: repoint the symlink, no rebuild ─────────────────────── +if [ "$ROLLBACK" -eq 1 ]; then + say "Available releases (newest last)" + "${SSH[@]}" "ls -1tr $APP_DIR/releases" + CURRENT=$("${SSH[@]}" "readlink $APP_DIR/current | xargs basename") + PREV=$("${SSH[@]}" "ls -1t $APP_DIR/releases | grep -v '^$CURRENT\$' | head -1") + [ -n "$PREV" ] || die "no other release to roll back to" + echo " current=$CURRENT -> rolling back to $PREV" + if [ "$ASSUME_YES" -ne 1 ]; then + read -r -p " proceed? [y/N] " r; [[ "$r" =~ ^[Yy]$ ]] || { echo aborted; exit 1; } + fi + "${SSH[@]}" "sudo -n -u $SVC_USER bash -c 'ln -sfn $APP_DIR/releases/$PREV $APP_DIR/current.new && mv -Tf $APP_DIR/current.new $APP_DIR/current'" + "${SSH[@]}" "sudo -n systemctl restart drinktracker" + health_poll || die "app unhealthy after rollback - journalctl -u drinktracker" + say "Rolled back to $PREV" + exit 0 +fi + # ─── Preflight (local) ─────────────────────────────────────────────── say "Preflight" cd "$(dirname "$0")/.." - [ -n "$(git status --porcelain)" ] && die "working tree is dirty - commit or stash first" BRANCH=$(git rev-parse --abbrev-ref HEAD) [ "$BRANCH" = "main" ] || die "on branch '$BRANCH', expected main" - npx tsc --noEmit || die "typecheck failed" echo " typecheck ok, tree clean, on main" -# Push first so the server pulls exactly what was verified here. git -c credential.helper=store push origin main -LOCAL_SHA=$(git rev-parse HEAD) -echo " pushed $(git rev-parse --short HEAD)" +SHA=$(git rev-parse HEAD); SHORT=${SHA:0:7} +echo " pushed $SHORT" # ─── Preflight (remote) ────────────────────────────────────────────── say "Checking $HOST" "${SSH[@]}" true || die "cannot reach $HOST with key $SSH_KEY" -"${SSH[@]}" 'sudo -n true' 2>/dev/null \ - || die "passwordless sudo required on the LXC (needed for $REMOTE_DIR). See deploy/README.md" - -# A dirty server checkout means someone edited production by hand; clobbering that -# silently would destroy the only copy of the change. -if ! "${SSH[@]}" "sudo -n git -C $REMOTE_DIR diff --quiet && sudo -n git -C $REMOTE_DIR diff --cached --quiet"; then - die "$REMOTE_DIR has uncommitted changes - inspect before deploying" -fi +for unit in postgresql@16-main minio drinktracker; do + st=$("${SSH[@]}" "systemctl is-active $unit" || true) + echo " $unit: $st" + [ "$st" = "active" ] || die "$unit is not active - fix before deploying" +done if [ "$ASSUME_YES" -ne 1 ]; then - REMOTE_SHA=$("${SSH[@]}" "sudo -n git -C $REMOTE_DIR rev-parse --short HEAD") - echo " remote is at $REMOTE_SHA, deploying ${LOCAL_SHA:0:7}" - read -r -p " proceed? [y/N] " reply - [[ "$reply" =~ ^[Yy]$ ]] || { echo "aborted"; exit 1; } + CURRENT=$("${SSH[@]}" "readlink $APP_DIR/current | xargs basename") + echo " current release $CURRENT, deploying $SHORT" + read -r -p " proceed? [y/N] " r; [[ "$r" =~ ^[Yy]$ ]] || { echo aborted; exit 1; } fi -# ─── Deploy ────────────────────────────────────────────────────────── -say "Syncing $REMOTE_DIR to $LOCAL_SHA" -"${SSH[@]}" "sudo -n git -C $REMOTE_DIR fetch origin main --quiet && sudo -n git -C $REMOTE_DIR reset --hard $LOCAL_SHA --quiet && sudo -n git -C $REMOTE_DIR log --oneline -1" - -# SWITCHBOARD_BASE_URL arrived with the gateway migration and will be missing from -# any .env.production written before it. Compose supplies a default, but making it -# explicit keeps the file honest about what the app reads. -say "Checking .env.production for new variables" -"${SSH[@]}" "sudo -n grep -q '^SWITCHBOARD_BASE_URL=' $REMOTE_DIR/.env.production" \ - && echo " SWITCHBOARD_BASE_URL present" \ - || echo " NOTE: SWITCHBOARD_BASE_URL absent; compose default (http://192.168.2.11:8787/v1) applies" +# ─── Build (out-of-place; the running release keeps serving) ───────── +say "Syncing and building on $HOST" +"${SSH[@]}" "sudo -n -u $SVC_USER git -C $APP_DIR/repo fetch --quiet origin main && sudo -n -u $SVC_USER git -C $APP_DIR/repo reset --hard $SHA --quiet && sudo -n -u $SVC_USER git -C $APP_DIR/repo log --oneline -1" if [ "$BUILD" -eq 1 ]; then - say "Building image on the LXC (this takes a few minutes)" - # Pass the build context as a path rather than cd-ing in: the shell runs as the - # deploy user, which cannot enter /root even though sudo can build there. - "${SSH[@]}" "sudo -n docker build -t $IMAGE $REMOTE_DIR" || die "image build failed" + # Built WITHOUT the env file, deliberately - same as the Dockerfile. next.config.mjs + # evaluates rewrites() at build time, so sourcing the env here would bake a different + # /minio-images destination into routes-manifest.json. + "${SSH[@]}" "sudo -n -u $SVC_USER bash -lc ' + set -e + cd $APP_DIR/repo + export npm_config_cache=$APP_DIR/.npm NODE_OPTIONS=--max-old-space-size=3072 + npm ci --prefer-offline --no-audit --fund=false + npx prisma generate + npm run build + '" || die "build failed" fi -if [ "$PUSH" -eq 1 ]; then - say "Pushing image to registry" - "${SSH[@]}" "sudo -n docker push $IMAGE" || die "registry push failed (is docker login done on the LXC?)" -fi +say "Assembling release $SHORT" +# Mirrors Dockerfile:24-30 - standalone bundle plus static and public copied in. +"${SSH[@]}" "sudo -n -u $SVC_USER bash -lc ' + set -e + REL=$APP_DIR/releases/$SHORT + rm -rf \$REL && mkdir -p \$REL/.next/cache + cp -a $APP_DIR/repo/.next/standalone/. \$REL/ + cp -a $APP_DIR/repo/.next/static \$REL/.next/static + cp -a $APP_DIR/repo/public \$REL/public + du -sh \$REL +'" || die "release assembly failed" -say "Restarting stack" -# --no-build: compose must use the image we just built, not rebuild or pull it. -# Compose takes the project directory from the compose file's location, which keeps -# the project name "drinktracker" and so reuses the existing containers and volumes. -"${SSH[@]}" "sudo -n docker compose -f $REMOTE_DIR/$COMPOSE_FILE up -d --no-build" +# ─── Schema ────────────────────────────────────────────────────────── +say "Backing up database before schema push" +"${SSH[@]}" "sudo -n install -d -m 0750 /var/backups/drinktracker && sudo -n -u postgres pg_dump drinkman | gzip -9 | sudo -n tee /var/backups/drinktracker/pre-$SHORT.sql.gz >/dev/null && sudo -n ls -lh /var/backups/drinktracker/pre-$SHORT.sql.gz" + +say "Applying schema" +# db push, NOT migrate deploy: production has no _prisma_migrations table, so +# migrate deploy would try to apply the init migration against populated tables. +# --accept-data-loss means removing a field from schema.prisma DROPS the column. +"${SSH[@]}" "sudo -n -u $SVC_USER bash -lc ' + set -a; . /etc/drinktracker/drinktracker.env; set +a + cd $APP_DIR/repo && npx prisma db push --skip-generate --accept-data-loss +'" || die "prisma db push failed" + +# ─── Swap and restart ──────────────────────────────────────────────── +say "Swapping to $SHORT and restarting" +"${SSH[@]}" "sudo -n -u $SVC_USER bash -c 'ln -sfn $APP_DIR/releases/$SHORT $APP_DIR/current.new && mv -Tf $APP_DIR/current.new $APP_DIR/current'" +"${SSH[@]}" "sudo -n systemctl restart drinktracker" +health_poll || die "app unhealthy - check: ${SSH[*]} 'journalctl -u drinktracker -n 50'" # ─── Verify ────────────────────────────────────────────────────────── say "Verifying" -"${SSH[@]}" "sudo -n docker compose -f $REMOTE_DIR/$COMPOSE_FILE ps --format 'table {{.Name}}\t{{.Status}}'" || true +"${SSH[@]}" "curl -s -o /dev/null -w ' public=%{http_code}\n' -m 15 https://drinktracker.tenseconddelay.net/ || true" +"${SSH[@]}" "sudo -n journalctl -u drinktracker --since '2 min ago' | grep '\[switchboard\]' | tail -3 || echo ' (no AI calls yet)'" -ok=0 -for i in $(seq 1 30); do - code=$("${SSH[@]}" "curl -s -o /dev/null -w '%{http_code}' -m 5 $HEALTH_URL" || echo 000) - # 307 is the unauthenticated redirect to /login - a healthy response here. - case "$code" in 200|307|302) echo " app responding (HTTP $code) after ${i}0s"; ok=1; break ;; esac - sleep 10 -done -[ "$ok" -eq 1 ] || die "app did not become healthy - check: ${SSH[*]} 'sudo docker logs drinktracker-app-1 --tail 50'" +say "Pruning old releases (keeping $KEEP_RELEASES)" +"${SSH[@]}" "sudo -n -u $SVC_USER bash -c 'cd $APP_DIR/releases && ls -1t | tail -n +$((KEEP_RELEASES+1)) | xargs -r rm -rf; ls -1t'" -say "Recent gateway activity" -"${SSH[@]}" "sudo -n docker logs drinktracker-app-1 --tail 200 2>&1 | grep '\[switchboard\]' | tail -5 || echo ' (no AI calls yet)'" - -say "Deployed ${LOCAL_SHA:0:7}" -echo "To roll back: ./deploy/deploy.sh after 'git revert', or on the LXC:" -echo " sudo git -C $REMOTE_DIR reset --hard && cd $REMOTE_DIR && sudo docker build -t $IMAGE . && sudo docker compose -f $COMPOSE_FILE up -d --no-build" +say "Deployed $SHORT" +echo "Roll back with: ./deploy/deploy.sh --rollback" diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index 761a7cf..c01f847 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -75,7 +75,10 @@ services: restart: "no" app: + # `image:` names the tag this produces; `build:` lets it come up on a host that + # can't reach the LAN-only Gitea registry (e.g. a VPS) via `docker compose build`. image: 192.168.2.140:3000/jpscott84/drinktracker:latest + build: . restart: unless-stopped network_mode: host security_opt: diff --git a/install.sh b/install.sh index f15d1f9..726a051 100644 --- a/install.sh +++ b/install.sh @@ -2,7 +2,13 @@ set -euo pipefail # ───────────────────────────────────────────────────────────── -# DrinkTracker — Production Install Script +# DrinkTracker — Production Install Script (Docker) +# +# For a fresh Docker-based install on a normal host or VPS. +# +# NOTE: the production LXC (192.168.2.169) no longer uses this — it runs the app +# natively under systemd because Docker can't work in an unprivileged LXC. To +# deploy there, use ./deploy/deploy.sh. See deploy/README.md. # ───────────────────────────────────────────────────────────── # Colors