jpscott84/drinktracker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Run natively under systemd; Docker was unworkable in this LXC

Browse Source 
Docker in this unprivileged Proxmox LXC was broken two independent ways:
AppArmor could not load the docker-default profile, so the daemon could
not start any new container, and runc could not set the
net.ipv4.ip_unprivileged_port_start sysctl, which is why every service
needed network_mode: host. `docker build` was impossible outright since
Docker 29 removed the classic builder. The stack only survived because
the containers predated the breakage - a reboot would have left the app
down, and nothing could be redeployed.

Postgres, MinIO and the app now run natively under systemd. Deploys
build out-of-place into releases/<sha> and swap a symlink, so the build
happens while the old release keeps serving and downtime is the ~3s
restart rather than the ~3min build. Rollback is the same swap in
reverse with no rebuild.

Dockerfile and docker-compose.prod.yml are unchanged and still work on
a normal host; the compose app service gains `build: .` so it can come
up on a VPS that cannot reach the LAN-only Gitea registry.

Documents three traps found during the migration: rewrites() in
next.config.mjs is evaluated at build time so MINIO_ENDPOINT changes
silently do nothing, prisma migrate deploy would fail because
production has no _prisma_migrations table, and the image proxy relies
on the bucket's anonymous-download policy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
JP
2026-08-08 17:58:34 +00:00
parent f3865c7901
commit 87f951bf94
5 changed files with 229 additions and 152 deletions
Download Patch File Download Diff File Expand all files Collapse all files

108
.claude/skills/deploy-drinktracker/SKILL.md
View File

@@ -5,65 +5,83 @@ description: Deploy drinktracker to the production LXC and verify it came back h
# Deploying drinktracker # Deploying drinktracker
Run `./deploy/deploy.sh` from the repo root. It typechecks, pushes to Gitea, rebuilds the Run `./deploy/deploy.sh` from the repo root (`--yes` to skip the prompt when the user has
image on the LXC, restarts the stack and polls until the app answers. Use `--yes` to skip already approved). It typechecks, pushes to Gitea, builds natively on the LXC into
the confirmation prompt when the user has already approved the deploy. `releases/<sha>`, swaps the `current` symlink, restarts, and polls until healthy.
`./deploy/deploy.sh --rollback` repoints at the previous release in ~3 seconds with no rebuild.
`deploy/README.md` has the full topology table, the one-time sudoers setup, and rollback `deploy/README.md` has the full topology, backup/restore, and manual verification commands.
commands. Read it before improvising — the layout is unusual in ways that waste time.
## Facts that are not discoverable from the repo **The app runs natively under systemd. There is no Docker on this host.**
- **The LXC is `192.168.2.169`.** `drinktracker.tenseconddelay.net` (and `NEXTAUTH_URL`) ## Facts not discoverable from the repo
resolve to the reverse proxy at `192.168.2.172`. SSHing there fails with what looks like
a rejected key but is simply the wrong machine. - **The LXC is `192.168.2.169`.** `drinktracker.tenseconddelay.net` and `NEXTAUTH_URL` resolve
- **The live stack is `/root/drinktracker`.** The checkouts under `/home/drinkadmin/` to the reverse proxy at `192.168.2.172`. SSHing there fails in a way that looks like a
(`drinktracker`, `drinktracker-latest`) are stale June copies. Confirm with rejected key but is a different machine entirely.
`docker inspect drinktracker-app-1 --format '{{index .Config.Labels "com.docker.compose.project.working_dir"}}'`.
- **There is no CI.** No Gitea Actions, no GitHub workflows. `git push` updates the repo
and nothing else; the image must be built and the stack restarted separately. If someone
reports "I pushed but nothing changed", this is why.
- **Access:** `ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169`. Root SSH is - **Access:** `ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169`. Root SSH is
disabled. `drinkadmin` needs the sudoers drop-in from `deploy/README.md` for the script disabled. Confirm what's live with `readlink /opt/drinktracker/current`.
to work unattended. - **Layout:** app `/opt/drinktracker/{repo,releases/<sha>,current}`, service user
`drinktracker`, secrets `/etc/drinktracker/drinktracker.env`, units `drinktracker`,
`minio`, `postgresql@16-main`.
- **Database is `drinkman`, user `drinkdb`** — not `drinktracker`/`drinktracker`.
- **No CI.** `git push` updates the Gitea repo and nothing else. If someone reports "I pushed
but nothing changed", this is why.
## Three things that will bite you
**`next.config.mjs` evaluates `rewrites()` at build time.** The `/minio-images/:path*`
destination is baked into `routes-manifest.json` as `http://localhost:9000/drink-images/:path*`.
Changing `MINIO_ENDPOINT` in the env file looks like it works and does nothing; images break
only after the next rebuild. MinIO must stay on `localhost:9000`. For the same reason the
deploy builds *without* sourcing the env file, exactly as the Dockerfile does — don't "fix" that.
**`prisma db push`, never `migrate deploy`.** Production has no `_prisma_migrations` table, so
`migrate deploy` would try to apply `20260228220749_init` against 16 populated tables and fail.
This is the obvious-looking modernization that breaks production.
**`--accept-data-loss` drops columns.** Removing a field from `schema.prisma` deletes the column
and its data on the next deploy, in both directions — so rolling back past a schema change is
destructive too. The deploy script dumps to `/var/backups/drinktracker/pre-<sha>.sql.gz` first.
This is also why `UserPreference.defaultProvider` is still in the schema despite being unused.
Manual dump: `sudo -u postgres pg_dump drinkman > ~/dt-$(date +%F).sql`
## Verifying ## Verifying
An unauthenticated request to port 3000 returns **307** (redirect to `/login`) — that is a An unauthenticated request to port 3000 returns **307** (redirect to `/login`) — healthy, not a
healthy response, not a failure. The script accepts 200, 302 and 307. failure. The script accepts 200, 302 and 307.
Each AI call emits one line:
``` ```
sudo docker logs drinktracker-app-1 2>&1 | grep '\[switchboard\]' sudo journalctl -u drinktracker | grep '\[switchboard\]'
``` ```
`FAILOVER` lines are expected and harmless: the gateway's `:batch` model variants fail on Each AI call emits one line with the model and cost. `FAILOVER` lines are expected and harmless:
every request and fall back. `CONTEXT OVERFLOW` is worth investigating — it means a request the gateway's `:batch` model variants fail on every request and fall back. `CONTEXT OVERFLOW`
was too large for any model's window and may have been truncated. means a request exceeded every model's window and may have been truncated — worth investigating.
## Before deploying schema changes
The `migrate` service runs `prisma db push --skip-generate --accept-data-loss` on every
start. Removing a field from `schema.prisma` therefore **drops the column and its data** on
the next deploy, in both directions — so a rollback past a schema change is destructive too.
Dump the database first when the schema changed:
```
sudo docker exec drinktracker-db-1 pg_dump -U drinktracker drinktracker > ~/dt-$(date +%F).sql
```
This is also why `UserPreference.defaultProvider` is still in the schema despite being
unused: deleting it would drop the column and break restoring older backups.
## If the app does not come back ## If the app does not come back
``` ```
sudo docker compose -f /root/drinktracker/docker-compose.prod.yml ps systemctl status drinktracker
sudo docker logs drinktracker-app-1 --tail 100 sudo journalctl -u drinktracker -n 100
systemctl is-active postgresql@16-main minio
readlink /opt/drinktracker/current
``` ```
Most likely causes, in order: the build failed and compose restarted the previous image; Most likely, in order: the build failed and the symlink still points at the old release (so the
`.env.production` is missing a variable the new code reads (compare against `.env.example`); app is fine, the deploy just didn't take); `/etc/drinktracker/drinktracker.env` is missing a
or the `migrate` service exited non-zero and `app` never started, which `ps` will show as variable the new code reads (compare against `.env.example`); or postgres/minio didn't start,
`migrate` in a non-zero exited state. which `is-active` shows immediately.
Note the systemd unit deliberately omits `MemoryDenyWriteExecute` (it breaks the V8 JIT) and
includes `AF_NETLINK` in `RestrictAddressFamilies` (Next needs it at startup). If you harden
that unit further and the app stops booting, those are the two to suspect.
## Docker
`Dockerfile` and `docker-compose.prod.yml` are kept for running this app on a VPS and still work
on a normal Docker host. They cannot work on this LXC: unprivileged-LXC nesting breaks AppArmor
profile loading and the `net.ipv4.ip_unprivileged_port_start` sysctl, and `docker build` is
impossible since Docker 29 removed the classic builder. Docker is installed but disabled; old
containers and volumes are retained until 2026-09-07 as a rollback path.

98
deploy/README.md
View File

@@ -1,71 +1,91 @@
# Deploying drinktracker # Deploying drinktracker
```bash ```bash
./deploy/deploy.sh # typecheck, push, build on the LXC, restart, verify ./deploy/deploy.sh # typecheck, push, build, swap, restart, verify
./deploy/deploy.sh --yes # skip the confirmation prompt ./deploy/deploy.sh --yes # skip the confirmation prompt
./deploy/deploy.sh --push # also publish the image to the Gitea registry ./deploy/deploy.sh --rollback # repoint at the previous release (~3s, no rebuild)
``` ```
## Where things actually live The app runs **natively under systemd — there is no Docker on this host.**
## Where things live
| | | | | |
|---|---| |---|---|
| Production LXC | `192.168.2.169` (hostname `drinktracker`) | | Production LXC | `192.168.2.169` (hostname `drinktracker`, Proxmox vmid 100) |
| Deploy user | `drinkadmin`, key `~/.ssh/drinktracker_ed25519` | | Deploy access | `ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169` |
| Live compose dir | `/root/drinktracker` | | App | `/opt/drinktracker/{repo,releases/<sha>,current}`, service user `drinktracker` |
| Image tag | `192.168.2.140:3000/jpscott84/drinktracker:latest` | | Secrets | `/etc/drinktracker/drinktracker.env` (`root:drinktracker` `0640`) |
| App port | 3000 (host network), unauthenticated requests return 307 → `/login` | | Services | `drinktracker`, `minio`, `postgresql@16-main` |
| Database | `drinkman`, user `drinkdb`, `127.0.0.1:5432` |
| Object storage | MinIO at `127.0.0.1:9000`, bucket `drink-images` |
| App port | 3000 on `0.0.0.0`; unauthenticated requests return 307 → `/login` |
| Backups | `/var/backups/drinktracker`, nightly 03:25, 14-day retention |
Two traps worth knowing: ## Traps
**`drinktracker.tenseconddelay.net` is the reverse proxy (192.168.2.172), not the container.** `NEXTAUTH_URL` points at the public name, so resolving it and SSHing there fails confusingly — the key looks rejected because it's a different machine. **`drinktracker.tenseconddelay.net` resolves to the reverse proxy (192.168.2.172), not the container.** `NEXTAUTH_URL` points at the public name, so resolving it and SSHing there fails in a way that looks like a rejected key but is simply a different machine.
**The two checkouts under `/home/drinkadmin/` are stale.** Only `/root/drinktracker` is live; that's what the running container's compose labels point to. **`next.config.mjs` evaluates `rewrites()` at BUILD time.** The `/minio-images/:path*` destination is baked into `routes-manifest.json` as `http://localhost:9000/drink-images/:path*`. Changing `MINIO_ENDPOINT` in the env file appears to work and does nothing — images break only after the *next* rebuild, so the change and the breakage are separated in time. MinIO must stay on `localhost:9000` serving `drink-images`.
## How it works **The image proxy is an unauthenticated GET.** Next rewrites don't sign requests, so the bucket's anonymous-download policy is load-bearing. Verify with `mc anonymous get local/drink-images` → must say `download`.
The image is built **on the LXC** and tagged with the registry name that `docker-compose.prod.yml` already references. Compose finds it locally and never pulls, so no registry credentials are needed on either machine. `--push` publishes it as well, which is only useful if something else pulls that tag. **`prisma db push`, not `migrate deploy`.** Production has no `_prisma_migrations` table, so `migrate deploy` would try to apply the init migration against populated tables and fail. `--accept-data-loss` means removing a field from `schema.prisma` **drops the column and its data** — the deploy script dumps first for this reason.
There is no CI, so **`git push` on its own deploys nothing** — it only updates the Gitea repo. The build and restart have to happen separately, which is what this script is for. ## How a deploy works
## One-time setup Build happens in `/opt/drinktracker/repo` and is assembled **out-of-place** into `releases/<sha>`, so the ~3 minute build runs while the current release keeps serving. Only the symlink swap and `systemctl restart` cause downtime, about 3 seconds. Rollback is the same swap in reverse and needs no rebuild.
The script needs passwordless sudo on the LXC, because the deployment lives under `/root`. On the container as root: The build deliberately does **not** source the env file — same as the Dockerfile — because of the build-time rewrite baking described above.
```bash ## Docker still works, just not here
echo 'drinkadmin ALL=(root) NOPASSWD: ALL' > /etc/sudoers.d/drinkadmin-deploy
chmod 440 /etc/sudoers.d/drinkadmin-deploy
visudo -c
```
This grants less than it appears to: `drinkadmin` is already in the `docker` group, and anyone who can talk to the Docker socket can mount the host filesystem into a privileged container, so they are root-equivalent already. If you'd rather not, drop the sudoers file and run the deploy steps by hand from a root shell. `Dockerfile` and `docker-compose.prod.yml` are maintained for running this app elsewhere (a VPS). `docker compose build && docker compose up -d` works on any normal Docker host.
**Alternative worth considering:** move the deployment out of `/root` into a `drinkadmin`-owned directory and skip sudo entirely. Keep the directory basename `drinktracker` so the compose project name — and therefore the existing container and volume names — stay the same. `.env.production` must be copied across intact or Postgres will reject the app's credentials, since the volume was initialised with the current password. It does **not** work on this LXC, which is why the app runs natively. Two independent problems, both from unprivileged-LXC nesting:
1. AppArmor — the container can't load the `docker-default` profile (`apparmor_parser: Access denied`), so any container without `--security-opt apparmor=unconfined` fails.
2. runc can't set `net.ipv4.ip_unprivileged_port_start`, so containers only work with `--network host`.
`docker build` is impossible regardless: Docker 29 removed the classic builder and BuildKit spawns a builder container without the unconfined option. Docker is installed but disabled; the old containers and volumes are retained until 2026-09-07 as a rollback path.
## Verifying by hand ## Verifying by hand
```bash ```bash
ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169 ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169
sudo docker compose -f /root/drinktracker/docker-compose.prod.yml ps systemctl is-active postgresql@16-main minio drinktracker
sudo docker logs drinktracker-app-1 --tail 50 sudo journalctl -u drinktracker -n 50
sudo docker logs drinktracker-app-1 2>&1 | grep '\[switchboard\]' # AI routing + cost per call sudo journalctl -u drinktracker | grep '\[switchboard\]' # AI routing + cost per call
readlink /opt/drinktracker/current # which release is live
``` ```
Each AI call logs one `[switchboard] feature=… model=… cost=… latency_ms=…` line. `FAILOVER` warnings are expected — the gateway's `:batch` model variants currently fail on every request and fall back. Each AI call logs one `[switchboard] feature=… model=… cost=… latency_ms=…` line. `FAILOVER` warnings are expected — the gateway's `:batch` model variants currently fail on every request and fall back. `CONTEXT OVERFLOW` is worth investigating.
## Rollback ## Backups
`drinktracker-backup.timer` runs nightly at 03:25: `pg_dump` + a tar of `/var/lib/minio/data` + a copy of the env file, into `/var/backups/drinktracker`, retained 14 days (~1 GB).
**TODO — the chain has no off-site leg.** `/usr/local/bin/drinktracker-backup` supports `OFFSITE_DEST` (an scp target) and `OFFSITE_KEY`; set them via `Environment=` in `drinktracker-backup.service`. Until then everything lives on one LXC on one Proxmox host, and the nightly job prints a warning saying so.
Restore:
```bash ```bash
git revert <bad-sha> && ./deploy/deploy.sh sudo systemctl stop drinktracker
zcat /var/backups/drinktracker/pg-<stamp>.sql.gz | sudo -u postgres psql -d drinkman
sudo systemctl stop minio
sudo tar -C /var/lib/minio -xzf /var/backups/drinktracker/minio-<stamp>.tgz
sudo chown -R minio:minio /var/lib/minio/data
sudo systemctl start minio drinktracker
``` ```
Or directly on the LXC, without touching git history: ## Notes
```bash - The deploy needs unattended sudo on the LXC (`/etc/sudoers.d/drinkadmin-deploy`, currently
sudo git -C /root/drinktracker reset --hard <previous-sha> `NOPASSWD: ALL`). A narrowly-scoped allowlist was considered and rejected: the script calls
cd /root/drinktracker root for `install`, `tee`, `systemctl`, `journalctl` and `sudo -u postgres pg_dump`, so an
sudo docker build -t 192.168.2.140:3000/jpscott84/drinktracker:latest . allowlist would need updating every time the script changes and would fail in confusing ways
sudo docker compose -f docker-compose.prod.yml up -d --no-build when it drifted. Be aware this buys little security either way — `drinkadmin` is in the `sudo`
``` group, so full root is a password away regardless. What it grants is *unattended* root.
- `nodejs` is `apt-mark hold`-ed at v20.20.0 so the native runtime can't drift from the Dockerfile's pinned `node:20-alpine`.
Note the `migrate` service runs `prisma db push --accept-data-loss` on every start, so a rollback that removes a column will drop it and its data. Take a database dump first if the schema changed. - Postgres was created with `en_US.UTF-8`; the Ubuntu default of `C`/`SQL_ASCII` would re-sort every drink list and mangle names like `Grüner Veltliner`. Preserve the locale if the cluster is ever rebuilt.
- `unattended-upgrades` is not installed, so nothing will surprise-restart these services — but postgres will never auto-patch either. A deliberate open question.

164
deploy/deploy.sh
View File

@@ -2,28 +2,27 @@
# #
# Deploy drinktracker to the production LXC. # Deploy drinktracker to the production LXC.
# #
# The image is built ON the LXC and tagged with the registry name that # The app runs natively under systemd - no Docker. Each deploy builds into
# docker-compose.prod.yml expects. Compose then finds it locally and never pulls, so # releases/<sha> out-of-place and swaps a symlink, so the ~3 minute build happens
# this needs no registry credentials on either machine. Pass --push to additionally # while the old release keeps serving and downtime is only the restart.
# publish the image to the Gitea registry (requires `docker login` on the LXC).
# #
# Usage: ./deploy/deploy.sh [--push] [--no-build] [--yes] # Usage: ./deploy/deploy.sh [--yes] [--no-build] [--rollback]
# #
set -euo pipefail set -euo pipefail
HOST="${DT_HOST:-drinkadmin@192.168.2.169}" HOST="${DT_HOST:-drinkadmin@192.168.2.169}"
SSH_KEY="${DT_SSH_KEY:-$HOME/.ssh/drinktracker_ed25519}" SSH_KEY="${DT_SSH_KEY:-$HOME/.ssh/drinktracker_ed25519}"
REMOTE_DIR="${DT_REMOTE_DIR:-/root/drinktracker}" APP_DIR="${DT_APP_DIR:-/opt/drinktracker}"
IMAGE="${DT_IMAGE:-192.168.2.140:3000/jpscott84/drinktracker:latest}" SVC_USER="${DT_SVC_USER:-drinktracker}"
COMPOSE_FILE="docker-compose.prod.yml" KEEP_RELEASES=5
HEALTH_URL="http://localhost:3000/" HEALTH_URL="http://localhost:3000/"
PUSH=0; BUILD=1; ASSUME_YES=0 ASSUME_YES=0; BUILD=1; ROLLBACK=0
for arg in "$@"; do for arg in "$@"; do
case "$arg" in case "$arg" in
--push) PUSH=1 ;;
--no-build) BUILD=0 ;;
--yes|-y) ASSUME_YES=1 ;; --yes|-y) ASSUME_YES=1 ;;
--no-build) BUILD=0 ;;
--rollback) ROLLBACK=1 ;;
*) echo "unknown option: $arg" >&2; exit 2 ;; *) echo "unknown option: $arg" >&2; exit 2 ;;
esac esac
done done
@@ -32,87 +31,118 @@ SSH=(ssh -o BatchMode=yes -o ConnectTimeout=10 -i "$SSH_KEY" "$HOST")
say() { printf '\n\033[1;36m==> %s\033[0m\n' "$*"; } say() { printf '\n\033[1;36m==> %s\033[0m\n' "$*"; }
die() { printf '\n\033[1;31mFAILED: %s\033[0m\n' "$*" >&2; exit 1; } die() { printf '\n\033[1;31mFAILED: %s\033[0m\n' "$*" >&2; exit 1; }
health_poll() {
for i in $(seq 1 30); do
code=$("${SSH[@]}" "curl -s -o /dev/null -w '%{http_code}' -m 5 $HEALTH_URL" || echo 000)
# 307 is the unauthenticated redirect to /login - healthy.
case "$code" in 200|302|307) echo " healthy (HTTP $code) after $((i*2))s"; return 0 ;; esac
sleep 2
done
return 1
}
# ─── Rollback: repoint the symlink, no rebuild ───────────────────────
if [ "$ROLLBACK" -eq 1 ]; then
say "Available releases (newest last)"
"${SSH[@]}" "ls -1tr $APP_DIR/releases"
CURRENT=$("${SSH[@]}" "readlink $APP_DIR/current | xargs basename")
PREV=$("${SSH[@]}" "ls -1t $APP_DIR/releases | grep -v '^$CURRENT\$' | head -1")
[ -n "$PREV" ] || die "no other release to roll back to"
echo " current=$CURRENT -> rolling back to $PREV"
if [ "$ASSUME_YES" -ne 1 ]; then
read -r -p " proceed? [y/N] " r; [[ "$r" =~ ^[Yy]$ ]] || { echo aborted; exit 1; }
fi
"${SSH[@]}" "sudo -n -u $SVC_USER bash -c 'ln -sfn $APP_DIR/releases/$PREV $APP_DIR/current.new && mv -Tf $APP_DIR/current.new $APP_DIR/current'"
"${SSH[@]}" "sudo -n systemctl restart drinktracker"
health_poll || die "app unhealthy after rollback - journalctl -u drinktracker"
say "Rolled back to $PREV"
exit 0
fi
# ─── Preflight (local) ─────────────────────────────────────────────── # ─── Preflight (local) ───────────────────────────────────────────────
say "Preflight" say "Preflight"
cd "$(dirname "$0")/.." cd "$(dirname "$0")/.."
[ -n "$(git status --porcelain)" ] && die "working tree is dirty - commit or stash first" [ -n "$(git status --porcelain)" ] && die "working tree is dirty - commit or stash first"
BRANCH=$(git rev-parse --abbrev-ref HEAD) BRANCH=$(git rev-parse --abbrev-ref HEAD)
[ "$BRANCH" = "main" ] || die "on branch '$BRANCH', expected main" [ "$BRANCH" = "main" ] || die "on branch '$BRANCH', expected main"
npx tsc --noEmit || die "typecheck failed" npx tsc --noEmit || die "typecheck failed"
echo " typecheck ok, tree clean, on main" echo " typecheck ok, tree clean, on main"
# Push first so the server pulls exactly what was verified here.
git -c credential.helper=store push origin main git -c credential.helper=store push origin main
LOCAL_SHA=$(git rev-parse HEAD) SHA=$(git rev-parse HEAD); SHORT=${SHA:0:7}
echo " pushed $(git rev-parse --short HEAD)" echo " pushed $SHORT"
# ─── Preflight (remote) ────────────────────────────────────────────── # ─── Preflight (remote) ──────────────────────────────────────────────
say "Checking $HOST" say "Checking $HOST"
"${SSH[@]}" true || die "cannot reach $HOST with key $SSH_KEY" "${SSH[@]}" true || die "cannot reach $HOST with key $SSH_KEY"
"${SSH[@]}" 'sudo -n true' 2>/dev/null \ for unit in postgresql@16-main minio drinktracker; do
|| die "passwordless sudo required on the LXC (needed for $REMOTE_DIR). See deploy/README.md" st=$("${SSH[@]}" "systemctl is-active $unit" || true)
echo " $unit: $st"
# A dirty server checkout means someone edited production by hand; clobbering that [ "$st" = "active" ] || die "$unit is not active - fix before deploying"
# silently would destroy the only copy of the change. done
if ! "${SSH[@]}" "sudo -n git -C $REMOTE_DIR diff --quiet && sudo -n git -C $REMOTE_DIR diff --cached --quiet"; then
die "$REMOTE_DIR has uncommitted changes - inspect before deploying"
fi
if [ "$ASSUME_YES" -ne 1 ]; then if [ "$ASSUME_YES" -ne 1 ]; then
REMOTE_SHA=$("${SSH[@]}" "sudo -n git -C $REMOTE_DIR rev-parse --short HEAD") CURRENT=$("${SSH[@]}" "readlink $APP_DIR/current | xargs basename")
echo " remote is at $REMOTE_SHA, deploying ${LOCAL_SHA:0:7}" echo " current release $CURRENT, deploying $SHORT"
read -r -p " proceed? [y/N] " reply read -r -p " proceed? [y/N] " r; [[ "$r" =~ ^[Yy]$ ]] || { echo aborted; exit 1; }
[[ "$reply" =~ ^[Yy]$ ]] || { echo "aborted"; exit 1; }
fi fi
# ─── Deploy ────────────────────────────────────────────────────────── # ─── Build (out-of-place; the running release keeps serving) ─────────
say "Syncing $REMOTE_DIR to $LOCAL_SHA" say "Syncing and building on $HOST"
"${SSH[@]}" "sudo -n git -C $REMOTE_DIR fetch origin main --quiet && sudo -n git -C $REMOTE_DIR reset --hard $LOCAL_SHA --quiet && sudo -n git -C $REMOTE_DIR log --oneline -1" "${SSH[@]}" "sudo -n -u $SVC_USER git -C $APP_DIR/repo fetch --quiet origin main && sudo -n -u $SVC_USER git -C $APP_DIR/repo reset --hard $SHA --quiet && sudo -n -u $SVC_USER git -C $APP_DIR/repo log --oneline -1"
# SWITCHBOARD_BASE_URL arrived with the gateway migration and will be missing from
# any .env.production written before it. Compose supplies a default, but making it
# explicit keeps the file honest about what the app reads.
say "Checking .env.production for new variables"
"${SSH[@]}" "sudo -n grep -q '^SWITCHBOARD_BASE_URL=' $REMOTE_DIR/.env.production" \
&& echo " SWITCHBOARD_BASE_URL present" \
|| echo " NOTE: SWITCHBOARD_BASE_URL absent; compose default (http://192.168.2.11:8787/v1) applies"
if [ "$BUILD" -eq 1 ]; then if [ "$BUILD" -eq 1 ]; then
say "Building image on the LXC (this takes a few minutes)" # Built WITHOUT the env file, deliberately - same as the Dockerfile. next.config.mjs
# Pass the build context as a path rather than cd-ing in: the shell runs as the # evaluates rewrites() at build time, so sourcing the env here would bake a different
# deploy user, which cannot enter /root even though sudo can build there. # /minio-images destination into routes-manifest.json.
"${SSH[@]}" "sudo -n docker build -t $IMAGE $REMOTE_DIR" || die "image build failed" "${SSH[@]}" "sudo -n -u $SVC_USER bash -lc '
set -e
cd $APP_DIR/repo
export npm_config_cache=$APP_DIR/.npm NODE_OPTIONS=--max-old-space-size=3072
npm ci --prefer-offline --no-audit --fund=false
npx prisma generate
npm run build
'" || die "build failed"
fi fi
if [ "$PUSH" -eq 1 ]; then say "Assembling release $SHORT"
say "Pushing image to registry" # Mirrors Dockerfile:24-30 - standalone bundle plus static and public copied in.
"${SSH[@]}" "sudo -n docker push $IMAGE" || die "registry push failed (is docker login done on the LXC?)" "${SSH[@]}" "sudo -n -u $SVC_USER bash -lc '
fi set -e
REL=$APP_DIR/releases/$SHORT
rm -rf \$REL && mkdir -p \$REL/.next/cache
cp -a $APP_DIR/repo/.next/standalone/. \$REL/
cp -a $APP_DIR/repo/.next/static \$REL/.next/static
cp -a $APP_DIR/repo/public \$REL/public
du -sh \$REL
'" || die "release assembly failed"
say "Restarting stack" # ─── Schema ──────────────────────────────────────────────────────────
# --no-build: compose must use the image we just built, not rebuild or pull it. say "Backing up database before schema push"
# Compose takes the project directory from the compose file's location, which keeps "${SSH[@]}" "sudo -n install -d -m 0750 /var/backups/drinktracker && sudo -n -u postgres pg_dump drinkman | gzip -9 | sudo -n tee /var/backups/drinktracker/pre-$SHORT.sql.gz >/dev/null && sudo -n ls -lh /var/backups/drinktracker/pre-$SHORT.sql.gz"
# the project name "drinktracker" and so reuses the existing containers and volumes.
"${SSH[@]}" "sudo -n docker compose -f $REMOTE_DIR/$COMPOSE_FILE up -d --no-build" say "Applying schema"
# db push, NOT migrate deploy: production has no _prisma_migrations table, so
# migrate deploy would try to apply the init migration against populated tables.
# --accept-data-loss means removing a field from schema.prisma DROPS the column.
"${SSH[@]}" "sudo -n -u $SVC_USER bash -lc '
set -a; . /etc/drinktracker/drinktracker.env; set +a
cd $APP_DIR/repo && npx prisma db push --skip-generate --accept-data-loss
'" || die "prisma db push failed"
# ─── Swap and restart ────────────────────────────────────────────────
say "Swapping to $SHORT and restarting"
"${SSH[@]}" "sudo -n -u $SVC_USER bash -c 'ln -sfn $APP_DIR/releases/$SHORT $APP_DIR/current.new && mv -Tf $APP_DIR/current.new $APP_DIR/current'"
"${SSH[@]}" "sudo -n systemctl restart drinktracker"
health_poll || die "app unhealthy - check: ${SSH[*]} 'journalctl -u drinktracker -n 50'"
# ─── Verify ────────────────────────────────────────────────────────── # ─── Verify ──────────────────────────────────────────────────────────
say "Verifying" say "Verifying"
"${SSH[@]}" "sudo -n docker compose -f $REMOTE_DIR/$COMPOSE_FILE ps --format 'table {{.Name}}\t{{.Status}}'" || true "${SSH[@]}" "curl -s -o /dev/null -w ' public=%{http_code}\n' -m 15 https://drinktracker.tenseconddelay.net/ || true"
"${SSH[@]}" "sudo -n journalctl -u drinktracker --since '2 min ago' | grep '\[switchboard\]' | tail -3 || echo ' (no AI calls yet)'"
ok=0 say "Pruning old releases (keeping $KEEP_RELEASES)"
for i in $(seq 1 30); do "${SSH[@]}" "sudo -n -u $SVC_USER bash -c 'cd $APP_DIR/releases && ls -1t | tail -n +$((KEEP_RELEASES+1)) | xargs -r rm -rf; ls -1t'"
code=$("${SSH[@]}" "curl -s -o /dev/null -w '%{http_code}' -m 5 $HEALTH_URL" || echo 000)
# 307 is the unauthenticated redirect to /login - a healthy response here.
case "$code" in 200|307|302) echo " app responding (HTTP $code) after ${i}0s"; ok=1; break ;; esac
sleep 10
done
[ "$ok" -eq 1 ] || die "app did not become healthy - check: ${SSH[*]} 'sudo docker logs drinktracker-app-1 --tail 50'"
say "Recent gateway activity" say "Deployed $SHORT"
"${SSH[@]}" "sudo -n docker logs drinktracker-app-1 --tail 200 2>&1 | grep '\[switchboard\]' | tail -5 || echo ' (no AI calls yet)'" echo "Roll back with: ./deploy/deploy.sh --rollback"
say "Deployed ${LOCAL_SHA:0:7}"
echo "To roll back: ./deploy/deploy.sh after 'git revert', or on the LXC:"
echo " sudo git -C $REMOTE_DIR reset --hard <previous-sha> && cd $REMOTE_DIR && sudo docker build -t $IMAGE . && sudo docker compose -f $COMPOSE_FILE up -d --no-build"

3
docker-compose.prod.yml
View File

@@ -75,7 +75,10 @@ services:
restart: "no" restart: "no"
app: app:
# `image:` names the tag this produces; `build:` lets it come up on a host that
# can't reach the LAN-only Gitea registry (e.g. a VPS) via `docker compose build`.
image: 192.168.2.140:3000/jpscott84/drinktracker:latest image: 192.168.2.140:3000/jpscott84/drinktracker:latest
build: .
restart: unless-stopped restart: unless-stopped
network_mode: host network_mode: host
security_opt: security_opt:

8
install.sh
View File

@@ -2,7 +2,13 @@
set -euo pipefail set -euo pipefail
# ───────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────
# DrinkTracker — Production Install Script # DrinkTracker — Production Install Script (Docker)
#
# For a fresh Docker-based install on a normal host or VPS.
#
# NOTE: the production LXC (192.168.2.169) no longer uses this — it runs the app
# natively under systemd because Docker can't work in an unprivileged LXC. To
# deploy there, use ./deploy/deploy.sh. See deploy/README.md.
# ───────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────
# Colors # Colors