jpscott84/drinktracker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Run natively under systemd; Docker was unworkable in this LXC

Browse Source 
Docker in this unprivileged Proxmox LXC was broken two independent ways:
AppArmor could not load the docker-default profile, so the daemon could
not start any new container, and runc could not set the
net.ipv4.ip_unprivileged_port_start sysctl, which is why every service
needed network_mode: host. `docker build` was impossible outright since
Docker 29 removed the classic builder. The stack only survived because
the containers predated the breakage - a reboot would have left the app
down, and nothing could be redeployed.

Postgres, MinIO and the app now run natively under systemd. Deploys
build out-of-place into releases/<sha> and swap a symlink, so the build
happens while the old release keeps serving and downtime is the ~3s
restart rather than the ~3min build. Rollback is the same swap in
reverse with no rebuild.

Dockerfile and docker-compose.prod.yml are unchanged and still work on
a normal host; the compose app service gains `build: .` so it can come
up on a VPS that cannot reach the LAN-only Gitea registry.

Documents three traps found during the migration: rewrites() in
next.config.mjs is evaluated at build time so MINIO_ENDPOINT changes
silently do nothing, prisma migrate deploy would fail because
production has no _prisma_migrations table, and the image proxy relies
on the bucket's anonymous-download policy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
JP
2026-08-08 17:58:34 +00:00
parent f3865c7901
commit 87f951bf94
5 changed files with 229 additions and 152 deletions
Download Patch File Download Diff File Expand all files Collapse all files

108
.claude/skills/deploy-drinktracker/SKILL.md
View File

@@ -5,65 +5,83 @@ description: Deploy drinktracker to the production LXC and verify it came back h
# Deploying drinktracker
Run `./deploy/deploy.sh` from the repo root. It typechecks, pushes to Gitea, rebuilds the
image on the LXC, restarts the stack and polls until the app answers. Use `--yes` to skip
the confirmation prompt when the user has already approved the deploy.
Run `./deploy/deploy.sh` from the repo root (`--yes` to skip the prompt when the user has
already approved). It typechecks, pushes to Gitea, builds natively on the LXC into
`releases/<sha>`, swaps the `current` symlink, restarts, and polls until healthy.
`./deploy/deploy.sh --rollback` repoints at the previous release in ~3 seconds with no rebuild.
`deploy/README.md` has the full topology table, the one-time sudoers setup, and rollback
commands. Read it before improvising — the layout is unusual in ways that waste time.
`deploy/README.md` has the full topology, backup/restore, and manual verification commands.
## Facts that are not discoverable from the repo
**The app runs natively under systemd. There is no Docker on this host.**
- **The LXC is `192.168.2.169`.** `drinktracker.tenseconddelay.net` (and `NEXTAUTH_URL`)
resolve to the reverse proxy at `192.168.2.172`. SSHing there fails with what looks like
a rejected key but is simply the wrong machine.
- **The live stack is `/root/drinktracker`.** The checkouts under `/home/drinkadmin/`
(`drinktracker`, `drinktracker-latest`) are stale June copies. Confirm with
`docker inspect drinktracker-app-1 --format '{{index .Config.Labels "com.docker.compose.project.working_dir"}}'`.
- **There is no CI.** No Gitea Actions, no GitHub workflows. `git push` updates the repo
and nothing else; the image must be built and the stack restarted separately. If someone
reports "I pushed but nothing changed", this is why.
## Facts not discoverable from the repo
- **The LXC is `192.168.2.169`.** `drinktracker.tenseconddelay.net` and `NEXTAUTH_URL` resolve
to the reverse proxy at `192.168.2.172`. SSHing there fails in a way that looks like a
rejected key but is a different machine entirely.
- **Access:** `ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169`. Root SSH is
disabled. `drinkadmin` needs the sudoers drop-in from `deploy/README.md` for the script
to work unattended.
disabled. Confirm what's live with `readlink /opt/drinktracker/current`.
- **Layout:** app `/opt/drinktracker/{repo,releases/<sha>,current}`, service user
`drinktracker`, secrets `/etc/drinktracker/drinktracker.env`, units `drinktracker`,
`minio`, `postgresql@16-main`.
- **Database is `drinkman`, user `drinkdb`** — not `drinktracker`/`drinktracker`.
- **No CI.** `git push` updates the Gitea repo and nothing else. If someone reports "I pushed
but nothing changed", this is why.
## Three things that will bite you
**`next.config.mjs` evaluates `rewrites()` at build time.** The `/minio-images/:path*`
destination is baked into `routes-manifest.json` as `http://localhost:9000/drink-images/:path*`.
Changing `MINIO_ENDPOINT` in the env file looks like it works and does nothing; images break
only after the next rebuild. MinIO must stay on `localhost:9000`. For the same reason the
deploy builds *without* sourcing the env file, exactly as the Dockerfile does — don't "fix" that.
**`prisma db push`, never `migrate deploy`.** Production has no `_prisma_migrations` table, so
`migrate deploy` would try to apply `20260228220749_init` against 16 populated tables and fail.
This is the obvious-looking modernization that breaks production.
**`--accept-data-loss` drops columns.** Removing a field from `schema.prisma` deletes the column
and its data on the next deploy, in both directions — so rolling back past a schema change is
destructive too. The deploy script dumps to `/var/backups/drinktracker/pre-<sha>.sql.gz` first.
This is also why `UserPreference.defaultProvider` is still in the schema despite being unused.
Manual dump: `sudo -u postgres pg_dump drinkman > ~/dt-$(date +%F).sql`
## Verifying
An unauthenticated request to port 3000 returns **307** (redirect to `/login`) — that is a
healthy response, not a failure. The script accepts 200, 302 and 307.
Each AI call emits one line:
An unauthenticated request to port 3000 returns **307** (redirect to `/login`) — healthy, not a
failure. The script accepts 200, 302 and 307.
```
sudo docker logs drinktracker-app-1 2>&1 | grep '\[switchboard\]'
sudo journalctl -u drinktracker | grep '\[switchboard\]'
```
`FAILOVER` lines are expected and harmless: the gateway's `:batch` model variants fail on
every request and fall back. `CONTEXT OVERFLOW` is worth investigating — it means a request
was too large for any model's window and may have been truncated.
## Before deploying schema changes
The `migrate` service runs `prisma db push --skip-generate --accept-data-loss` on every
start. Removing a field from `schema.prisma` therefore **drops the column and its data** on
the next deploy, in both directions — so a rollback past a schema change is destructive too.
Dump the database first when the schema changed:
```
sudo docker exec drinktracker-db-1 pg_dump -U drinktracker drinktracker > ~/dt-$(date +%F).sql
```
This is also why `UserPreference.defaultProvider` is still in the schema despite being
unused: deleting it would drop the column and break restoring older backups.
Each AI call emits one line with the model and cost. `FAILOVER` lines are expected and harmless:
the gateway's `:batch` model variants fail on every request and fall back. `CONTEXT OVERFLOW`
means a request exceeded every model's window and may have been truncated — worth investigating.
## If the app does not come back
```
sudo docker compose -f /root/drinktracker/docker-compose.prod.yml ps
sudo docker logs drinktracker-app-1 --tail 100
systemctl status drinktracker
sudo journalctl -u drinktracker -n 100
systemctl is-active postgresql@16-main minio
readlink /opt/drinktracker/current
```
Most likely causes, in order: the build failed and compose restarted the previous image;
`.env.production` is missing a variable the new code reads (compare against `.env.example`);
or the `migrate` service exited non-zero and `app` never started, which `ps` will show as
`migrate` in a non-zero exited state.
Most likely, in order: the build failed and the symlink still points at the old release (so the
app is fine, the deploy just didn't take); `/etc/drinktracker/drinktracker.env` is missing a
variable the new code reads (compare against `.env.example`); or postgres/minio didn't start,
which `is-active` shows immediately.
Note the systemd unit deliberately omits `MemoryDenyWriteExecute` (it breaks the V8 JIT) and
includes `AF_NETLINK` in `RestrictAddressFamilies` (Next needs it at startup). If you harden
that unit further and the app stops booting, those are the two to suspect.
## Docker
`Dockerfile` and `docker-compose.prod.yml` are kept for running this app on a VPS and still work
on a normal Docker host. They cannot work on this LXC: unprivileged-LXC nesting breaks AppArmor
profile loading and the `net.ipv4.ip_unprivileged_port_start` sysctl, and `docker build` is
impossible since Docker 29 removed the classic builder. Docker is installed but disabled; old
containers and volumes are retained until 2026-09-07 as a rollback path.