Files

JP 87f951bf94 Run natively under systemd; Docker was unworkable in this LXC

Docker in this unprivileged Proxmox LXC was broken two independent ways:
AppArmor could not load the docker-default profile, so the daemon could
not start any new container, and runc could not set the
net.ipv4.ip_unprivileged_port_start sysctl, which is why every service
needed network_mode: host. `docker build` was impossible outright since
Docker 29 removed the classic builder. The stack only survived because
the containers predated the breakage - a reboot would have left the app
down, and nothing could be redeployed.

Postgres, MinIO and the app now run natively under systemd. Deploys
build out-of-place into releases/<sha> and swap a symlink, so the build
happens while the old release keeps serving and downtime is the ~3s
restart rather than the ~3min build. Rollback is the same swap in
reverse with no rebuild.

Dockerfile and docker-compose.prod.yml are unchanged and still work on
a normal host; the compose app service gains `build: .` so it can come
up on a VPS that cannot reach the LAN-only Gitea registry.

Documents three traps found during the migration: rewrites() in
next.config.mjs is evaluated at build time so MINIO_ENDPOINT changes
silently do nothing, prisma migrate deploy would fail because
production has no _prisma_migrations table, and the image proxy relies
on the bucket's anonymous-download policy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-08 17:58:34 +00:00

4.6 KiB

Raw Permalink Blame History

name, description

name	description
deploy-drinktracker	Deploy drinktracker to the production LXC and verify it came back healthy. Use when asked to deploy, ship, release, push to prod, or update the running app - and after merging changes that need to go live. Also covers rollback and diagnosing a deploy that did not take effect.

Deploying drinktracker

Run ./deploy/deploy.sh from the repo root (--yes to skip the prompt when the user has already approved). It typechecks, pushes to Gitea, builds natively on the LXC into releases/<sha>, swaps the current symlink, restarts, and polls until healthy. ./deploy/deploy.sh --rollback repoints at the previous release in ~3 seconds with no rebuild.

deploy/README.md has the full topology, backup/restore, and manual verification commands.

The app runs natively under systemd. There is no Docker on this host.

Facts not discoverable from the repo

The LXC is 192.168.2.169. drinktracker.tenseconddelay.net and NEXTAUTH_URL resolve to the reverse proxy at 192.168.2.172. SSHing there fails in a way that looks like a rejected key but is a different machine entirely.
Access: ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169. Root SSH is disabled. Confirm what's live with readlink /opt/drinktracker/current.
Layout: app /opt/drinktracker/{repo,releases/<sha>,current}, service user drinktracker, secrets /etc/drinktracker/drinktracker.env, units drinktracker, minio, postgresql@16-main.
Database is drinkman, user drinkdb — not drinktracker/drinktracker.
No CI. git push updates the Gitea repo and nothing else. If someone reports "I pushed but nothing changed", this is why.

Three things that will bite you

next.config.mjs evaluates rewrites() at build time. The /minio-images/:path* destination is baked into routes-manifest.json as http://localhost:9000/drink-images/:path*. Changing MINIO_ENDPOINT in the env file looks like it works and does nothing; images break only after the next rebuild. MinIO must stay on localhost:9000. For the same reason the deploy builds without sourcing the env file, exactly as the Dockerfile does — don't "fix" that.

prisma db push, never migrate deploy. Production has no _prisma_migrations table, so migrate deploy would try to apply 20260228220749_init against 16 populated tables and fail. This is the obvious-looking modernization that breaks production.

--accept-data-loss drops columns. Removing a field from schema.prisma deletes the column and its data on the next deploy, in both directions — so rolling back past a schema change is destructive too. The deploy script dumps to /var/backups/drinktracker/pre-<sha>.sql.gz first. This is also why UserPreference.defaultProvider is still in the schema despite being unused.

Manual dump: sudo -u postgres pg_dump drinkman > ~/dt-$(date +%F).sql

Verifying

An unauthenticated request to port 3000 returns 307 (redirect to /login) — healthy, not a failure. The script accepts 200, 302 and 307.

sudo journalctl -u drinktracker | grep '\[switchboard\]'

Each AI call emits one line with the model and cost. FAILOVER lines are expected and harmless: the gateway's :batch model variants fail on every request and fall back. CONTEXT OVERFLOW means a request exceeded every model's window and may have been truncated — worth investigating.

If the app does not come back

systemctl status drinktracker
sudo journalctl -u drinktracker -n 100
systemctl is-active postgresql@16-main minio
readlink /opt/drinktracker/current

Most likely, in order: the build failed and the symlink still points at the old release (so the app is fine, the deploy just didn't take); /etc/drinktracker/drinktracker.env is missing a variable the new code reads (compare against .env.example); or postgres/minio didn't start, which is-active shows immediately.

Note the systemd unit deliberately omits MemoryDenyWriteExecute (it breaks the V8 JIT) and includes AF_NETLINK in RestrictAddressFamilies (Next needs it at startup). If you harden that unit further and the app stops booting, those are the two to suspect.

Docker

Dockerfile and docker-compose.prod.yml are kept for running this app on a VPS and still work on a normal Docker host. They cannot work on this LXC: unprivileged-LXC nesting breaks AppArmor profile loading and the net.ipv4.ip_unprivileged_port_start sysctl, and docker build is impossible since Docker 29 removed the classic builder. Docker is installed but disabled; old containers and volumes are retained until 2026-09-07 as a rollback path.

4.6 KiB Raw Permalink Blame History