jpscott84/drinktracker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Run natively under systemd; Docker was unworkable in this LXC

Browse Source 
Docker in this unprivileged Proxmox LXC was broken two independent ways:
AppArmor could not load the docker-default profile, so the daemon could
not start any new container, and runc could not set the
net.ipv4.ip_unprivileged_port_start sysctl, which is why every service
needed network_mode: host. `docker build` was impossible outright since
Docker 29 removed the classic builder. The stack only survived because
the containers predated the breakage - a reboot would have left the app
down, and nothing could be redeployed.

Postgres, MinIO and the app now run natively under systemd. Deploys
build out-of-place into releases/<sha> and swap a symlink, so the build
happens while the old release keeps serving and downtime is the ~3s
restart rather than the ~3min build. Rollback is the same swap in
reverse with no rebuild.

Dockerfile and docker-compose.prod.yml are unchanged and still work on
a normal host; the compose app service gains `build: .` so it can come
up on a VPS that cannot reach the LAN-only Gitea registry.

Documents three traps found during the migration: rewrites() in
next.config.mjs is evaluated at build time so MINIO_ENDPOINT changes
silently do nothing, prisma migrate deploy would fail because
production has no _prisma_migrations table, and the image proxy relies
on the bucket's anonymous-download policy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
JP
2026-08-08 17:58:34 +00:00
parent f3865c7901
commit 87f951bf94
5 changed files with 229 additions and 152 deletions
Download Patch File Download Diff File Expand all files Collapse all files

98
deploy/README.md
View File

@@ -1,71 +1,91 @@
# Deploying drinktracker
```bash
./deploy/deploy.sh # typecheck, push, build on the LXC, restart, verify
./deploy/deploy.sh --yes # skip the confirmation prompt
./deploy/deploy.sh --push # also publish the image to the Gitea registry
./deploy/deploy.sh # typecheck, push, build, swap, restart, verify
./deploy/deploy.sh --yes # skip the confirmation prompt
./deploy/deploy.sh --rollback # repoint at the previous release (~3s, no rebuild)
```
## Where things actually live
The app runs **natively under systemd — there is no Docker on this host.**
## Where things live
| | |
|---|---|
| Production LXC | `192.168.2.169` (hostname `drinktracker`) |
| Deploy user | `drinkadmin`, key `~/.ssh/drinktracker_ed25519` |
| Live compose dir | `/root/drinktracker` |
| Image tag | `192.168.2.140:3000/jpscott84/drinktracker:latest` |
| App port | 3000 (host network), unauthenticated requests return 307 → `/login` |
| Production LXC | `192.168.2.169` (hostname `drinktracker`, Proxmox vmid 100) |
| Deploy access | `ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169` |
| App | `/opt/drinktracker/{repo,releases/<sha>,current}`, service user `drinktracker` |
| Secrets | `/etc/drinktracker/drinktracker.env` (`root:drinktracker` `0640`) |
| Services | `drinktracker`, `minio`, `postgresql@16-main` |
| Database | `drinkman`, user `drinkdb`, `127.0.0.1:5432` |
| Object storage | MinIO at `127.0.0.1:9000`, bucket `drink-images` |
| App port | 3000 on `0.0.0.0`; unauthenticated requests return 307 → `/login` |
| Backups | `/var/backups/drinktracker`, nightly 03:25, 14-day retention |
Two traps worth knowing:
## Traps
**`drinktracker.tenseconddelay.net` is the reverse proxy (192.168.2.172), not the container.** `NEXTAUTH_URL` points at the public name, so resolving it and SSHing there fails confusingly — the key looks rejected because it's a different machine.
**`drinktracker.tenseconddelay.net` resolves to the reverse proxy (192.168.2.172), not the container.** `NEXTAUTH_URL` points at the public name, so resolving it and SSHing there fails in a way that looks like a rejected key but is simply a different machine.
**The two checkouts under `/home/drinkadmin/` are stale.** Only `/root/drinktracker` is live; that's what the running container's compose labels point to.
**`next.config.mjs` evaluates `rewrites()` at BUILD time.** The `/minio-images/:path*` destination is baked into `routes-manifest.json` as `http://localhost:9000/drink-images/:path*`. Changing `MINIO_ENDPOINT` in the env file appears to work and does nothing — images break only after the *next* rebuild, so the change and the breakage are separated in time. MinIO must stay on `localhost:9000` serving `drink-images`.
## How it works
**The image proxy is an unauthenticated GET.** Next rewrites don't sign requests, so the bucket's anonymous-download policy is load-bearing. Verify with `mc anonymous get local/drink-images` → must say `download`.
The image is built **on the LXC** and tagged with the registry name that `docker-compose.prod.yml` already references. Compose finds it locally and never pulls, so no registry credentials are needed on either machine. `--push` publishes it as well, which is only useful if something else pulls that tag.
**`prisma db push`, not `migrate deploy`.** Production has no `_prisma_migrations` table, so `migrate deploy` would try to apply the init migration against populated tables and fail. `--accept-data-loss` means removing a field from `schema.prisma` **drops the column and its data** — the deploy script dumps first for this reason.
There is no CI, so **`git push` on its own deploys nothing** — it only updates the Gitea repo. The build and restart have to happen separately, which is what this script is for.
## How a deploy works
## One-time setup
Build happens in `/opt/drinktracker/repo` and is assembled **out-of-place** into `releases/<sha>`, so the ~3 minute build runs while the current release keeps serving. Only the symlink swap and `systemctl restart` cause downtime, about 3 seconds. Rollback is the same swap in reverse and needs no rebuild.
The script needs passwordless sudo on the LXC, because the deployment lives under `/root`. On the container as root:
The build deliberately does **not** source the env file — same as the Dockerfile — because of the build-time rewrite baking described above.
```bash
echo 'drinkadmin ALL=(root) NOPASSWD: ALL' > /etc/sudoers.d/drinkadmin-deploy
chmod 440 /etc/sudoers.d/drinkadmin-deploy
visudo -c
```
## Docker still works, just not here
This grants less than it appears to: `drinkadmin` is already in the `docker` group, and anyone who can talk to the Docker socket can mount the host filesystem into a privileged container, so they are root-equivalent already. If you'd rather not, drop the sudoers file and run the deploy steps by hand from a root shell.
`Dockerfile` and `docker-compose.prod.yml` are maintained for running this app elsewhere (a VPS). `docker compose build && docker compose up -d` works on any normal Docker host.
**Alternative worth considering:** move the deployment out of `/root` into a `drinkadmin`-owned directory and skip sudo entirely. Keep the directory basename `drinktracker` so the compose project name — and therefore the existing container and volume names — stay the same. `.env.production` must be copied across intact or Postgres will reject the app's credentials, since the volume was initialised with the current password.
It does **not** work on this LXC, which is why the app runs natively. Two independent problems, both from unprivileged-LXC nesting:
1. AppArmor — the container can't load the `docker-default` profile (`apparmor_parser: Access denied`), so any container without `--security-opt apparmor=unconfined` fails.
2. runc can't set `net.ipv4.ip_unprivileged_port_start`, so containers only work with `--network host`.
`docker build` is impossible regardless: Docker 29 removed the classic builder and BuildKit spawns a builder container without the unconfined option. Docker is installed but disabled; the old containers and volumes are retained until 2026-09-07 as a rollback path.
## Verifying by hand
```bash
ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169
sudo docker compose -f /root/drinktracker/docker-compose.prod.yml ps
sudo docker logs drinktracker-app-1 --tail 50
sudo docker logs drinktracker-app-1 2>&1 | grep '\[switchboard\]' # AI routing + cost per call
systemctl is-active postgresql@16-main minio drinktracker
sudo journalctl -u drinktracker -n 50
sudo journalctl -u drinktracker | grep '\[switchboard\]' # AI routing + cost per call
readlink /opt/drinktracker/current # which release is live
```
Each AI call logs one `[switchboard] feature=… model=… cost=… latency_ms=…` line. `FAILOVER` warnings are expected — the gateway's `:batch` model variants currently fail on every request and fall back.
Each AI call logs one `[switchboard] feature=… model=… cost=… latency_ms=…` line. `FAILOVER` warnings are expected — the gateway's `:batch` model variants currently fail on every request and fall back. `CONTEXT OVERFLOW` is worth investigating.
## Rollback
## Backups
`drinktracker-backup.timer` runs nightly at 03:25: `pg_dump` + a tar of `/var/lib/minio/data` + a copy of the env file, into `/var/backups/drinktracker`, retained 14 days (~1 GB).
**TODO — the chain has no off-site leg.** `/usr/local/bin/drinktracker-backup` supports `OFFSITE_DEST` (an scp target) and `OFFSITE_KEY`; set them via `Environment=` in `drinktracker-backup.service`. Until then everything lives on one LXC on one Proxmox host, and the nightly job prints a warning saying so.
Restore:
```bash
git revert <bad-sha> && ./deploy/deploy.sh
sudo systemctl stop drinktracker
zcat /var/backups/drinktracker/pg-<stamp>.sql.gz | sudo -u postgres psql -d drinkman
sudo systemctl stop minio
sudo tar -C /var/lib/minio -xzf /var/backups/drinktracker/minio-<stamp>.tgz
sudo chown -R minio:minio /var/lib/minio/data
sudo systemctl start minio drinktracker
```
Or directly on the LXC, without touching git history:
## Notes
```bash
sudo git -C /root/drinktracker reset --hard <previous-sha>
cd /root/drinktracker
sudo docker build -t 192.168.2.140:3000/jpscott84/drinktracker:latest .
sudo docker compose -f docker-compose.prod.yml up -d --no-build
```
Note the `migrate` service runs `prisma db push --accept-data-loss` on every start, so a rollback that removes a column will drop it and its data. Take a database dump first if the schema changed.
- The deploy needs unattended sudo on the LXC (`/etc/sudoers.d/drinkadmin-deploy`, currently
`NOPASSWD: ALL`). A narrowly-scoped allowlist was considered and rejected: the script calls
root for `install`, `tee`, `systemctl`, `journalctl` and `sudo -u postgres pg_dump`, so an
allowlist would need updating every time the script changes and would fail in confusing ways
when it drifted. Be aware this buys little security either way — `drinkadmin` is in the `sudo`
group, so full root is a password away regardless. What it grants is *unattended* root.
- `nodejs` is `apt-mark hold`-ed at v20.20.0 so the native runtime can't drift from the Dockerfile's pinned `node:20-alpine`.
- Postgres was created with `en_US.UTF-8`; the Ubuntu default of `C`/`SQL_ASCII` would re-sort every drink list and mangle names like `Grüner Veltliner`. Preserve the locale if the cluster is ever rebuilt.
- `unattended-upgrades` is not installed, so nothing will surprise-restart these services — but postgres will never auto-patch either. A deliberate open question.