87f951bf94
Docker in this unprivileged Proxmox LXC was broken two independent ways: AppArmor could not load the docker-default profile, so the daemon could not start any new container, and runc could not set the net.ipv4.ip_unprivileged_port_start sysctl, which is why every service needed network_mode: host. `docker build` was impossible outright since Docker 29 removed the classic builder. The stack only survived because the containers predated the breakage - a reboot would have left the app down, and nothing could be redeployed. Postgres, MinIO and the app now run natively under systemd. Deploys build out-of-place into releases/<sha> and swap a symlink, so the build happens while the old release keeps serving and downtime is the ~3s restart rather than the ~3min build. Rollback is the same swap in reverse with no rebuild. Dockerfile and docker-compose.prod.yml are unchanged and still work on a normal host; the compose app service gains `build: .` so it can come up on a VPS that cannot reach the LAN-only Gitea registry. Documents three traps found during the migration: rewrites() in next.config.mjs is evaluated at build time so MINIO_ENDPOINT changes silently do nothing, prisma migrate deploy would fail because production has no _prisma_migrations table, and the image proxy relies on the bucket's anonymous-download policy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
92 lines
6.1 KiB
Markdown
92 lines
6.1 KiB
Markdown
# Deploying drinktracker
|
|
|
|
```bash
|
|
./deploy/deploy.sh # typecheck, push, build, swap, restart, verify
|
|
./deploy/deploy.sh --yes # skip the confirmation prompt
|
|
./deploy/deploy.sh --rollback # repoint at the previous release (~3s, no rebuild)
|
|
```
|
|
|
|
The app runs **natively under systemd — there is no Docker on this host.**
|
|
|
|
## Where things live
|
|
|
|
| | |
|
|
|---|---|
|
|
| Production LXC | `192.168.2.169` (hostname `drinktracker`, Proxmox vmid 100) |
|
|
| Deploy access | `ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169` |
|
|
| App | `/opt/drinktracker/{repo,releases/<sha>,current}`, service user `drinktracker` |
|
|
| Secrets | `/etc/drinktracker/drinktracker.env` (`root:drinktracker` `0640`) |
|
|
| Services | `drinktracker`, `minio`, `postgresql@16-main` |
|
|
| Database | `drinkman`, user `drinkdb`, `127.0.0.1:5432` |
|
|
| Object storage | MinIO at `127.0.0.1:9000`, bucket `drink-images` |
|
|
| App port | 3000 on `0.0.0.0`; unauthenticated requests return 307 → `/login` |
|
|
| Backups | `/var/backups/drinktracker`, nightly 03:25, 14-day retention |
|
|
|
|
## Traps
|
|
|
|
**`drinktracker.tenseconddelay.net` resolves to the reverse proxy (192.168.2.172), not the container.** `NEXTAUTH_URL` points at the public name, so resolving it and SSHing there fails in a way that looks like a rejected key but is simply a different machine.
|
|
|
|
**`next.config.mjs` evaluates `rewrites()` at BUILD time.** The `/minio-images/:path*` destination is baked into `routes-manifest.json` as `http://localhost:9000/drink-images/:path*`. Changing `MINIO_ENDPOINT` in the env file appears to work and does nothing — images break only after the *next* rebuild, so the change and the breakage are separated in time. MinIO must stay on `localhost:9000` serving `drink-images`.
|
|
|
|
**The image proxy is an unauthenticated GET.** Next rewrites don't sign requests, so the bucket's anonymous-download policy is load-bearing. Verify with `mc anonymous get local/drink-images` → must say `download`.
|
|
|
|
**`prisma db push`, not `migrate deploy`.** Production has no `_prisma_migrations` table, so `migrate deploy` would try to apply the init migration against populated tables and fail. `--accept-data-loss` means removing a field from `schema.prisma` **drops the column and its data** — the deploy script dumps first for this reason.
|
|
|
|
## How a deploy works
|
|
|
|
Build happens in `/opt/drinktracker/repo` and is assembled **out-of-place** into `releases/<sha>`, so the ~3 minute build runs while the current release keeps serving. Only the symlink swap and `systemctl restart` cause downtime, about 3 seconds. Rollback is the same swap in reverse and needs no rebuild.
|
|
|
|
The build deliberately does **not** source the env file — same as the Dockerfile — because of the build-time rewrite baking described above.
|
|
|
|
## Docker still works, just not here
|
|
|
|
`Dockerfile` and `docker-compose.prod.yml` are maintained for running this app elsewhere (a VPS). `docker compose build && docker compose up -d` works on any normal Docker host.
|
|
|
|
It does **not** work on this LXC, which is why the app runs natively. Two independent problems, both from unprivileged-LXC nesting:
|
|
|
|
1. AppArmor — the container can't load the `docker-default` profile (`apparmor_parser: Access denied`), so any container without `--security-opt apparmor=unconfined` fails.
|
|
2. runc can't set `net.ipv4.ip_unprivileged_port_start`, so containers only work with `--network host`.
|
|
|
|
`docker build` is impossible regardless: Docker 29 removed the classic builder and BuildKit spawns a builder container without the unconfined option. Docker is installed but disabled; the old containers and volumes are retained until 2026-09-07 as a rollback path.
|
|
|
|
## Verifying by hand
|
|
|
|
```bash
|
|
ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169
|
|
systemctl is-active postgresql@16-main minio drinktracker
|
|
sudo journalctl -u drinktracker -n 50
|
|
sudo journalctl -u drinktracker | grep '\[switchboard\]' # AI routing + cost per call
|
|
readlink /opt/drinktracker/current # which release is live
|
|
```
|
|
|
|
Each AI call logs one `[switchboard] feature=… model=… cost=… latency_ms=…` line. `FAILOVER` warnings are expected — the gateway's `:batch` model variants currently fail on every request and fall back. `CONTEXT OVERFLOW` is worth investigating.
|
|
|
|
## Backups
|
|
|
|
`drinktracker-backup.timer` runs nightly at 03:25: `pg_dump` + a tar of `/var/lib/minio/data` + a copy of the env file, into `/var/backups/drinktracker`, retained 14 days (~1 GB).
|
|
|
|
**TODO — the chain has no off-site leg.** `/usr/local/bin/drinktracker-backup` supports `OFFSITE_DEST` (an scp target) and `OFFSITE_KEY`; set them via `Environment=` in `drinktracker-backup.service`. Until then everything lives on one LXC on one Proxmox host, and the nightly job prints a warning saying so.
|
|
|
|
Restore:
|
|
|
|
```bash
|
|
sudo systemctl stop drinktracker
|
|
zcat /var/backups/drinktracker/pg-<stamp>.sql.gz | sudo -u postgres psql -d drinkman
|
|
sudo systemctl stop minio
|
|
sudo tar -C /var/lib/minio -xzf /var/backups/drinktracker/minio-<stamp>.tgz
|
|
sudo chown -R minio:minio /var/lib/minio/data
|
|
sudo systemctl start minio drinktracker
|
|
```
|
|
|
|
## Notes
|
|
|
|
- The deploy needs unattended sudo on the LXC (`/etc/sudoers.d/drinkadmin-deploy`, currently
|
|
`NOPASSWD: ALL`). A narrowly-scoped allowlist was considered and rejected: the script calls
|
|
root for `install`, `tee`, `systemctl`, `journalctl` and `sudo -u postgres pg_dump`, so an
|
|
allowlist would need updating every time the script changes and would fail in confusing ways
|
|
when it drifted. Be aware this buys little security either way — `drinkadmin` is in the `sudo`
|
|
group, so full root is a password away regardless. What it grants is *unattended* root.
|
|
- `nodejs` is `apt-mark hold`-ed at v20.20.0 so the native runtime can't drift from the Dockerfile's pinned `node:20-alpine`.
|
|
- Postgres was created with `en_US.UTF-8`; the Ubuntu default of `C`/`SQL_ASCII` would re-sort every drink list and mangle names like `Grüner Veltliner`. Preserve the locale if the cluster is ever rebuilt.
|
|
- `unattended-upgrades` is not installed, so nothing will surprise-restart these services — but postgres will never auto-patch either. A deliberate open question.
|