1710e6147b
Nightly backups now copy to TrueNAS over SSH rather than living only on the LXC they are protecting. Chose scp over the existing NFS share deliberately: a mount that goes stale writes into the local mountpoint instead, silently filling the LXC disk while every run reports success. No mount, no such class of failure - and the backup script already spoke scp, so nothing changed but two Environment lines. Set as a systemd drop-in so re-installing the service cannot quietly return to local-only backups. Recorded the failure mode that cost time here: sshd refuses authorized_keys when the user's home directory is group- or world-writable, which TrueNAS datasets often are. The key is offered and silently rejected, which looks exactly like a key that was never added. Verified end to end rather than assumed: the copy lands, the dump passes gzip -t, contains 25 CREATE TABLE statements including the new MCP and OAuth tables, and the MinIO tarball reads back with 194 entries. Still open: the dumps are unencrypted, and TrueNAS is on the same site. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1Ee4Mc1X1SX8HgYa52zu7
139 lines
10 KiB
Markdown
139 lines
10 KiB
Markdown
# Deploying drinktracker
|
|
|
|
```bash
|
|
./deploy/deploy.sh # typecheck, push, build, swap, restart, verify
|
|
./deploy/deploy.sh --yes # skip the confirmation prompt
|
|
./deploy/deploy.sh --rollback # repoint at the previous release (~3s, no rebuild)
|
|
```
|
|
|
|
The app runs **natively under systemd — there is no Docker on this host.**
|
|
|
|
## Where things live
|
|
|
|
| | |
|
|
|---|---|
|
|
| Production LXC | `192.168.2.169` (hostname `drinktracker`, Proxmox vmid 100) |
|
|
| Deploy access | `ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169` |
|
|
| App | `/opt/drinktracker/{repo,releases/<sha>,current}`, service user `drinktracker` |
|
|
| Secrets | `/etc/drinktracker/drinktracker.env` (`root:drinktracker` `0640`) |
|
|
| Services | `drinktracker`, `minio`, `postgresql@16-main` |
|
|
| Database | `drinkman`, user `drinkdb`, `127.0.0.1:5432` |
|
|
| Object storage | MinIO at `127.0.0.1:9000`, bucket `drink-images` |
|
|
| App port | 3000 on `0.0.0.0`; unauthenticated requests return 307 → `/login` |
|
|
| Backups | `/var/backups/drinktracker`, nightly 03:25, 14-day retention |
|
|
|
|
## Traps
|
|
|
|
**`drinktracker.tenseconddelay.net` resolves to the reverse proxy (192.168.2.172), not the container.** `NEXTAUTH_URL` points at the public name, so resolving it and SSHing there fails in a way that looks like a rejected key but is simply a different machine.
|
|
|
|
**Images are served by an authenticated route, not a rewrite.** `next.config.mjs` has no `rewrites()` block at all. `/minio-images/:path*` is a route handler (`src/app/minio-images/[...key]/route.ts`) that requires a session, rejects `..` in the key, and enforces per-user ownership by key prefix (`<userId>/` or `scans/<userId>/`). It reads `MINIO_ENDPOINT` at request time, so changing it takes effect on restart with no rebuild.
|
|
|
|
> This replaced an unauthenticated rewrite that, combined with an anonymous-download bucket policy, exposed every stored image to the public internet. **The bucket must not have an anonymous policy, and MinIO must stay bound to loopback** (`127.0.0.1:9000`, verify with `ss -lntp | grep 9000`). `mc anonymous get local/drink-images` is not a usable check here — the `local` alias on this host has no working credentials and returns `Access Denied` regardless. Check from outside instead: `curl -si https://drinktracker.tenseconddelay.net/minio-images?list-type=2` must **not** return a `200` with an XML key listing.
|
|
|
|
**Never use `next/image` for stored images.** The optimizer fetches them server-side without the session cookie, so every image would 401. All renders are plain `<img>`, and there is deliberately no `images.remotePatterns` config.
|
|
|
|
**`NEXTAUTH_URL` must exactly equal the public origin, with no trailing slash.** Beyond login redirects it now drives the OAuth `issuer` and the MCP `resource` identifier, both of which clients compare byte-for-byte. A mismatch does not error — connectors just fail to complete with an unhelpful "couldn't reach the server".
|
|
|
|
**`prisma db push`, not `migrate deploy`.** Production has no `_prisma_migrations` table, so `migrate deploy` would try to apply the init migration against populated tables and fail. `--accept-data-loss` means removing a field from `schema.prisma` **drops the column and its data** — the deploy script dumps first for this reason.
|
|
|
|
## How a deploy works
|
|
|
|
Build happens in `/opt/drinktracker/repo` and is assembled **out-of-place** into `releases/<sha>`, so the ~3 minute build runs while the current release keeps serving. Only the symlink swap and `systemctl restart` cause downtime, about 3 seconds. Rollback is the same swap in reverse and needs no rebuild.
|
|
|
|
The build deliberately does **not** source the env file, same as the Dockerfile. Nothing in this app needs configuration baked into the bundle: `NEXTAUTH_URL`, `MINIO_ENDPOINT` and the rest are read at request time by dynamic routes, so the build stays identical regardless of which host it runs on. (This originally guarded against a `rewrites()` block that baked a MinIO URL into `routes-manifest.json`; that block is gone, but building without secrets is still the right default.)
|
|
|
|
## Docker still works, just not here
|
|
|
|
`Dockerfile` and `docker-compose.prod.yml` are maintained for running this app elsewhere (a VPS). `docker compose build && docker compose up -d` works on any normal Docker host.
|
|
|
|
It does **not** work on this LXC, which is why the app runs natively. Two independent problems, both from unprivileged-LXC nesting:
|
|
|
|
1. AppArmor — the container can't load the `docker-default` profile (`apparmor_parser: Access denied`), so any container without `--security-opt apparmor=unconfined` fails.
|
|
2. runc can't set `net.ipv4.ip_unprivileged_port_start`, so containers only work with `--network host`.
|
|
|
|
`docker build` is impossible regardless: Docker 29 removed the classic builder and BuildKit spawns a builder container without the unconfined option. Docker is installed but disabled; the old containers and volumes are retained until 2026-09-07 as a rollback path.
|
|
|
|
## Verifying by hand
|
|
|
|
```bash
|
|
ssh -i ~/.ssh/drinktracker_ed25519 drinkadmin@192.168.2.169
|
|
systemctl is-active postgresql@16-main minio drinktracker
|
|
sudo journalctl -u drinktracker -n 50
|
|
sudo journalctl -u drinktracker | grep '\[switchboard\]' # AI routing + cost per call
|
|
sudo journalctl -u drinktracker | grep '\[mcp\]' # one line per MCP tool call
|
|
sudo journalctl -u drinktracker | grep '\[oauth\]' # connector grants and token issuance
|
|
readlink /opt/drinktracker/current # which release is live
|
|
```
|
|
|
|
Each AI call logs one `[switchboard] feature=… model=… cost=… latency_ms=…` line. `FAILOVER` warnings are expected — the gateway's `:batch` model variants currently fail on every request and fall back. `CONTEXT OVERFLOW` is worth investigating.
|
|
|
|
MCP tool calls log `[mcp] user=… tool=… ok=… ms=…`, and the same events are rows in `McpAuditLog` (arguments are deliberately never recorded). `[oauth]` lines cover consent grants and token issuance; a `PKCE mismatch` or `refresh token replay` warning there means a grant was revoked defensively and is worth looking at.
|
|
|
|
Discovery can be checked without credentials:
|
|
|
|
```bash
|
|
curl -s https://drinktracker.tenseconddelay.net/.well-known/oauth-protected-resource | jq
|
|
curl -si https://drinktracker.tenseconddelay.net/api/mcp -X POST \
|
|
-H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
|
|
-H 'MCP-Protocol-Version: 2026-07-28' -H 'Mcp-Method: tools/list' \
|
|
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{"_meta":{"io.modelcontextprotocol/protocolVersion":"2026-07-28","io.modelcontextprotocol/clientCapabilities":{}}}}'
|
|
```
|
|
|
|
The second must return **401** with a `WWW-Authenticate: Bearer … resource_metadata="…"` header. A `307` there means `/.well-known` fell out of `PUBLIC_ROUTES` in `src/lib/auth.ts` and discovery is broken.
|
|
|
|
## Backups
|
|
|
|
`drinktracker-backup.timer` runs nightly at 03:25: `pg_dump` + a tar of `/var/lib/minio/data` + a copy of the env file, into `/var/backups/drinktracker`, retained 14 days (~1 GB).
|
|
|
|
Each run also copies the dump and the MinIO tarball off-site to TrueNAS:
|
|
|
|
| | |
|
|
|---|---|
|
|
| Destination | `jpscott84@192.168.2.196:/mnt/Beefcake/backup/drinktracker` (dataset `Beefcake/backup`) |
|
|
| Transport | `scp` over SSH — **not** NFS, so there is no mount to go stale and silently swallow a night's backup into the LXC's own disk |
|
|
| Key | `/root/.ssh/drinktracker_backup_ed25519`, a dedicated backup-only keypair |
|
|
| Config | `Environment=` lines in `/etc/systemd/system/drinktracker-backup.service.d/offsite.conf` |
|
|
|
|
Set as a **drop-in** rather than an edit to the unit, so re-installing the service cannot silently drop the off-site leg and return to local-only backups.
|
|
|
|
A successful run logs `offsite copy ok -> …`. If it ever logs `NOTE: OFFSITE_DEST unset` again, the drop-in has gone missing. A failed copy logs `WARNING: offsite copy FAILED` and **keeps the local copy** — the job still exits 0, so watch the log rather than the exit status.
|
|
|
|
```bash
|
|
sudo systemctl start drinktracker-backup.service # run it now
|
|
sudo journalctl -u drinktracker-backup -n 20 # look for "offsite copy ok"
|
|
```
|
|
|
|
Verify what landed is actually restorable, not merely present:
|
|
|
|
```bash
|
|
sudo ssh -i /root/.ssh/drinktracker_backup_ed25519 jpscott84@192.168.2.196 \
|
|
'ls -lh /mnt/Beefcake/backup/drinktracker && gzip -t /mnt/Beefcake/backup/drinktracker/pg-*.sql.gz && echo OK'
|
|
```
|
|
|
|
**Two gaps that remain.** The dumps are **unencrypted** and contain every user's data plus bcrypt password hashes — acceptable writing to a NAS on the same LAN, but encrypt before this ever leaves the network. And TrueNAS is in the same building, so this survives losing the LXC or the Proxmox host, not fire or theft.
|
|
|
|
> If SSH key auth to TrueNAS ever breaks: sshd refuses `authorized_keys` when the user's **home directory** is group- or world-writable, which TrueNAS datasets often are by default. `/mnt/local/jpscott84` must not be `777`. The symptom is indistinguishable from a missing key — the key is offered and silently rejected — so check `journalctl -u ssh | grep 'bad ownership'` on the NAS first.
|
|
|
|
Restore:
|
|
|
|
```bash
|
|
sudo systemctl stop drinktracker
|
|
zcat /var/backups/drinktracker/pg-<stamp>.sql.gz | sudo -u postgres psql -d drinkman
|
|
sudo systemctl stop minio
|
|
sudo tar -C /var/lib/minio -xzf /var/backups/drinktracker/minio-<stamp>.tgz
|
|
sudo chown -R minio:minio /var/lib/minio/data
|
|
sudo systemctl start minio drinktracker
|
|
```
|
|
|
|
## Notes
|
|
|
|
- The deploy needs unattended sudo on the LXC (`/etc/sudoers.d/drinkadmin-deploy`, currently
|
|
`NOPASSWD: ALL`). A narrowly-scoped allowlist was considered and rejected: the script calls
|
|
root for `install`, `tee`, `systemctl`, `journalctl` and `sudo -u postgres pg_dump`, so an
|
|
allowlist would need updating every time the script changes and would fail in confusing ways
|
|
when it drifted. Be aware this buys little security either way — `drinkadmin` is in the `sudo`
|
|
group, so full root is a password away regardless. What it grants is *unattended* root.
|
|
- `nodejs` is `apt-mark hold`-ed at v20.20.0 so the native runtime can't drift from the Dockerfile's pinned `node:20-alpine`.
|
|
- Postgres was created with `en_US.UTF-8`; the Ubuntu default of `C`/`SQL_ASCII` would re-sort every drink list and mangle names like `Grüner Veltliner`. Preserve the locale if the cluster is ever rebuilt.
|
|
- `unattended-upgrades` is not installed, so nothing will surprise-restart these services — but postgres will never auto-patch either. A deliberate open question.
|