Document the off-site backup leg now that it exists

Nightly backups now copy to TrueNAS over SSH rather than living only on the
LXC they are protecting. Chose scp over the existing NFS share deliberately:
a mount that goes stale writes into the local mountpoint instead, silently
filling the LXC disk while every run reports success. No mount, no such class
of failure - and the backup script already spoke scp, so nothing changed but
two Environment lines.

Set as a systemd drop-in so re-installing the service cannot quietly return to
local-only backups.

Recorded the failure mode that cost time here: sshd refuses authorized_keys
when the user's home directory is group- or world-writable, which TrueNAS
datasets often are. The key is offered and silently rejected, which looks
exactly like a key that was never added.

Verified end to end rather than assumed: the copy lands, the dump passes
gzip -t, contains 25 CREATE TABLE statements including the new MCP and OAuth
tables, and the MinIO tarball reads back with 194 entries.

Still open: the dumps are unencrypted, and TrueNAS is on the same site.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W1Ee4Mc1X1SX8HgYa52zu7

deploy/README.md

@@ -85,7 +85,34 @@ The second must return **401** with a `WWW-Authenticate: Bearer … resource_met
 
 `drinktracker-backup.timer` runs nightly at 03:25: `pg_dump` + a tar of `/var/lib/minio/data` + a copy of the env file, into `/var/backups/drinktracker`, retained 14 days (~1 GB).
 
-**TODO — the chain has no off-site leg.** `/usr/local/bin/drinktracker-backup` supports `OFFSITE_DEST` (an scp target) and `OFFSITE_KEY`; set them via `Environment=` in `drinktracker-backup.service`. Until then everything lives on one LXC on one Proxmox host, and the nightly job prints a warning saying so.
+Each run also copies the dump and the MinIO tarball off-site to TrueNAS:
+
+| | |
+|---|---|
+| Destination | `jpscott84@192.168.2.196:/mnt/Beefcake/backup/drinktracker` (dataset `Beefcake/backup`) |
+| Transport | `scp` over SSH — **not** NFS, so there is no mount to go stale and silently swallow a night's backup into the LXC's own disk |
+| Key | `/root/.ssh/drinktracker_backup_ed25519`, a dedicated backup-only keypair |
+| Config | `Environment=` lines in `/etc/systemd/system/drinktracker-backup.service.d/offsite.conf` |
+
+Set as a **drop-in** rather than an edit to the unit, so re-installing the service cannot silently drop the off-site leg and return to local-only backups.
+
+A successful run logs `offsite copy ok -> …`. If it ever logs `NOTE: OFFSITE_DEST unset` again, the drop-in has gone missing. A failed copy logs `WARNING: offsite copy FAILED` and **keeps the local copy** — the job still exits 0, so watch the log rather than the exit status.
+
+```bash
+sudo systemctl start drinktracker-backup.service       # run it now
+sudo journalctl -u drinktracker-backup -n 20           # look for "offsite copy ok"
+```
+
+Verify what landed is actually restorable, not merely present:
+
+```bash
+sudo ssh -i /root/.ssh/drinktracker_backup_ed25519 jpscott84@192.168.2.196 \
+  'ls -lh /mnt/Beefcake/backup/drinktracker && gzip -t /mnt/Beefcake/backup/drinktracker/pg-*.sql.gz && echo OK'
+```
+
+**Two gaps that remain.** The dumps are **unencrypted** and contain every user's data plus bcrypt password hashes — acceptable writing to a NAS on the same LAN, but encrypt before this ever leaves the network. And TrueNAS is in the same building, so this survives losing the LXC or the Proxmox host, not fire or theft.
+
+> If SSH key auth to TrueNAS ever breaks: sshd refuses `authorized_keys` when the user's **home directory** is group- or world-writable, which TrueNAS datasets often are by default. `/mnt/local/jpscott84` must not be `777`. The symptom is indistinguishable from a missing key — the key is offered and silently rejected — so check `journalctl -u ssh | grep 'bad ownership'` on the NAS first.
 
 Restore: