systems

NixOS configurations for all hosts.

TODO: server-side clock in?

TODO: in @lib/overlays.nix , the npmdepshash has to be updated, and it causes a build failure when we fail to do it. i feel like a nicer way to do this would be to load it from a file that lives beside the respective package.json, with an xtask to update it (xtask artifacts update-npm-deps-hash etc), and a commit hook that checks it when a package.lock is changed.

Tailscale / Headscale

All hosts connect via Headscale (self-hosted Tailscale control plane) running on samarium at https://hs.finiteleibniz.com.

MagicDNS is enabled: hosts are reachable as <hostname>.ts.internal. Internal services on erbium (prometheus, perses, vlogs, ksd) are accessible at <service>.internal.finiteleibniz.com via headscale extra_records + nginx with ACME certs.

Enrolling a new node

1. Add ../../modules/nixos/tailscale.nix to the host's imports and deploy.

2. Generate a preauth key:

   ssh samarium headscale preauthkeys create --user s
   

3. On the new node:

   tailscale up --login-server=https://hs.finiteleibniz.com --authkey=<key>
   

4. Verify:

   ssh samarium headscale nodes list
   

5. If the node should be scraped by Prometheus, add it to the peers list in modules/nixos/monitoring.nix and import modules/nixos/node-exporter.nix.

6. Update tsIPs in modules/nixos/headscale.nix if the assigned IP doesn't match.

Administration

ssh samarium headscale nodes list          # list all nodes
ssh samarium headscale nodes rename <id> <name>  # rename a node
ssh samarium headscale preauthkeys list --user s  # list preauth keys

Secrets

Secrets are managed with sops-nix using age encryption. Encrypted secrets live in-repo under each host (e.g. hosts/erbium/secrets.yaml) and are decrypted on the target machine at activation time.

Editing secrets

sops hosts/erbium/secrets.yaml

This prompts for your age key passphrase, then opens $EDITOR.

Adding a new device

1. Generate a passphrase-protected age key:

   age-keygen | age -p > ~/.config/sops/age/keys.txt
   chmod 600 ~/.config/sops/age/keys.txt
   

   This prints the public key to stderr. Copy it.

2. Add the public key to .sops.yaml:

   keys:
     - &mydevice age1...
   

   And add it to the relevant creation_rules key groups.

3. Re-encrypt all affected secrets so the new key can decrypt them. This requires a device that can already decrypt:

   sops updatekeys hosts/erbium/secrets.yaml
   

Adding a new host

Hosts decrypt secrets using their SSH host key, converted to age. To get a host's age public key:

ssh-keyscan -t ed25519 <host> | ssh-to-age

Add this to .sops.yaml alongside the personal keys.

Org-mode deadline management (erbium)

Org files live in a Forgejo repo. A pair of services on erbium parse deadlines out of them and make them useful outside of Emacs.

How it works now

Forgejo repo (*.org files)
  │
  │  git pull (every 6h + webhook on push)
  ▼
sync_org_deadlines.py ──▶ SQLite (deadlines table)
  │
  │  every 5 min
  ▼
notify_deadlines.py
  ├──▶ main.ics (served at calendar.tensegrist.com/<token>/main.ics)
  └──▶ ntfy push notifications (escalating: 1d → 1h → 5m before deadline)

Components:

• sync_org_deadlines.py — clones/pulls the org repo, parses DEADLINE and SCHEDULED timestamps with orgparse, upserts into SQLite, prunes removed items
• notify_deadlines.py — reads SQLite, sends ntfy notifications at three urgency tiers (skips DONE/CANCELLED), generates an ICS feed with VALARM reminders (Europe/Paris timezone), prunes old alert records
• webhook_sync.py — tiny Flask app that receives Forgejo webhook POSTs, verifies HMAC signature, touches a trigger file that a systemd path unit watches to start a sync
• hosts/erbium/org-deadlines.nix — systemd services/timers, nginx vhost, sops secrets, user/group setup
• hosts/erbium/ntfy.nix — self-hosted ntfy server at ntfy.tensegrist.com

Setup: see comments at the top of hosts/erbium/org-deadlines.nix.

Systemd units:

• sync-org-deadlines.service / .timer — pulls org repo + updates SQLite (every 6h)
• notify-org-deadlines.service / .timer — sends notifications + generates ICS (every 5min)
• webhook-org-deadlines.service — Flask webhook listener on localhost:8783
• sync-org-deadlines-trigger.path — watches trigger file, starts sync on webhook
• org-deadlines-serve-setup.service — creates serve directory + symlink from sops token on boot

Check all statuses:

systemctl status sync-org-deadlines.{service,timer} notify-org-deadlines.{service,timer} webhook-org-deadlines.service org-deadlines-serve-setup.service sync-org-deadlines-trigger.path

Permissions model:

• All services run as the org-deadlines system user
• nginx is added to the org-deadlines group to read ICS files from the serve directory
• The data directory (/var/lib/org-deadlines) is 0750 — owner full access, group traverse+read
• ICS files are explicitly chmod 0640 so only owner and group (nginx) can read them
• Secrets are decrypted to /run/secrets/ by sops-nix, each scoped to its service (sync gets repo URL, notify gets ntfy credentials, webhook gets HMAC secret, ntfy gets its user password)
• The ICS feed URL contains a random token as a path component for unguessable access

ntfy push notifications (erbium)

A self-hosted ntfy instance at ntfy.tensegrist.com, used by the deadline notification system.

• Config: hosts/erbium/ntfy.nix
• Topic: org-main — receives deadline alerts from notify_deadlines.py
• Auth: anonymous access is denied (auth-default-access: deny-all). A ntfy admin user is provisioned declaratively via NTFY_AUTH_USERS — the bcrypt hash is generated from the sops password at boot by ntfy-sh-auth.service.
• Subscribe: in the ntfy app, add https://ntfy.tensegrist.com/org-main with username ntfy and the password from ntfy-password in secrets.

Systemd units:

• ntfy-sh.service — the ntfy server (localhost:2586, behind nginx)
• ntfy-sh-auth.service — hashes the sops password into an env file for ntfy-sh (runs before ntfy-sh)

Monitoring & alerting (erbium)

Prometheus, Alertmanager, and Perses run on erbium behind the tailscale interface.

• Config: modules/nixos/monitoring.nix
• Alertmanager sends notifications to Telegram. The bot token and chat ID are stored as sops secrets (alertmanager-telegram-token, alertmanager-telegram-chat-id in hosts/erbium/secrets.yaml), decrypted at activation to /run/secrets/ and owned by the alertmanager user.
• Alerts are grouped by alertname. Timing uses alertmanager defaults: 30s group wait, 5m group interval, 4h repeat.
• Perses dashboard at perses.internal.finiteleibniz.com (tailscale only).

Testing alertmanager:

curl -XPOST http://127.0.0.1:9093/api/v2/alerts \
  -H 'Content-Type: application/json' \
  -d '[{"labels":{"alertname":"TestAlert","severity":"info"},"annotations":{"summary":"test"}}]'

Should deliver to Telegram within 30s.

Testing

nix build .#checks.x86_64-linux.ksd --print-build-logs

Future: bidirectional calendar sync

Two possible directions for letting calendar clients create events that flow back into org files.

Option A: Radicale CalDAV (separate calendar for capture)

Keep the ICS feed for deadlines (read-only). Add a Radicale CalDAV server for a second, writable calendar used to capture new events.

• Radicale runs on erbium, behind nginx at e.g. calendar.tensegrist.com/dav/
• Calendar clients subscribe to both: the ICS feed (read-only deadlines) and the CalDAV calendar (read-write inbox)
• A cron job polls Radicale's file store for new events and appends them as * TODO headings with DEADLINE: or SCHEDULED: to refile.org in the Forgejo repo, then commits and pushes
• No conflict resolution needed — each direction is one-way. Org files are authoritative for deadlines; Radicale is a write-only inbox that gets drained into org.

Pros: simple, no conflicts, calendar clients get full CalDAV support for the inbox calendar. Cons: two calendars in the client, new events don't appear as org items until the drain cron runs + the next sync.

Option B: Full CalDAV with org as backend

Replace the ICS feed entirely with a CalDAV server that reads/writes org files directly.

• A CalDAV server (Radicale or custom) uses org files as its storage backend
• Reads: deadlines/scheduled items from org → CalDAV events (like the current ICS generation, but served via CalDAV protocol)
• Writes: new/modified CalDAV events → appended or patched in org files, committed and pushed to Forgejo
• Deletes: mark the org heading as CANCELLED (or actually remove it, configurable)

Pros: single calendar, true bidirectional sync, feels native. Cons: complex conflict resolution (what if an org heading is edited and the CalDAV event is modified simultaneously?), CalDAV clients may set properties that don't map to org-mode, significant implementation effort. Likely needs a custom CalDAV backend rather than stock Radicale.

Although org-caldav is fine, and i'm fine with the incoming sync overwriting anything in the inbox.org lol the inbox is strictly client write-only