Skip to content

Runbook — first live privsep ENFORCE cutover (Thinkpad, Linux)

Status: first live enforce deployment (epic #83, the "one host → observe → next" step). Linux/cap-only path (proven on the Dell). Run on the Thinkpad. Reversible — keep the old service + binary until the wall-bite check (step 7) passes.

What changes (read first)

The Thinkpad runs ductile today as a --user LaunchAgent/systemd-user service as your own uid (hygiene-only, unconfined). Enforce requires a system service running as the ductile account with CAP_SETUID+CAP_SETGID, dropping each plugin to an account uid. That means the config, state, vault, and age key move to ductile-owned system paths — the ductile user (uid 984) cannot read files under your $HOME (0700). New topology (from deploy/systemd/):

today (--user) enforce (system)
runs as your uid ductile (984) + CAP_SETUID/SETGID
config ~/.config/ductile (or ~/admin/ductile-local) /etc/ductile (ductile-owned 0700)
state under the above /var/lib/ductile
age key ~/.config/secrets/… inside /etc/ductile, ductile-owned 0600
plugins grants → ungranted = your uid sys_exec → untrusted; others → default (uid 1001)

The boot fs-reconcile tightens the config dir to 0700 ductile-owned (accounts can't read it) and verifies each account's 0700 state_dir — fail-closed: a misconfig refuses to boot, it does not half-confine.


0. Recon (on the Thinkpad — paste the output back to finalize per-plugin fixups)

cd /path/to/ductile && git fetch && git checkout feat/privsep-uid-separation && git pull
CFG=$HOME/.config/ductile           # adjust if yours differs (e.g. ~/admin/ductile-local/config)
ductile config check --config "$CFG" --json | tee /tmp/ductile-preflight.json
# Enabled plugins + any operator-owned paths they read/write (these are the EACCES candidates):
grep -nE '^[a-zA-Z0-9_-]+:|enabled:|run_as:|allowed_(read|write)_paths:|output_dir:|root:|path:|working_dir:|command:' "$CFG/plugins.yaml"
ls -la "$CFG"                        # note where config/state/vault/age key live + current owner
systemctl --user status ductile-local 2>/dev/null | head -3 || launchctl list | grep ductile

Per-plugin rule: under enforce a plugin runs as its account uid → it can only write its own state_dir, read world-readable code, and get secrets over stdin. Known breakage classes from the audit: file_handler (writes allowed_write_paths), file_watch/folder_watch (read watched paths), sys_exec (→ untrusted), and any plugin reading a secret from a $HOME dotfile (e.g. fabric → ~/.config/fabric/.env) → move that secret into the vault or copy it into the account's state_dir. Per your risk call, it's also fine to cut over and read the EACCES failures out of the job logs, then fix the few that break.

1. Rollback point (do not skip)

ductile system backup --to ~/backups/pre-privsep-$(date -u +%Y%m%dT%H%M%SZ).tar.gz --scope config --config "$CFG"
cp "$(command -v ductile)" ~/backups/ductile-prev        # keep the current (hygiene-only) binary
cp -a "$CFG" ~/backups/ductile-config-snapshot           # plain copy of config+vault for hand-revert

2. Build + stage the branch binary (don't install yet)

cd /path/to/ductile && go build -o ~/staging/ductile-new ./cmd/ductile
~/staging/ductile-new --version                          # confirm it's the privsep branch build

3. Provision accounts as root (the deploy/systemd/ templates)

sudo install -m0644 deploy/systemd/ductile-accounts.sysusers.conf /etc/sysusers.d/ductile-accounts.conf
sudo install -m0644 deploy/systemd/ductile-accounts.tmpfiles.conf /etc/tmpfiles.d/ductile-accounts.conf
sudo systemd-sysusers                                    # creates ductile(984) default(1001) untrusted(1002)
sudo systemd-tmpfiles --create /etc/tmpfiles.d/ductile-accounts.conf   # creates the 0700 state dirs
id ductile ductile-default ductile-untrusted             # verify they exist

4. Relocate config + secrets to ductile-owned system paths

sudo mkdir -p /etc/ductile
sudo cp -a "$CFG"/. /etc/ductile/                        # config.yaml, plugins.yaml, pipelines.yaml, vault.age, .checksums, ductile.db
sudo cp -a ~/.config/secrets/ductile/age.key /etc/ductile/age.key   # key moves INTO the (soon 0700) config dir
sudo chown -R ductile:ductile /etc/ductile /var/lib/ductile
sudo chmod 600 /etc/ductile/age.key
# Edit /etc/ductile/config.yaml as root:
#   state.path: /var/lib/ductile/ductile.db   (or keep under /etc/ductile if you copied the DB there)
#   secrets.age_key_file: /etc/ductile/age.key
#   accounts:
#     default:   { uid: 1001, gid: 1001, state_dir: /var/lib/ductile/accounts/default }
#     untrusted: { uid: 1002, gid: 1002, state_dir: /var/lib/ductile/accounts/untrusted }
#   plugins.sys_exec.run_as: untrusted        # ungranted plugins fall back to default
sudo -u ductile /usr/local/bin/ductile config check --config /etc/ductile/config.yaml   # MUST be clean

5. Per-plugin fs fixups (from step 0 / first-boot EACCES)

For each plugin that reads/writes an operator-owned path: repoint it under /var/lib/ductile/accounts/<tier>/… (owned by the account), make the input world-readable, or move its secret into the vault. sys_exec's working_dir must be owned by ductile-untrusted.

6. Cutover

sudo cp ~/staging/ductile-new /usr/local/bin/ductile
systemctl --user disable --now ductile-local 2>/dev/null || launchctl bootout gui/$(id -u)/com.mattjoyce.ductile-local
sudo install -m0644 deploy/systemd/ductile.service /etc/systemd/system/ductile.service
sudo systemctl daemon-reload && sudo systemctl enable --now ductile

7. VERIFY — including the wall-bite check (the part revert can't save you from)

journalctl -u ductile -n 60        # expect: "privsep enforcing: plugins drop to their resolved account"
                                   # NOT "privsep boot gate ... refusing", NOT "privsep UNCONFINED"
curl -s localhost:8081/healthz     # plugins_loaded == pre-deploy baseline
# run one job per enabled plugin; watch job logs for EACCES / "account uid/gid drop failed"

# WALL-BITE — prove the wall physically bites (availability revert won't catch a silent no-wall):
sudo -u ductile-default cat /etc/ductile/age.key        # MUST print: Permission denied
sudo -u ductile-default ls /var/lib/ductile/accounts/untrusted   # MUST print: Permission denied
# (optional, the headline threat) cross-uid ptrace between two plugins on DIFFERENT accounts must fail.

If the age key is readable by ductile-default, the wall is NOT real — stop and fix ownership before trusting it.

8. Rollback (if it refuses to boot or breaks plugins you can't quickly fix)

sudo systemctl disable --now ductile
sudo cp ~/backups/ductile-prev "$(command -v ductile)"
systemctl --user enable --now ductile-local              # or launchctl bootstrap the old LaunchAgent
# config/vault are untouched at $CFG; the /etc/ductile copy is additive and harmless to the old service.

Notes

  • This is the first live enforce host. Observe it before the macOS rollout (#95) — and remember the Mac needs a different shape (root LaunchDaemon, no cap-only; see #102).
  • The age key now lives inside the 0700 ductile-owned config dir, so accounts can't read it. If you prefer it outside the bundle (ADR §8 floor), keep it root-owned 0600 at a path ductile can read and point secrets.age_key_file there — just confirm the wall-bite check still denies accounts.

Learnings from the first live run (2026-06-07) — VALIDATED

This runbook was executed live on the Thinkpad; the wall is proven (enforce + uid drop to 1002 + cross-account EACCES). What we learned:

  • Deploy-as-NEW, don't migrate. The live config was too home-entangled to re-home (plugin code under ~/Projects/*, ~12 secret .env under ~/.config/secrets/*, config under ~/.config). A fresh /etc/ductile + /opt code + minimal confinable config booted enforce green in minutes; the migration would have been multi-hour with whole-gateway breakage.
  • Footgun (costs a fail-closed boot refusal): chown -R ductile:ductile /var/lib/ductile clobbers the account-owned state dirs and trips the boot gate. chown only /etc/ductile; tmpfiles.d owns the account dirs — leave them.
  • EMPTY DB for a newer binary. Carrying the old prod DB risks a schema-validation boot refusal vs a newer binary; a fresh DB sidesteps it entirely (history stays in the backup).
  • Wall-proof that worked: sys_exec(command=id) via API → stdout uid=1002(ductile-untrusted) with groups reset to 1002 only; sudo -u ductile-default ls .../untrusted → Permission denied.
  • DX: config dir 0700 ductile-owned → ductile job inspect as a human user hits EACCES; read job results via the API or as the ductile user.
  • Confinable vs unconfinable split: docker compose / apt scripts / fabric(~/.config/fabric) / file_handler(reads /home/matt) cannot run as an account uid — keep admin automation on an unconfined gateway. Restoring the confinable plugins enforced is Phase 2 (card #103).

Phase 2 learnings (2026-06-07, live)

  • plugin lock is a REQUIRED migration step, not an end-cap. Relocating plugin code to /opt changes its bytes' provenance → no recorded fingerprint → privsep's #93 binding refuses to honor the default grant and fails SAFE, downgrading the plugin to the least-priv tier (untrusted 1002). The plugins still run, dropped + confined, just least-priv. Run ductile plugin lock --all against the /opt tree to record fingerprints, then they land on their intended default (1001). Witnessed live: youtube_playlist + check_db_garmin both downgraded→untrusted until locked. (This is a clean in-the-wild proof of the #93 swap-defence.)
  • Plugin secrets are NOT a config reconcile. PluginConf has no nested secret_ref; plugin secret delivery is either a <field>_ref: key-suffix (loader substitution — unverified for plugins) or the spawn-time stdin secrets map (principal == plugin name, kebab-case only, and the plugin must read that map). Snake-case plugin names are rejected as principals (tested: 400). So secret-holding plugins need a dedicated vault-native workstream (card #107) — only keyless plugins restore via config alone. Worse, the vault compose path is fail-OPEN on an unknown principal (card #108) — the one non-fail-closed seam in an otherwise fail-closed design.
  • migrate everything ≠ everything enforced. The estate splits three ways: keyless-confinable (enforced now), secret-holding-confinable (enforced after #107), and unconfinable admin/docker (a second unconfined instance, card #106). The enforced gateway is the data plane, not the whole world.
  • Vault-native plugin recipe (#107, PROVEN 2026-06-07): see card #107 — read the stdin secrets map, manifest config_keys.required:[], register a kebab principal, set vault_principal + requires_vault, relocate+lock. First proof: discord_notify posts from the encrypted vault over stdin, uid 1001, fail-closed.

Redeploy / binary refresh (validated live 2026-06-07; v1.1.0 upgrade validated 2026-10-28)

The repeatable way to push a new binary onto the privsep host. The unit is the system service ductile and the config is /etc/ductile, owned by ductile mode 0700. make install does not apply here, it targets ~/.local/bin and a --user unit. Run every CLI call against the live config as the service user (sudo -u ductile), because a human user gets EACCES.

cd /path/to/ductile && git pull --ff-only && make build
mkdir -p ~/staging && cp ductile ~/staging/ductile-new
CFG=/etc/ductile/config.yaml

# 1. Back up with the OLD binary. Write somewhere the service user can write, then move it home.
sudo -u ductile ductile system backup --scope all --to /tmp/ductile-pre.tar.gz --config /etc/ductile
sudo install -o "$USER" -m0600 /tmp/ductile-pre.tar.gz ~/staging/ && sudo rm /tmp/ductile-pre.tar.gz

# 2. Pre-flight the NEW binary against the live config while the old gateway still serves.
#    Only possible when .checksums is already the format the new binary reads. Coming from a
#    BLAKE3 (version 2) install, rehearse on a copy instead, see DEPLOYMENT.md section 8a.
sudo -u ductile ~/staging/ductile-new config check --config "$CFG"   # must say valid

# 3. Swap the binary. Idempotent FHS layer, leaves the old process running from memory.
sudo BIN_SRC=~/staging/ductile-new bash deploy/install.sh

# 4. Re-lock ONLY if the release notes say the .checksums format changed (see below), then restart.
sudo -u ductile ductile config lock --config "$CFG"
sudo systemctl restart ductile

# 5. Verify.
sudo -u ductile ductile config check --config "$CFG"                  # VALID
journalctl -u ductile -n 40 --no-pager    # "privsep enforcing", "compose-time attestation on"
curl -s localhost:8081/healthz            # status ok, plugins_loaded == your pre-deploy count
# verify a job still drops to its account (live `ps` shows uid 1001/1002), wall-bite holds
  • deploy/install.sh is safe to re-run: idempotent; lays/asserts only the FHS structure + binary + unit; it does NOT touch config/vault/plugin code and does NOT chown /var/lib/ductile (no account-dir clobber). Confirmed live: created /run/ductile, account dirs intact, new binary root:root 0755. It does not stop or restart the service, step 4 does.
  • A plain binary swap needs no re-lock. Plugin fingerprints are of the plugin code keyed by the vault nonce, not the gateway binary; config lock/plugin lock only matter when config/plugin bytes change.
  • A release that changes the .checksums format needs one config lock, before the restart. v1.1.0 is the first release to read SHA-256 (version 3) only, and it refuses to boot on a BLAKE3 (version 2) file. Follow DEPLOYMENT.md section 8a for that upgrade, including the rehearsal on a copy and stopping the old gateway before the live lock, because the old binary cannot run on a version 3 file. If the host already ran a build with the SHA-256 change (v1.1.0 development builds from card 148.3 onward), .checksums is already version 3 and the lock is a plain re-lock; it prints no REHASH [plugin] lines, which is how you tell nothing was migrated. The lock needs the vault nonce, which the service user reads from /etc/ductile/secret/age.key, so no admin token is involved.
  • Check token scopes before the restart. v1.1.0 refuses a token holding a scope no route grants, and two tokens sharing a principal name. The step-2 config check with the new binary catches both while the old gateway is still serving, and prints token policy: ... decisions as proof the policy loaded. After restart the log has an API policy cost margin line.
  • Do not chain a cd into /etc/ductile with &&. Your user cannot enter it, the cd fails, and the rest of the chain is silently skipped. Use absolute paths with sudo -u ductile.
  • selfcheck against the running gateway reports FAIL. pid_lock fails and the DB checks are skipped (WAL safety). That is expected. Run it with the service stopped if you need the DB invariants.
  • The vault admin token lives at /etc/ductile/secret/vault-admin-token (ductile-owned, 0600), beside age.key. Only the keyless vault routes (vault set/roll/revoke) need it. A deploy does not.
  • #101 verified live: config check on a config with accounts: removed but secrets kept fires config is VALID but privsep is UNCONFINED ... a popped plugin can read the decrypted secrets. And a run_as with no accounts: is a hard config-load error (fail-closed, not per-job).
  • #108 verified by unit tests: plugins.<name>.requires_vault: true makes an unknown principal fail the spawn closed (live demo skipped, it would break a plugin).