# Ductile > An automation runtime designed to be operated by AI agents. Ductile is a self-hosted automation runtime designed to be operated by AI agents. You describe a goal in natural language; your AI agent authors a YAML pipeline, runs it, watches the logs, and iterates until your goal is met. A single Go binary orchestrates polyglot plugins via a JSON-over-stdin/stdout subprocess protocol; every surface (NOUN ACTION CLI, REST API with OpenAPI discovery, queryable execution ledger) is shaped so an LLM can drive it. Sized for personal-scale automation (50–500 jobs/day). # Introduction # Getting Started with Ductile Welcome to **Ductile**, an automation runtime AI agents can run, debug, and build for — and humans can audit. This guide will help you get up and running in minutes. See [`CONSTITUTION.md`](https://ductile.run/CONSTITUTION.md) for why the system is shaped this way. ______________________________________________________________________ ## 1. Installation Ductile is written in Go and requires version **1.25.0** or newer (see `go.mod`). 1. **Clone the repository:** ```bash git clone https://github.com/mattjoyce/ductile.git cd ductile ``` 1. **Build the gateway:** ```bash go build -o ductile ./cmd/ductile ``` This creates a single executable named `ductile` in your project root. ______________________________________________________________________ ## 2. Basic Usage (The Echo Showcase) After building the binary, you can run the included `echo` plugin to verify the system. ### Step 1: Verify Plugin Discovery Ductile discovers plugins from `plugin_roots`. For this repo, the local `plugins/` directory includes `echo`: ```bash ls -F plugins/echo/manifest.yaml ``` ### Step 2: Configure the Plugin Ductile uses a directory-based config layout (typically `~/.config/ductile/`). This repo ships example files in `config/` — copy that folder to your config dir and edit. ```bash cp -R ./config ~/.config/ductile cp -R ./plugins/echo ~/.config/ductile/plugins/ # put echo in the config's plugin root ``` ```yaml # ~/.config/ductile/config.yaml excerpt plugin_roots: # Relative entries resolve against the config directory; "~" expands to home. - "plugins" include: - api.yaml - plugins.yaml - pipelines.yaml - webhooks.yaml ``` ```yaml # ~/.config/ductile/plugins.yaml excerpt plugins: echo: enabled: true schedules: - id: default every: 5m jitter: 30s config: message: "Hello from Ductile!" ``` ### Step 2b: Add an External Plugin Root (Optional) You can mount additional plugin volumes and add them to `plugin_roots` in priority order: ```yaml plugin_roots: - "plugins" # relative to the config directory - "/opt/ductile/plugins-private" # absolute roots work anywhere ``` Container example: ```bash docker run --rm \ -v "$PWD/config:/config" \ -v "$PWD/plugins:/config/plugins" \ -v "/srv/ductile-private-plugins:/opt/ductile/plugins-private:ro" \ ductile:latest ./ductile system start --config /config ``` ### Step 3: Start the Gateway Run the service in the foreground (defaults to `~/.config/ductile`): ```bash ./ductile system start ``` Or explicitly point to a config directory: ```bash ./ductile system start --config ~/.config/ductile ``` You will see logs indicating the scheduler has started. After 5 minutes (or however you configured it), you'll see the echo job execute and complete. ### Step 4: Graceful Shutdown Press `Ctrl+C` to stop the gateway. It will wait for any in-flight jobs to finish before releasing the process lock. ### Step 5: (Optional) Initialize the vault for plugin secrets The echo demo needs no secrets. But when you add a plugin that needs an API token, the credential comes from the **vault** — not an env var, not config. Stand it up once, with the gateway stopped: ```bash ductile secrets keygen --out ~/.config/ductile/age.key # an age key — back this up ductile vault init --vault ~/.config/ductile/vault.age --key ~/.config/ductile/age.key # prints a one-time admin token — store it; it authorizes vault writes ``` Then start the gateway and grant a plugin its secret. The full lifecycle (register → set → roll → revoke, and the `plugin lock` attestation that gates delivery) is in the [Secrets & Vault guide](https://ductile.run/SECRETS/index.md). ______________________________________________________________________ ## 3. CLI Principles Ductile is designed to be operated by both humans and LLMs. All commands follow a strict **NOUN ACTION** hierarchy: - **Hierarchy:** `ductile job inspect`, `ductile config lock`, `ductile system status`. - **Verbosity:** `-v` / `--verbose` gives detailed logic traces where a command supports it. - **Safety:** mutating config commands accept `--dry-run` to preview changes. - **Machine-Readability:** read commands accept `--json` for structured output. Check `ductile --help` for the flags a specific command takes. ______________________________________________________________________ ## Next Steps - **Operators:** Read the [Operator Guide](https://ductile.run/OPERATOR_GUIDE/index.md) to learn about monitoring and system maintenance. - **Developers:** Visit the [Plugin Development Guide](https://ductile.run/PLUGIN_DEVELOPMENT/index.md) to start building your own skills. - **Architects:** Deep dive into the [Architecture](https://ductile.run/ARCHITECTURE/index.md) and [Pipelines](https://ductile.run/PIPELINES/index.md) model. - **Secrets:** See [Secrets & Vault](https://ductile.run/SECRETS/index.md) for delivering credentials to plugins (the vault, principals, and attestation). # The 8 Idioms of Ductile These are the design rules. Not preferences — the discipline that keeps Ductile small enough to reason about and predictable enough for an agent to drive. When a change feels off, it usually violates one of these; when proposing a new plugin, pipeline DSL feature, or core surface, name the idioms it serves. The 8 are arranged: **foundation → flow → discipline → audience.** Read them in order; later idioms presuppose earlier ones. For why Ductile exists at all, see [`../CONSTITUTION.md`](https://ductile.run/CONSTITUTION.md). For the contributor mechanics, see [`../AGENTS.md`](https://ductile.run/AGENTS.md). This document is the authoring contract those two reference. ______________________________________________________________________ ## Foundation — what the system is ### 1. Every unit of work is a queued job. There is no other path. Webhooks, schedules, API calls, plugin output routes, even `ductile run ` invocations from the CLI — all enqueue into the single SQLite-backed FIFO and dispatch from there. **Why.** A single source of truth makes execution deterministic, retryable, and observable by construction. Side channels (a "quick" direct call that bypasses the queue) destroy all three properties at once: you lose the retry budget, the trace, and the ledger entry that RCA depends on. **How it shows up.** `internal/queue/queue.go` is the only writer to `job_queue`. Every producer — scheduler, webhook receiver, router, CLI — funnels through `Enqueue`. No code path serves a plugin's output without queueing first. ### 2. Spawn-per-command. No daemon plugins. For every command: fork the plugin's entrypoint, write JSON to its stdin, read JSON from its stdout, kill the process. Lifetime is one invocation. **Why.** This single decision buys language agnosticism, crash isolation, and the absence of shared mutable plugin state in one move. There are no zombie processes to manage, no plugin memory leaks to chase, no protocol versions to negotiate over a long-lived connection. A plugin that breaks breaks one job; the supervisor stays up. **How it shows up.** `internal/dispatch/dispatcher.go` calls `spawnPlugin` with a fresh subprocess per invocation. Timeouts are SIGTERM → 5s grace → SIGKILL. The wire protocol envelope is the entire contract between core and plugin. ### 3. Core owns orchestration; plugins own side-effects. Routing, fan-out (`split:`), conditional branching (`if:`), payload remapping (`with:`) — all live in YAML pipelines, evaluated by the core. Plugins do domain work and emit facts. Plugins do not decide what runs next. **Why.** Orchestration must be inspectable: a human reads the pipeline file in `git diff`, an agent reads it via the API, and both see the same flow. If orchestration lives inside plugins, neither can audit it without reading source in three languages. Side-effects are necessarily messy and plugin-local; orchestration is necessarily structural and shared. **How it shows up.** Authored `if:` predicates compile into an internal `core.switch` hop in the dispatcher. Plugin code does not branch on payload to decide downstream routing; that decision is in YAML. The legacy `plugins/switch/` reference plugin remains for compatibility but its own manifest names it "Legacy payload classifier. Prefer pipeline `if:` conditions for new authoring." ______________________________________________________________________ ## Flow — how data moves ### 4. Events are the contract; payloads are the currency. Event types are stable, named, and routed. Payloads carry the data, shaped by the producing plugin's declared `fact_outputs`. Renaming an event is a breaking change. Adding a new event is not. **Why.** Stable contracts let plugins evolve independently. A consumer that subscribes to `github.webhook.pull_request` should not care that the underlying GitHub plugin is now in version 0.7.x. The event type is the join key; the payload shape is documented in the manifest. **How it shows up.** The router (`internal/router/`) matches on event type only — exact match, no wildcards. Payload validation against `fact_outputs` is the producer's responsibility at emit time. ### 5. Value, state, and identity are kept separate. `config` is a value (static, env-interpolated, immutable for the run). `plugin_facts` is an append-only series of values (the durable record of what a plugin observed). `plugin_state` is a derived view (a compatibility/cache projection of the latest fact). `pipeline` is an identity (a stable name for a series of executions; its runs are values). `job` is an identity (a queued unit; its status is state). **Why.** Conflating these is the code path that breaks under retry and crash. The classic mistake is "let's just update the row" — but the question is whether the row is a value (you can't update; you append a new one) or state (you can, with care). The plugin_facts vs plugin_state split is this idiom made concrete: facts are values you append; state is the latest-value view rebuilt from them. **How it shows up.** `internal/state/` separates the two storage shapes. The manifest's `compatibility_view` declaration tells the core how to project facts into a state view for backward compatibility. Pipelines have stable names (identity); pipeline runs are immutable records (values). ______________________________________________________________________ ## Discipline — how to extend it ### 6. Idempotent by design. At-least-once delivery is the contract. Every command must be safe to repeat. Plugins that need uniqueness use a `dedupe_key` on the event, not an "exactly-once" myth. **Why.** Retries are guaranteed by the architecture: the queue replays crashed jobs, the scheduler can fire twice if the clock jitters, the webhook receiver may see the same delivery twice if the sender retries. A plugin that corrupts state on the second call is a plugin that will corrupt state in production. There is no path to "exactly once" — only "idempotent + retried." **How it shows up.** The router carries `dedupe_key` through the event envelope; the queue deduplicates pending jobs by `(plugin, dedupe_key)` within the active window. Plugins that mutate external state are expected to either accept duplicate calls cleanly or use the dedupe key themselves. ### 7. Composable over configurable. Many small plugins chained with `with:` remap > one plugin with twenty flags. Many short pipelines > one long pipeline with eighteen conditionals. Configuration surfaces grow forever; composition surfaces grow only where there's a real new shape. **Why.** "Simple is the goal, not easy." A plugin with a giant option matrix looks easy ("you can do anything!") but is hard to reason about, test, and operate. A small plugin with one job composes with others to do the same work, but each piece is independently verifiable. **How it shows up.** The pipeline `with:` step was added specifically to avoid the proliferation of one-off plugin aliases that differ only in how they relabel their input. Step-level remapping does the relabeling; the underlying plugin stays focused. ______________________________________________________________________ ## Audience — who operates it ### 8. Every surface is agent-drivable. NOUN ACTION CLI, OpenAPI on every endpoint, structured JSON I/O, queryable execution ledger, exit codes that mean something. No path through Ductile requires reading source or clicking a canvas. The agent is the primary operator; the human is the auditor. **Why.** This is the alignment paragraph in the Constitution made practical. Observability is not a feature here — it's the substrate this idiom rests on. A surface that an agent cannot drive blind (CLI without machine-readable output, an endpoint without OpenAPI, a config knob without schema) violates the alignment. **How it shows up.** - CLI: `ductile `; `--output json` on every read command. - API: every endpoint listed in `/openapi.json`; `/skills` registry enumerates pipelines as discoverable tools. - Diagnostics: `GET /system/doctor`, `GET /system/selfcheck`, `GET /stopwatch/{plugin}` (p50/p95/p99), `GET /topology` (plugin- signal-plugin graph). - Ledger: every job, every step, every plugin invocation persisted in SQLite; queryable directly or via `ductile inspect`. If you add a feature whose only interface is a curl command an agent has to construct from documentation, you've violated this idiom. Add the OpenAPI entry, the CLI verb, the structured exit code. ______________________________________________________________________ ## What's not in the 8 (deliberately) These appear in older docs or look idiom-shaped but are not rules: - **"Ductile is upstream and downstream."** A capability claim, not a design rule. Lives in the capabilities list, not here. - **"Switch decides; plugins implement."** Superseded by idiom 3 (core owns orchestration). The Switch plugin is legacy; `if:` predicates are the authoring concept. - **"Observability is a feature."** Subsumed by idiom 8 — observability isn't an *added* feature, it's how the agent-drivable surfaces work at all. - **"Workflow logic belongs in the plugin."** The pre-Sprint-6 version of idiom 3. Inverted by current architecture; left here as a warning. ______________________________________________________________________ ## What this list is for Two readers: **A pipeline or plugin author** (human or agent) uses these as a checklist. Does this connector hold state across invocations? (Violates 2.) Does this YAML stash branching logic inside the plugin's config? (Violates 3.) Does this command have a `--quiet` flag but no JSON output mode? (Violates 8.) **A reviewer of changes to the core** uses these as the bar. A PR that adds a long-lived plugin connection (violates 2), or a route-by-string- matching feature (violates 4), or a new endpoint without an OpenAPI schema (violates 8), is a PR that needs to defend the deviation, not land quietly. When in doubt, check against the [Constitution](https://ductile.run/CONSTITUTION.md) pillar the change is supposed to serve. An idiom that doesn't fit any pillar is probably wrong; a pillar a change doesn't serve is probably the wrong pillar. # Ductile Glossary Key terms used throughout Ductile's documentation and configuration. ______________________________________________________________________ ## Gateway The `ductile` binary — the central runtime that manages plugins, schedules work, routes events, and maintains the execution ledger. ## Plugin / Connector A polyglot adapter that connects Ductile to an external system (an API, a database, a shell command). Written in any language; communicates via JSON over stdin/stdout. - **Plugin:** The code and manifest (the implementation). - **Connector:** The logical integration point (the "skill"). ## Alias (Plugin Instance) A uniquely named and configured instance of a base plugin. Defined in `plugins.yaml` using the `uses:` field. Allows running multiple copies of the same logic (e.g., `discord_alerts` vs `discord_logs`) with different settings. ## Command A discrete operation provided by a plugin. Common commands include: - **`poll`** — proactive; Ductile calls the plugin on a schedule to pull data. - **`handle`** — reactive; the plugin processes an incoming event. - **`health`** — diagnostic; verifies the plugin's prerequisites are met. - **`init`** — one-time setup; runs when a plugin is first registered. ## Pipeline A high-level workflow orchestration defined in YAML. Pipelines react to a single trigger event and execute a sequence of plugin steps, automatically passing data between them. ## Event Router (Pipeline & Event Routing) The internal deterministic routing layer (`internal/router`) that maps events emitted by plugins, webhook receivers, or the API to downstream actions. Rather than running as an asynchronous in-memory pub/sub "event bus," the Event Router evaluates incoming events and context triggers against the configured pipelines and global routes, producing new job dispatches that are transactionally enqueued back into the SQLite Work Queue. This keeps the work queue as the single source of truth and durability boundary for all execution states. ## Event Hub An in-memory pub/sub ring-buffer (`internal/events`) used exclusively for passive diagnostics, logging, and observation (such as real-time updates for the TUI or event stream API). Unlike the Event Router, the Event Hub is completely decoupled from the execution path to ensure slow observers never block core workers. ## Event A typed packet of data (e.g., `youtube.playlist_item`) that signals an occurrence and triggers routing within the gateway. ## Payload The JSON object attached to an event. Payload fields are passed to downstream plugins when the event is routed. ## Context (Baggage) Immutable metadata (e.g., `origin_user_id`, `trace_id`) that persists across every hop of a multi-step pipeline once a step claims it with `baggage`. Carried in the `event_context` ledger and merged into downstream requests. ## Worker Pool (Max Workers) The global set of execution slots that process jobs in parallel. Controlled by `service.max_workers` (defaults to `max(1, CPU-1)`). Operators can force whole-system serial dispatch by setting it to `1`. ## Worker / Consumer An individual execution slot within the bounded worker pool. Workers continuously pull (dequeue) eligible jobs from the SQLite Work Queue and execute them by spawning plugin subprocesses. ## Parallelism The maximum number of concurrent jobs allowed for a specific plugin or alias. Prevents a single resource-heavy plugin from saturating the worker pool. ## Concurrency Safe A boolean hint in a plugin's `manifest.yaml`. Omitted means `true`. If set to `false`, the plugin author is declaring that same-plugin concurrent execution is unsafe unless an operator deliberately constrains or overrides plugin parallelism. ## Smart Dequeue The logic that skips jobs in the queue if their target plugin has already reached its parallelism limit, allowing other plugins to proceed. ## Result The human-readable summary or data returned by a plugin in its protocol response. Often used as the input for the next step in a pipeline. ## Plugin Facts The append-only record of durable plugin observations. Each row carries a stable snapshot a plugin emitted as `state_updates`, plus a manifest-declared `fact_type` and a Ductile-owned monotonic `seq`. This is the durable record of what a plugin remembers across runs. See [PLUGIN_FACTS.md](https://ductile.run/PLUGIN_FACTS/index.md). ## Plugin State (Compatibility View) A single JSON row per plugin maintained as the compatibility/cache view of the latest fact. Existing readers (and legacy plugins that have not yet declared `fact_outputs`) see the same shape they always have. The view is rebuilt automatically by core when a new fact lands. New plugins should declare `fact_outputs` rather than treating this row as the place where durable truth lives. ## Producer Any system component that enqueues a job into the Work Queue. Primary producers are: the Scheduler (heartbeat ticks), Webhook Receiver (inbound HTTP payloads), Event Router (routing pipeline steps), and the CLI or API. ## Job The atomic unit of work in Ductile. Every command invocation creates an immutable Job record capturing input, output, logs, and status. ## Queue (SQLite-backed Work Queue) The persistent, SQLite-backed First-In, First-Out (FIFO) work queue that acts as the transactional durability boundary for Ductile. All job executions, retries, and step transitions pass through this queue to survive crashes, timeouts, and system restarts. ## Enqueue The action of transactionally appending a new job to the SQLite Work Queue. ## Dequeue The action of a worker pulling the next eligible job from the SQLite Work Queue for execution. ## FIFO (First-In, First-Out) The ordering principle of the SQLite Work Queue. Jobs are processed strictly in the order they are enqueued, subject to concurrency and parallelism caps. ## Schedule A configuration entry that tells the scheduler when and how to run a plugin command (e.g., `every: 5m`, `cron: "0 * * * *"`). ## Jitter A random offset applied to schedules to prevent multiple jobs from triggering at the exact same millisecond (the "thundering herd" problem). ## Deduplication (Dedupe Key) The mechanism to prevent duplicate enqueues or redundant processing. If a job is submitted with a `dedupe_key`, enqueue is suppressed if a matching job is already in-flight (status `queued` or `running`) or succeeded within the deduplication window (`dedupe_ttl`). ## Circuit Breaker An automated safety switch that "opens" after repeated plugin failures, temporarily blocking scheduled runs to allow the system or external API to recover. ## Webhook An HMAC-verified HTTP endpoint that accepts external events and transactionally enqueues them into the Ductile SQLite Work Queue via the Event Router. ## Skill A machine-readable description of a capability (either an atomic plugin command or an orchestrated pipeline), exported via `/skills` or `/openapi.json`. ## Workspace (historical) Formerly: a per-job, hard-link-cloned directory the core provisioned for each plugin invocation. Removed; the core no longer touches the filesystem on a job's behalf. Plugins that need a scratch path manage it themselves. ## Execution Ledger The persistent history of all jobs, pipeline steps, and event transitions. Used for the TUI "Overwatch" and audit logging. ______________________________________________________________________ ## Vault Ductile's owned secret store: an age-encrypted blob (`vault.age`) held in memory at runtime. The daemon is its **sole writer and reader** (the sole-writer owner). Distinct from encryption-at-rest of config files — the vault has a *lifecycle* (register, set, roll, revoke). See `docs/SECRETS.md`. ## Principal A registered deliver-to identity in the vault — a plugin, a consumer, or the gateway. Secrets are granted *to* principals; at spawn the core delivers a principal's authorized secrets to the matching plugin. Registering a principal is authorization only; identity/attestation is separate (see Plugin lock). ## authorized_principals The list on a secret naming the principals allowed to receive it. An empty list means no principal receives it via delivery. A grant to an unregistered principal is refused at write time (no orphan grants). ## Compose The pure, read-locked query that resolves a principal's deliverable secrets at spawn time. An **unknown** principal composes to empty (the plugin runs with no secrets); a **revoked** principal composes fail-closed (the job fails rather than run secret-less). ## secret_ref A config reference (e.g. on a webhook/relay HMAC) that resolves to a secret value. Resolvable against the vault as the source of truth (and, during migration, the legacy `tokens.yaml` table). ## Attestation Binding a secret-receiving plugin to its exact bytes. A plugin must be attested (`plugin lock`) before it receives any secret; the core re-verifies at spawn and fails closed on a mismatch. ## Fingerprint The keyed hash (keyed-BLAKE3 over a plugin's manifest + entrypoint bytes, keyed by the `core` nonce) that `plugin lock` records and the core re-checks at spawn. Editing manifest or entrypoint changes the fingerprint and requires re-attestation. ## Nonce A 32-byte secret seeded on the reserved `core` principal at vault genesis. It *keys* the plugin fingerprints, so a fingerprint cannot be recomputed without the vault — binding attestation to this gateway's vault. ## Plugin lock The explicit act (`ductile plugin lock `) that attests a plugin's bytes by recording its fingerprint. **Decoupled** from Config lock: it is the only thing that re-blesses changed plugin bytes, and it gates secret delivery. ## Config lock The act (`ductile config lock`) that re-hashes the config *files* into `.checksums`. It **preserves** existing plugin fingerprints but does **not** re-attest plugin bytes — that is Plugin lock. The decoupling closes "lock-laundering" (a lock taken for one reason silently approving a swapped binary). ## Reserved entity A vault entity the data plane cannot mutate: the `core` principal (holds the nonce) and the `core-admin-token` secret (the management-API credential). Guarded against ordinary set/roll/revoke; the admin token rotates through a dedicated path. # Audiences Three orthogonal axes define who reads Ductile's documentation and uses its surfaces. Eight cells. Each cell is a real reader, and every documentation or software affordance can be evaluated against the cells it serves. This document is **taxonomy-neutral**: it describes who the readers are, not how `docs/` is organised today. The doc layout should serve these audiences; when it stops doing so, the layout changes — these definitions do not. ______________________________________________________________________ ## Axes | Axis | Distinction | What it controls | | -------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------- | | **Agent ↔ Human** | Who is reading? | *Form*: machine-actionable schemas/skills vs narrative prose. | | **Coder ↔ Operator** | Are they changing Ductile, or running it? | *Domain*: `internal/`-facing code surfaces vs `~/.config/`-facing runtime surfaces. | | **Learner ↔ Expert** | Forming a mental model, or looking something up? | *Density*: tutorial + one example vs reference + invariants. | The axes are independent. A persona is *not* a stereotype; it is the intersection of three deliberate choices. ______________________________________________________________________ ## The eight cells | # | Persona | Axes | Landing surface today | Coverage | | --- | ------------------------------------------------- | --------- | -------------------------------------------------------------------- | ----------- | | 1 | New contributor / first-time plugin author | H · C · L | `README.md`, `docs/GETTING_STARTED.md`, `docs/PLUGIN_DEVELOPMENT.md` | **partial** | | 2 | Maintainer / experienced plugin author | H · C · E | `AGENTS.md`, `docs/ARCHITECTURE.md`, `docs/PIPELINES.md` | **served** | | 3 | Evaluator / first-time installer | H · O · L | `README.md`, `docs/GETTING_STARTED.md`, `docs/MACOS_INSTALLATION.md` | **partial** | | 4 | Veteran operator running Ductile in production | H · O · E | `docs/OPERATOR_GUIDE.md`, `docs/DEPLOYMENT.md`, `docs/DATABASE.md` | **partial** | | 5 | Cold-start agent generating a plugin | A · C · L | `schemas/`, `skills/ductile/`, `plugins/echo/` | **partial** | | 6 | In-repo coding agent | A · C · E | `AGENTS.md`, `internal/docs/lint_test.go` | **served** | | 7 | Agent doing first-time setup or config generation | A · O · L | none canonical | **gap** | | 8 | Agent operating a live Ductile instance | A · O · E | `/skills` registry, OpenAPI | **partial** | **Status legend:** *served* — surface exists and works for this cell. *partial* — surface exists but is incomplete, scattered, or known-stale. *gap* — net-new content or feature is needed. *deferred* — work is parked with explicit scope (see Coverage below). ______________________________________________________________________ ## One-paragraph stories **1. H · C · L — first-time contributor.** Cloned the repo to add a plugin for service X. Wants a 30-minute path from clone to working plugin on their machine, learning idioms by doing rather than by reading 800-line specs. Needs: a "start here" page ≤ 5 minutes of reading, one runnable teaching plugin, vocabulary table cross-linked so terms acquire meaning during the walkthrough. **2. H · C · E — maintainer.** Changing the queue, router, or protocol. Wants design decisions, invariants, and trade-offs in one click so they don't break a constraint they didn't know existed. Needs: authoritative architecture doc (today), per-package design notes for load-bearing packages (today: gap), enumerated non-negotiable constraints (today: `AGENTS.md §3d`). **3. H · O · L — evaluator.** Just heard about Ductile. Wants a five-minute path from install to seeing a real event flow, *before* reading documentation, so they can decide whether Ductile fits. Needs: a working minimal example, generated config (not copy-paste), one screenshot or asciinema of a live event flow (the standalone `ductile-watch` TUI is under redesign; see `ductile-hickey-tui-rip-and-rewrite` working note). **4. H · O · E — veteran operator.** Running Ductile on Unraid or a homelab. Wants runbooks for failure modes — orphaned jobs, integrity check failure, plugin crash loop, full disk — so they can recover without reading source. Needs: operator runbook organised by *symptom*, not by feature. **5. A · C · L — cold-start agent generating a plugin.** Invoked with no prior Ductile context. Wants to discover the plugin contract from machine-readable schemas and one canonical example, so it can produce a valid plugin without hallucinating field names. Needs: schemas at a stable path, a reference plugin explicitly tagged as the teaching example. **6. A · C · E — in-repo coding agent.** Working inside the repo on a real change. Wants the contract — style, safety, idioms, vocabulary — in one file at a predictable path. Needs: `AGENTS.md` (now done), per-package design notes (overlaps with persona 2), a `bd` workflow it can drive (already present). **7. A · O · L — agent doing first-time setup.** Helping a user adopt Ductile. Wants a deterministic way to generate a valid initial config and verify it, so it never produces config that fails `ductile config check`. Needs: `ductile init`, advertised use of `schemas/config.schema.json`, a curated `examples/` library. **8. A · O · E — agent operating live Ductile.** Building automations against a running instance. Wants every operation discoverable through `/skills` and OpenAPI, so it does not need to read prose docs to act safely. Needs: live OpenAPI, `/skills` as the *primary* surface for this cell, an agent-readable runbook for recovery flows. ______________________________________________________________________ ## Cross-cutting design implications - **Audiences share information; they need different forms.** The same fact ("baggage propagates downstream") needs narrative form for Learner cells, reference form for Expert cells, and machine-readable form for Agent cells. A doc plan that ignores this rebuilds the same content three times by accident; a deliberate plan maintains it once and projects it three ways. - **Coder ↔ Operator is the cleanest domain split** and maps naturally to the existing `internal/` vs `~/.config/` boundary. - **Learner ↔ Expert is information density.** A document trying to be both serves neither. Tutorial content and reference content should not share files. - **Agent ↔ Human is a form question.** Agent surfaces (`schemas/`, `skills/`, OpenAPI, `AGENTS.md`) are not separate documentation; they are the same content in machine-actionable form. They deserve first-class billing alongside `docs/`, not inside it. - **Two cells unblock from the same work.** Personas 3 (H·O·L) and 7 (A·O·L) both fail today for the same reason: there is no canonical, validated minimal config. A `ductile init` plus a curated `examples/` library serves both. - **One cell is a content gap, not a layout gap.** Persona 4 (H·O·E) needs symptom-organised runbooks that do not exist anywhere today. No reorganisation produces them. ______________________________________________________________________ ## Coverage today | Cell | Status | Notes | | ------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1 H·C·L | partial | No signposted "first 30 minutes" path; idioms now live in `AGENTS.md` but the entry experience does not yet guide a learner there. | | 2 H·C·E | served | `AGENTS.md` (vocabulary, design grounding, constraints) plus `docs/ARCHITECTURE.md` cover the steady-state need. Per-package design notes would strengthen it. | | 3 H·O·L | partial / **deferred (Phase 3)** | Root `config.yaml` is the welcome-mat and is known-stale. Rethinking the exemplar is parked: `ductile init` + curated `examples/`. Watch TUI ripped pending `ductile-watch` rewrite (see `ductile-hickey-tui-rip-and-rewrite`). | | 4 H·O·E | partial / **gap** | Operator guide and deployment exist; symptom-driven runbooks do not. Net-new content. Interim observability is the API and structured logs; `ductile-watch` redesign tracked in `ductile-hickey-tui-rip-and-rewrite`. | | 5 A·C·L | partial | Schemas exist but are not advertised; no plugin is explicitly labelled as the canonical teaching example. | | 6 A·C·E | served | Unified `AGENTS.md` is the contract; doc-smoke lint covers it. | | 7 A·O·L | **gap / deferred (Phase 3)** | Same unblock as persona 3. | | 8 A·O·E | partial | `/skills` and OpenAPI exist; agent-readable recovery runbooks do not. | ______________________________________________________________________ ## How to use this document - **As a reader:** find your cell. Follow its landing surface. If your cell is marked *partial* or *gap*, the documentation cannot fully serve you yet and the gap is acknowledged here. - **As a contributor proposing a change to docs or affordances:** name the cells it serves and the cells it does not. A change that serves a *gap* cell is high-leverage; a change that re-paints a *served* cell needs stronger justification. - **As a reviewer:** cite cells in review comments. *"This is for cell 4, currently a gap"* is a precise statement; *"this is too detailed for beginners"* is not. - **As a maintainer:** when the doc taxonomy changes, re-validate this file. Every cell must still resolve to a landing surface (or be honestly marked *gap* / *deferred*). ______________________________________________________________________ ## Maintenance This file is referenced from `AGENTS.md` and `CONTRIBUTING.md` and is expected to evolve alongside the doc taxonomy. Coverage status should be updated whenever a cell's landing surface materially changes. If a cell goes unserved for a release without being marked *deferred*, that is a planning signal, not a documentation signal. # Tutorials # Tutorial: From `system start` to Ready-to-Trigger A newcomer's walk through every step the gateway takes between the moment you run `ductile system start` and the moment a job can actually fire. Each step cites the real source location, and each gate explains *why it refuses* — because in ductile, the refusals are load-bearing. Provenance Built from the repo's knowledge graph (graphify, 5,409 nodes) — boot spine traced via `runStart()` → `buildRuntime()`, then verified line-by-line against `cmd/ductile/runtime.go`. ## 1. The big picture Ductile is an integration gateway: plugins do side-effects, the core owns all orchestration (*"plugins stay dumb"* — AGENTS.md, Idiom 3). Boot is therefore not "open a port and go"; it is a sequence of **refusal gates** followed by the assembly of four **trigger planes** that all converge on one queue and one dispatcher. If any gate fails, the gateway does not limp — it refuses to start, loudly, with the reason. ``` flowchart TD A["ductile system start
CLI entry · main.go:31 → runtime.go:908"] B["Config resolution & load (with vault decrypt)
runtime.go:916–937"] C["PID lock — exactly one instance
runtime.go:939"] D["buildRuntime(): symlink scan → plugin discovery
runtime.go:352–401"] E["Admission gates: integrity seal · doctor · strict decode
runtime.go:443–499"] F["SQLite open → pipelines compiled → config snapshot
runtime.go:508–571"] G["Queue · state stores · event hub · scheduler · admitter
runtime.go:573–610"] H["Boot posture decision + privsep boot gate
runtime.go:641–714"] I["Dispatcher · relay · scheduler start · API bind · webhooks
runtime.go:716–881"] J["READY — signal loop supervising, four trigger planes live
runtime.go:967–993"] A --> B B -->|"refuses: no config found / vault load error"| C C -->|"refuses: another instance running"| D D -->|"refuses: symlinks (unless allowed), discovery failure"| E E -->|"refuses: drift, invalid config, unknown keys"| F F --> G G --> H H -->|"refuses: capability/accounts mismatch, side-doors (strict)"| I I -->|"refuses: token resolution fails, bind fails"| J style D stroke:#ff7b72 style E stroke:#ff7b72 style H stroke:#c792ea style J stroke:#7ee08a ``` ## 2. Phase 0 — CLI entry & config resolution 1. `main()` hands straight to `runCLI()` (`cmd/ductile/main.go:31,35`) — a noun-verb CLI; `system start` routes to `runStart()` (`cmd/ductile/runtime.go:908`). 1. **Config resolution**, in strict precedence (`runtime.go:916–929`): 1. `--config ` (file or directory) — "explicit" 1. `$DUCTILE_CONFIG_DIR` — "env" 1. auto-discovery (e.g. `~/.config/ductile/config.yaml`) — "auto-discovered" The chosen *source* is remembered and logged, so you can always tell which config won. 1. **`config.LoadWithVault()`** (`runtime.go:933`) loads and deep-merges the config files (a directory means `config.yaml` plus its `include:` files — plugins.yaml, pipelines.yaml, api.yaml, webhooks.yaml) *and* decrypts the vault **exactly once**. The decrypted owner is threaded through the whole boot so nothing decrypts twice. 1. **PID lock** (`runtime.go:939–944`) — `lock.AcquirePIDLock()` guarantees a single instance. A second `system start` fails here with "another instance may be running". 1. A **`reloadManager`** is created (`runtime.go:947–952`) — this is the named supervisor that later handles `SIGHUP`/`system reload` by building a whole new runtime and restoring the old one if the new one fails. Why a single load-time vault decrypt? The vault is the only place secrets live (api tokens are vault-only on the release path). Resolving the owner once and passing it down means the admission gates, the posture decision, and secret delivery all see the *same* vault presence — resolving it twice caused real bugs (#134). ## 3. Phase 1 — Hygiene & plugin discovery Everything from here to readiness lives in `buildRuntime()` (`cmd/ductile/runtime.go:352`) — the highest-betweenness hub in the knowledge graph (54 edges), because boot *and* reload both run through it. 1. **Logging up first** — `log.Setup(cfg.Service.LogLevel)` (`runtime.go:353`), so every subsequent refusal has a voice. 1. **Symlink scan** (`runtime.go:356–371`) — `CollectConfigPaths` + `DetectSymlinks` walk every config path. Symlinks are warned, and if `service.allow_symlinks` is false the boot is **refused**. A symlinked config is a config you didn't review. 1. **Plugin discovery** (`runtime.go:373–393`) — `resolvePluginRoots()` then `plugin.DiscoverManyWithOptions()` scans each `plugin_roots` directory for `manifest.yaml` files (protocol v2). Discovery builds the **registry**: what exists on disk, what commands each plugin supports. 1. **Aliases applied** (`runtime.go:394–401`) — a config entry with `uses:` registers one plugin under another name (e.g. two `sys_exec` instances with different argv allowlists). 1. **Preflight report** (`runtime.go:403–435`) — the log states every config file loaded, plugins discovered vs configured vs enabled, and the API listen address. Read this block first when debugging a boot. Newcomer note Discovery ≠ enablement. A plugin found on disk does nothing until `plugins.yaml` enables it, and it cannot be *used* by a pipeline until routing references it. Config declares; the core controls flow. ## 4. Phase 2 — Admission gates (the bouncers) Ductile's admission policy (`runtime.go:443`) is a set of independent switches (decomplected from the old bundled `strict_mode`). Each one can refuse the boot: | Gate | What it checks | Where | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | `verify_integrity_on_boot` | Recomputes checksums of every config file against the **.checksums seal** written by `config lock`. With `fail_on_drift`, any edit since the lock refuses the boot. | `runtime.go:460–466` | | `validate_config_on_boot` | Runs the **doctor** (`doctor.New(cfg, registry).Validate()`) — plugin refs, routes, webhooks, token scopes, hook cycles, schedules. Vault-aware: a from-scratch vault gateway with no api token yet is a legitimate bootstrap posture, not an error (#129). | `runtime.go:468–481` | | Strict decode | YAML decoding is lenient, so a typo'd key would be *silently dropped* — you'd believe a setting is active when it is not. Every dropped key is warned always; under `validate_config_on_boot` it is an admission **failure** (#26). | `runtime.go:491–499` | Why gates instead of warnings? The constitution calls these the *load-bearing refusals*: a gateway that starts on drifted or invalid config will fail later, at trigger time, in a way that is much harder to diagnose. Fail-closed at boot is the cheap failure. ## 5. Phase 3 — State, routing & the snapshot 1. **Database opens** — `storage.OpenSQLite(ctx, cfg.State.Path)` (`runtime.go:508`). One SQLite file holds the job queue, job logs, plugin facts/state, contexts, circuit breakers, transitions, and the audit trail. (In the knowledge graph, `OpenSQLite()` bridges 30+ communities — nearly everything touches it.) 1. **Scheduled commands validated** against the registry (`runtime.go:516`) — a schedule naming a command the plugin doesn't support refuses the boot. 1. **Pipelines compile** — `router.LoadFromConfigFiles()` (`runtime.go:532–543`) turns the `pipelines:` YAML into compiled routes (the DSL compiler: steps, `if:` conditionals, fan-out edges, max route depths). Every registered pipeline is logged with its trigger. 1. **Config snapshot recorded** — `configsnapshot.Build()` + `Insert()` (`runtime.go:545–571`). The exact config (hash, source, ductile version, binary path, **plugin fingerprints**) is written to the DB with reason `startup`. Every job later created carries this snapshot ID — you can always answer *"what config was live when this job ran?"* 1. **Core plumbing** (`runtime.go:573–610`): `queue.New()` (with dedupe TTL and the snapshot-ID provider), `state.NewStore()`, `state.NewContextStore()`, `events.NewHub(256)`, `scheduler.New()`, and `state.NewAdmitter()` (caps context payload bytes at admission). ## 6. Phase 4 — Security posture (vault & privsep) 1. **Secret delivery wired** (`runtime.go:618–633`) — if a vault owner exists, it becomes the `SecretComposer` (spawn-time, stdin-delivered secrets) and a `pluginVerifier` is created: **compose-time attestation** re-verifies a plugin's live bytes against its recorded keyed fingerprint *right before* handing it secrets (ADR §3.3). Changed bytes ⇒ no secrets. 1. **Boot posture decided** — `config.DecideBootPosture()` (`runtime.go:641–645`). Two postures: - **Gateway** (normal): api tokens resolve, everything opens. - **Management-only** ("vault-operable / ductile-closed"): a vault exists but no api token yet — the credential-ladder bootstrap. Only `/vault/*` on a local unix socket is served; *no* trigger plane opens. The fail-closed guard still refuses a from-scratch gateway with API enabled, zero tokens, and no vault to bootstrap from. 1. **Privsep boot gate** — `dispatch.BootGate(cfg)` (`runtime.go:651–714`, #86). The capability to drop privilege and the configured `accounts:` table must *agree*, or startup is refused — never a silent run at gateway privilege. Three outcomes: - **Enforcing**: plugins drop to their resolved account. The filesystem floor is reconciled (`ReconcileAccountFilesystem`, #87 — secrets 0600/0700, per-account 0700 dirs, all-or-refuse), and each account is probed for root side-doors (`AuditAccountSideDoors`, #111 — nopasswd sudo, docker group, writable setuid). Strict mode refuses on a side-door. - **Unconfined override**: accounts exist but `service.unconfined` forces gateway-uid execution — warned loudly, never silent. - **Hygiene-only**: no accounts configured (dev box) — posture logged explicitly so "valid config" is never mistaken for "enforcing" (#101). Why is posture decided before any plane opens? Every plane that could fire a pipeline — scheduler, dispatcher, public listener, webhooks — is gated on the posture (#136). In management posture, no pipeline can fire and no secret can be composed until an operator mints an api token over the admin socket and runs `system reload` to activate the gateway. ## 7. Phase 5 — Trigger planes come up 1. **Dispatcher built** — `dispatch.New(q, st, contextStore, router, registry, hub, cfg, …)` (`runtime.go:716–720`), carrying the admitter, secret composer, plugin verifier, and the privsep-enforce flag. This is the engine that will execute every job. 1. **Relay receiver** configured (`runtime.go:722`) — accepts signed event envelopes from remote ductile instances. 1. **Recovery hooks wired** (`runtime.go:731`) — when crash recovery marks a dead orphan job, the dispatcher's hook machinery fires (e.g. a `job-failure-notify` pipeline). 1. **Scheduler & dispatcher start** (`runtime.go:738–752`) (gateway posture only). The scheduler begins evaluating cron/interval entries; the dispatcher starts its worker loop polling the queue. 1. **API tokens resolve, fail-closed** — `config.ResolveAPITokens()` (`runtime.go:767–780`). Bearer tokens are `secret_ref`s into the vault; an unresolvable or empty ref **aborts the boot** — the API never opens authenticating against an empty credential (card #94). 1. **Listener reserved synchronously** — `apiServer.Bind()` (`runtime.go:822–826`) happens *before* serving, so an activation reload whose bind fails fails the reload (and the old runtime is restored) instead of answering "ok" and dying later (#140). Then `Start()` serves — or `StartManagement()` serves only `/vault/*` on the unix socket in management posture (`runtime.go:834–852`). 1. **Webhook server** (`runtime.go:864–881`) — if endpoints are configured (and posture is gateway), the webhook listener opens with per-endpoint HMAC signature verification. ## 8. Phase 6 — Ready: the supervisor loop 1. Back in `runStart()`: the PID lock is logged, macOS **TCC paths are pre-warmed** (`runTCCPrewarm`, `runtime.go:966`) so Full-Disk-Access prompts happen now, not mid-job. 1. `"ductile running"` is logged (`runtime.go:967`) and the process enters its supervision loop (`runtime.go:972–993`): - `SIGHUP` → `manager.Reload()`: build a complete new runtime through the same buildRuntime gates; on failure the old runtime keeps running. - `SIGINT` / `SIGTERM` → orderly stop. - Any component error on `errCh` → supervisor stops the gateway (exit 1) — the service manager (launchd/systemd/docker) restarts it. Let-it-crash, applied at the operator layer. 1. Verify from outside: `curl http://127.0.0.1:8080/healthz` reports status and the boot posture; `ductile system status` gives the operator view. ## 9. Now trigger a job The gateway is "ready" precisely because four independent **trigger planes** are now live, and all of them converge on the same path: **enqueue → queue (dedupe, concurrency keys, breaker) → dispatcher `ExecuteJob()` → plugin subprocess**. | Plane | What it does | Where | | ------------- | ------------------------------------------------------------------------ | ---------------------------------------- | | **API** | Authenticated `POST` runs a plugin command or pipeline; sync or async. | `internal/api/server.go` · `handlers.go` | | **Webhook** | HMAC-verified external POST (e.g. GitHub) routes into a pipeline. | `internal/webhook/server.go` | | **Scheduler** | Cron/interval entries enqueue plugin commands on their tick. | `internal/scheduler/` | | **Relay** | Signed envelopes from a remote ductile instance are admitted and routed. | `internal/relay/receiver.go` | The simplest first job, via the API plane: ```bash # 1. health — confirms posture is "gateway" curl -s http://127.0.0.1:8080/healthz # 2. trigger the echo plugin synchronously curl -s -X POST http://127.0.0.1:8080/plugins/echo/run \ -H "Authorization: Bearer $DUCTILE_TOKEN" \ -H "Content-Type: application/json" \ -d '{"payload": {"message": "hello, ductile"}, "sync": true}' # 3. inspect what happened — full lineage, attempts, config snapshot ductile job inspect ``` What happens inside The API handler authenticates the bearer token and its scopes, the **admitter** checks the context payload size, `queue.New()`'s dedupe logic drops replays inside the TTL, the job row is stamped with the active **config snapshot ID**, and the dispatcher picks it up: resolves the plugin and its account (privsep), verifies its fingerprint (if vaulted secrets are involved), spawns the subprocess with payload as `DUCTILE_PAYLOAD_*` env, captures the protocol-v2 JSON response, records facts/state updates, evaluates routing for successor steps, and finalizes the job. `ductile job inspect` shows you every one of those decisions after the fact. ### Where to go next - [Tutorial: Bootstrap — genesis to `/healthz`](https://ductile.run/tutorials/bootstrap/index.md) — the credential ladder that gets you here - [ARCHITECTURE](https://ductile.run/ARCHITECTURE/index.md) — the component map you just walked through - [8 Idioms of Ductile](https://ductile.run/8_IDIOMS_OF_DUCTILE/index.md) — why the design refuses what it refuses - [Getting Started](https://ductile.run/GETTING_STARTED/index.md) & [Cookbook](https://ductile.run/COOKBOOK/index.md) — recipes for real pipelines - [Operator Guide](https://ductile.run/OPERATOR_GUIDE/index.md) §3 — the admission gates in operator terms - [ADR: Vault credential ladder](https://ductile.run/adr/vault-credential-ladder/index.md) — the management-posture bootstrap in full ______________________________________________________________________ *Generated 2026-06-10 from the ductile knowledge graph (graphify) + line-verified against `cmd/ductile/runtime.go` @ main (99d909a). If boot code moves, regenerate via `/graphify query` — the graph is the map.* # Tutorial: Bootstrap — Genesis to `/healthz` From an empty directory to a locked, vault-native gateway answering `curl http://127.0.0.1:8080/healthz` — with no secret ever written into a config file. This explainer teaches the **credential ladder**: the age key, the vault, minting the admin token, minting and using the API token, and the practices that keep all of it honest. Provenance Built from the repo's knowledge graph (graphify) — traced via the `Credential Ladder` concept node (docs/BOOTSTRAP.md §The shape) and the `vault.Init()` genesis cluster, then verified against `internal/vault/genesis.go`, `cmd/ductile/vault.go`, and `cmd/ductile/runtime.go`. ## 1. The credential ladder — the whole idea Ductile's bootstrap answers a chicken-and-egg problem every secure system has: *the gateway needs a token to serve, but the token should live in the vault, and the vault is operated through the gateway.* The answer is a ladder — every credential is issued by the one above it, and **nothing is ever written into a config file**: ``` flowchart TD R1["1 · The age key
A file on disk, mode 0600. The only thing that can decrypt the vault.
secrets.age_key_file · created by ductile secrets keygen"] R2["2 · The vault admin token
Printed exactly once at genesis. Operates the management API (/vault/*).
Never delivered to any plugin — verified by the API, stored as the
reserved core-admin-token entry.
internal/vault/genesis.go:38 · vault.go:218 AuthenticateAdmin()"] R3["3 · The API bearer token
Minted into the vault by the admin token, referenced from config as a
secret_ref (never a literal), resolved fail-closed at boot. Operates the gateway.
config: api.auth.tokens[].secret_ref · runtime.go:767 ResolveAPITokens()"] R1 -->|decrypts| R2 R2 -->|mints| R3 style R1 stroke:#ffb454 style R2 stroke:#c792ea style R3 stroke:#7ee08a ``` Why a ladder and not a setup wizard? Each rung has one job and one custody boundary. Compromise of an API token never exposes the vault; compromise of config files exposes *no* secret at all (they contain only `secret_ref` names). And because the ladder is also the **recovery path** (lose a rung → climb back up from the rung above), there is no separate, rarely-tested recovery procedure to rot. ## 2. Rung 1 — Genesis: the age key and the vault Start with a config directory that has **no tokens at all** — the repo's `config/` folder is a working copy of exactly this state: ```bash CFG=~/.config/ductile # config.yaml must carry, at minimum: # secrets: # age_key_file: age.key # vault_file: vault.age # plugin_roots: # - plugins # at least one EXISTING dir; "~" expands, relative paths resolve against $CFG # api: # enabled: true # listen: "127.0.0.1:8080" # management_socket: /tmp/ductile-admin.sock # explicit + SHORT (see good practice) # # Do NOT add api.auth.tokens yet — that absence is the point. # Daemon stopped. Generate the age key, then create the vault: ductile secrets keygen --out "$CFG/age.key" ductile vault init --vault "$CFG/vault.age" --key "$CFG/age.key" ``` What `vault init` actually does — `vault.Init()` (`internal/vault/genesis.go:38`): 1. **Refuses to clobber**: if `vault.age` already exists, init errors out (`genesis.go:39–44`). A re-init can never silently wipe a live vault's secrets (fail-closed, Armstrong-style). 1. **Seeds the reserved `core` principal** with a freshly generated 32-byte **fingerprint nonce** (`genesis.go:59`) — this nonce later keys plugin attestation (a plugin's recorded fingerprint is HMAC'd with it). 1. **Mints the admin token** (32 random bytes) and stores it as the reserved `core-admin-token` entry via the sanctioned `RotateAdminToken` path (`genesis.go:63`). It has *no* `authorized_principals`: it is verified by the API, never composed/delivered to any plugin. 1. **Saves the first encrypted blob** — age-encrypted with your key, written atomically (`writeFileAtomic` + directory fsync, `vault.go:328,365`). 1. **Prints the admin token exactly once.** It is not recoverable in plaintext afterward without the key. Capture it into 0600 custody now (see good practice), then shred the capture. The admin token prints once. If you lose it before capturing it, you haven't lost the vault — `ductile vault rotate-admin-token` (daemon stopped, age key present) mints a replacement in place (`cmd/ductile/vault.go:294`). The age key is the rung above; it can always re-issue rung 2. ## 3. The management posture — a gateway that is deliberately closed Now seal and validate, then boot. A config with the API enabled and **zero** `api.auth.tokens` is *not* an error when a vault is present — it is the recognized **bootstrap state** (#129): ```bash # Lock FIRST, then check — scope files (webhooks.yaml) are checksum-verified # even by `config check`, and zero-tokens + genesis-vault IS a clean verdict. ductile config lock --config "$CFG" ductile config check --config "$CFG" # First boot: ductile system start --config "$CFG" & ductile system status --config "$CFG" # expect: boot_posture: management-only (live) ``` At boot, `config.DecideBootPosture()` (`cmd/ductile/runtime.go:641`) sees *vault present + no API token* and chooses the management posture. The two postures side by side: | | management-only · "vault-operable, ductile-closed" | gateway · normal serving posture | | ------------------- | -------------------------------------------------- | -------------------------------------- | | `/vault/*` | ✓ on a local **unix socket** only | ✓ still available to the admin token | | Public TCP listener | ✗ never opened | ✓ (after fail-closed token resolution) | | Webhooks | ✗ closed | ✓ (HMAC-verified) | | Scheduler | ✗ gated, no ticks | ✓ live | | Dispatcher | ✗ gated, no jobs | ✓ live | | `/healthz` | — | ✓ reports `"posture": "gateway"` | | Where | `runtime.go:738–752, 835–849` (#136) | `runtime.go:822–852` | Why gate every trigger plane, not just the listener? "Vault operable" and "ductile operable" are different trust states. Until an operator has minted credentials, *no pipeline may fire and no secret may be composed* — a scheduler tick firing a half-bootstrapped pipeline would be an unaudited action. The boot log says this loudly (`runtime.go:848`): it's a named, intentional posture — "waiting for an api token", never "stuck". ## 4. Rung 2 — The admin token: operating `/vault/*` The management socket is the only door right now, and the admin token is the only key to it. Every `/vault/*` request authenticates via `Vault.AuthenticateAdmin()` (`internal/vault/vault.go:218`) against the stored `core-admin-token` entry. Through this plane you can: | Operation | Command | Notes | | ------------------------------- | ---------------------------------------- | ---------------------------------------------------------------------------- | | Store / mint a secret | `vault set --name --pattern manual` | Value via **stdin, never argv** — argv is visible in `ps` and shell history. | | Register a plugin principal | `vault register-principal` | Refuses the reserved `core` name — genesis-only seeding. | | Roll / revoke secrets | `vault roll · revoke · roll-principal …` | Lifecycle verbs; every action lands in the append-only vault audit. | | Rotate the vault encryption key | `vault rotate-key` | Re-encrypts the blob under a new age key. | | Rotate the admin token itself | `vault rotate-admin-token` | Offline (daemon stopped), needs the age key — rung 1 re-issues rung 2. | Separate credential domain. The admin token operates the vault but cannot run a single pipeline; the API token runs pipelines but cannot read a raw secret back out (`vault get` is admin-plane). Keep the two in different custody if different people hold those duties. ## 5. Rung 3 — Minting and using the API token 1. **Generate a strong token value yourself** (you are choosing the credential; the vault is its custody): ```bash API_TOKEN=$(openssl rand -hex 32) ``` 1. **Mint it into the vault over the unix socket** — stdin for the value, admin token for authority: ```bash printf '%s' "$API_TOKEN" | ductile vault set \ --api-url "unix:///tmp/ductile-admin.sock" \ --token "$ADMIN_TOKEN" \ --name core-api-token --pattern manual ``` 1. **Reference it from config — by name, never by value.** Stop the daemon, then add: ```yaml api: auth: tokens: - secret_ref: core-api-token scopes: ["*"] # first token is the operator token; scope later tokens narrowly ``` At boot, `ResolveAPITokens()` (`runtime.go:767–780`) resolves every `secret_ref` against the vault, **fail-closed**: an unresolvable or empty ref aborts the boot — the API never opens authenticating against an empty credential. Scopes are part of the mint, not an afterthought. `"*"` is for your operator token. Tokens for automations should carry only what they use — e.g. `["jobs:read"]` for a dashboard, `["plugins:run"]` for a trigger-only client. The doctor validates scope names at boot (`doctor.go:271`). ## 6. Seal, activate, verify ```bash # Seal config and plugins — ORDER MATTERS: # config lock writes the .checksums file that plugin lock extends, # and both must come after genesis (attestation is keyed by the vault nonce). ductile config lock --config "$CFG" ductile plugin lock --all --config "$CFG" # prints a confirm code ductile plugin lock --all --config "$CFG" # Boot the gateway (under your service manager in real deployments): ductile system start --config "$CFG" # ── The finish line ── curl -s http://127.0.0.1:8080/healthz # "posture": "gateway" # Prove the token works and the audit trail is intact: curl -s http://127.0.0.1:8080/jobs -H "Authorization: Bearer $API_TOKEN" ductile system vault-audit --config "$CFG" # genesis + mint recorded ``` What changed at this boot: `DecideBootPosture()` now sees a configured token, the posture is **gateway**, tokens resolve fail-closed, the listener is reserved synchronously (`Bind()` before serve, `runtime.go:822` — #140), and every trigger plane opens. The same transition works *without* a restart-from-scratch during recovery: an armed management-posture box activates on `system reload` once the token ref resolves. ## 7. Good practice — custody, rotation, recovery ### Custody - **Age key**: 0600, owned by the gateway user, on the gateway host only. It is rung 1 — anyone holding it holds the vault. Under enforced privsep, boot reconciles the whole secrets surface to gateway-owned 0600/0700 and **refuses to start** otherwise (`runtime.go:688–692`, #87). - **Admin token**: capture the one-time print into a password manager or 0600 file immediately, then shred the capture file (`shred -u` / `rm -P`). Don't export it in dotfiles; pass it per-invocation. - **API token values** exist in exactly two places: the vault, and the client that uses them. Configs carry only `secret_ref` names — safe to commit, sync, and back up. `config view` redacts sensitive values for the same reason. - **Secrets enter via stdin, never argv** — argv leaks through `ps`, shell history, and audit logs. ### Rotation — each rung rotates independently | Rung | Command | When | | ----------- | --------------------------------------------------- | ------------------------------------------------------------------------------- | | Age key | `ductile secrets rotate` / `vault rotate-key` | Key custody change, scheduled hygiene. Re-encrypts the blob; secrets untouched. | | Admin token | `ductile vault rotate-admin-token` (daemon stopped) | Operator turnover, suspected exposure, or simply lost — rung 1 re-issues it. | | API token | `vault roll` the secret, then `system reload` | Routine. The `secret_ref` in config never changes — only the vault value does. | ### Recovery is the same ladder, climbed again - **Lost API token?** Remove the `api.auth.tokens` block, restart → the daemon is back in management posture; mint a new token over the socket. The bootstrap path *is* the recovery path — it stays tested because it's the same code (#129/#136). - **Lost admin token?** `vault rotate-admin-token` with the age key, daemon stopped. - **Lost age key?** That is the one unrecoverable rung — the vault blob is ciphertext without it. Back the key up under separate custody from the vault file; the pair together is the secret, separately they are inert. ### Sanity rules that bite (from the field) - Unix socket paths cap near **104 bytes** (`sun_path`) — set `api.management_socket` explicitly and keep it short (`/tmp/…` or beside the state DB, `runtime.go:890`). - The daemon **refuses** to boot if the management socket path points at an existing non-socket file — it will not delete your file. - A config cannot *activate* while any configured `secret_ref` is unminted — mint everything you reference before the activating restart, or stage those includes until after bootstrap. - Lock order: `config lock` before `plugin lock` (the latter extends the former's `.checksums`), and both *after* genesis (attestation is keyed by the vault nonce from rung 1). - Verify after every change: `system status` for posture, `system vault-audit` for the append-only credential history, `/healthz` for the outside view. ### Where to go next - [Tutorial: From `system start` to ready-to-trigger](https://ductile.run/tutorials/boot-sequence/index.md) — what happens *inside* each boot - [BOOTSTRAP](https://ductile.run/BOOTSTRAP/index.md) — the canonical from-scratch procedure this page explains - [ADR: Vault credential ladder](https://ductile.run/adr/vault-credential-ladder/index.md) — the design rationale in full - [SECRETS](https://ductile.run/SECRETS/index.md) — the secrets model beyond bootstrap (principals, compose, attestation) - [DEPLOYMENT](https://ductile.run/DEPLOYMENT/index.md) §5b — privsep postures for production hosts ______________________________________________________________________ *Generated 2026-06-10 from the ductile knowledge graph (graphify) + verified against `internal/vault/genesis.go`, `cmd/ductile/vault.go`, `cmd/ductile/runtime.go`, and `docs/BOOTSTRAP.md` @ main (99d909a). Companion page: [boot sequence tutorial](https://ductile.run/tutorials/boot-sequence/index.md).* # Concepts # Ductile — Specification **Version:** 1.0 **Date:** 2026-02-08 **Author:** Matt Joyce **Sources:** RFC-001, RFC-002, RFC-002-Decisions This is the unified, buildable specification for Ductile. It supersedes all prior RFCs and review documents. ______________________________________________________________________ ## 1. Overview ### 1.1 Problem Ductile currently exists as a FastAPI monolith handling health data ETL, LLM processing, and various integrations. Adding new connectors means modifying the core application. Existing integration servers (n8n, Huginn, Node-RED) are too heavy for a personal service. ### 1.2 Solution An automation runtime built for AI agents to operate, diagnose, and extend. Where platforms impose workflow, Ductile provides primitives: a NOUN ACTION CLI, a manifest-contracted plugin protocol, and a queryable execution ledger. A compiled Go core orchestrates polyglot plugins via a subprocess protocol. Simple enough for a human to understand in an afternoon; structured enough for an agent to drive the full lifecycle without supervision. See [`../CONSTITUTION.md`](https://ductile.run/CONSTITUTION.md). ### 1.3 Scope This is a **personal integration server** processing roughly 50 jobs per day. Design decisions are calibrated to that scale. The system runs unattended and must behave predictably under crash, retry, and timeout conditions. ______________________________________________________________________ ## 2. Architecture ```text ┌────────────────────────────────────────────────────────────────────────┐ │ ductile │ │ (Go binary, ~1 process) │ │ │ │ INGRESS (Producers) │ │ ┌───────────┐ ┌──────────┐ ┌──────────┐ ┌───────────┐ │ │ │ Scheduler │ │ Webhook │ │ API │ │ CLI │ │ │ │ (heartbeat) │ Receiver │ │ Server │ │ Commands │ │ │ └─────┬─────┘ └────┬─────┘ └────┬─────┘ └─────┬─────┘ │ │ │ │ │ │ │ │ ▼ ▼ ▼ ▼ │ │ ┌──────────────────────────────────────────────────────────────────┐ │ │ │ SQLite WORK QUEUE (FIFO) │◄┼┐ │ │ (Single source of truth & transactional durability) │ ││ │ └──────────────────────────────┬───────────────────────────────────┘ ││ │ │ ││ │ ▼ ││ │ ┌──────────────────────────────────────────────────────────────────┐ ││ │ │ DISPATCHER LOOP │ ││ │ │ pull job → execute subprocess → collect result → route events │ ││ │ └─────────────────┬────────────────────────────────────────────────┘ ││ │ │ ││ │ │ plugin stdout ││ │ ▼ ││ │ [ emitted events? ] ││ │ │ ││ │ yes ▼ ││ │ ┌───────────┴──────────┐ ││ │ │ Event Router ├─ (calculates downstream dispatches) ──┘│ │ │ (Pipeline Engine) │ │ │ └──────────────────────┘ │ │ │ │ │ │ observation/telemetry │ │ ▼ │ │ ┌──────────────────────┐ ┌──────────────────────┐ │ │ │ Event Hub ├───────►│ TUI / Streaming │ │ │ │ (Observation Pub/Sub)│ │ API Observers │ │ │ └──────────────────────┘ └──────────────────────┘ │ │ │ │ ┌──────────┐ ┌──────────┐ ┌────────────┐ │ │ │ Config │ │ State │ │ Plugin │ │ │ │ Loader │ │ Store │ │ Registry │ │ │ │ (YAML) │ │ (SQLite) │ │ │ │ │ └──────────┘ └──────────┘ └────────────┘ │ └──────────────────────────────────┬─────────────────────────────────────┘ │ stdin/stdout JSON protocol ┌─────────────┼─────────────┐ ▼ ▼ ▼ ┌─────────┐ ┌──────────┐ ┌─────────┐ │withings/ │ │ google/ │ │ notify/ │ │ run.py │ │ run.py │ │ run.sh │ └─────────┘ └──────────┘ └─────────┘ ``` ### 2.1 Key Decisions | Decision | Choice | Rationale | | ------------------ | ------------------------------------------- | ---------------------------------------------------------------------------- | | Core language | Go | Single binary, easy deployment, natural subprocess spawning | | Plugin coupling | Subprocess (JSON over stdin/stdout) | Language-agnostic, fault-isolated, drop-in plugins | | Scheduling | Heartbeat with fuzzy intervals | Human-friendly, avoids thundering herd | | Execution | Bounded Worker Pool | High-throughput, resource-safe, per-plugin concurrency caps | | Routing | Config-declared, queue-centric, exact match | Plugins stay dumb; the core controls flow via the Work Queue | | Pipeline Execution | Async by default; Sync opt-in | Preserves event-driven core while enabling interactive results | | State | SQLite | Proven, zero-ops; append-only `plugin_facts` with derived compatibility view | | Delivery | At-least-once | Plugins own idempotency; core never drops work | | Plugin lifecycle | Spawn-per-command | Eliminates daemon management, memory leaks, zombie processes | ### 2.2 Execution Boundaries: Queue-Centric vs. Event Bus To maintain fault isolation, transactional durability, and predictable recovery, Ductile enforces a strict separation between **Core Execution** and **Observation**: - **Core Execution is strictly Queue-Centric and Router-Driven**: - Ductile does *not* utilize an active in-memory "event bus" for routing execution between plugins. - Instead, the **SQLite-backed Work Queue** serves as the transactional boundary. When a plugin completes and emits events, the **Event Router** (`internal/router`) deterministically evaluates active pipelines/global routing rules, calculates downstream dispatches, and enqueues them immediately back to the SQLite Work Queue. - This design guarantees that every execution transition is durable. If the service crashes or is restarted mid-pipeline, the exact state is recovered from the SQLite Work Queue on startup, avoiding lost messages. - **Observation is Pub/Sub-Driven via the Event Hub**: - For live diagnostics and TUI updates, the **Event Hub** (`internal/events`) provides a lightweight, in-memory pub/sub ring-buffer. - The Dispatcher publishes passive lifecycle telemetry events (e.g. `job.succeeded`, `router.enqueued`) to the Event Hub. - This observation stream is completely decoupled from execution. Slow or blocked subscribers (such as a stalled TUI client or a streaming API consumer) can never block core workers or delay job execution. ### 2.3 Core Component Roles 1. **Ingress Gateway (Producers)**: Handles external and internal triggers. Includes the **Scheduler** (initiating scheduled poll jobs), the **Webhook Receiver** (verifying HMAC signatures on inbound HTTP payloads), the **API Server** (accepting programmatic job/pipeline triggers), and the **CLI**. All producers write transactionally to the SQLite Work Queue. 1. **SQLite Work Queue**: The single, durable source of truth. Manages job FIFO sequencing, attempts, retries, and deduplication states. 1. **Dispatcher**: The bounded worker pool consumer. Dequeues eligible jobs, spawns isolated polyglot plugin subprocesses (communicating via protocol v2), enforces command timeouts, and captures execution results. 1. **Event Router (Pipeline & Event Routing Engine)**: The deterministic routing coordinator. Interprets event payloads emitted by plugins and matches them against YAML-defined DSL pipelines to generate downstream queue enqueues. 1. **Event Hub**: An observational sidecar providing real-time pub/sub streams of telemetry events for external diagnostic clients. 1. **State Store**: Backed by SQLite, maintains both the append-only `plugin_facts` observed history and the auto-derived `plugin_state` compatibility cache views. > [!NOTE] **Remote Event Relay Boundaries**: Remote Event Relay is purposefully omitted from the high-level overview diagram to keep boundaries clear and prevent cluttering core architecture with feature-level extensions. Both inbound and outbound relays leverage existing primitives: > > - **Inbound Relay** is a specialized HTTP API route mounted directly on the **API Server** (`/ingest/peer`). > - **Outbound Relay** executes as an on-demand pipeline step (`relay:` or `uses: core.relay`) dispatched by a standard **Dispatcher** worker via an outbound HTTP client. ### 2.4 Governance Hybrid (The "Control Plane") Ductile employs a "Governance Hybrid" model to manage state across multi-hop plugin chains. Filesystem state is the plugin's concern; the core is dispatch, routing, and durable state. - **Control Plane (Baggage):** Metadata about the execution (e.g., `origin_user_id`, `trace_id`). This data is stored in the `event_context` SQLite ledger. Values become durable only when a pipeline step claims them with `baggage`, and inherited baggage paths are immutable. - **No core-managed data plane.** The core does not provision per-job filesystem workspaces. Plugins that need a scratch path (`mktemp -d`) or a persistent cache (`~/.cache//`) manage it themselves; pipelines that need step-to-step file passing wire absolute paths via `with:` baggage. See `docs/PLUGIN_DEVELOPMENT.md` §9 for guidance. ______________________________________________________________________ ## 3. Work Queue The central abstraction. All producers submit to a single queue. ### 3.1 Producers | Producer | Trigger | | ---------------- | ------------------------------------ | | Scheduler | Heartbeat tick finds a plugin is due | | Webhook receiver | Inbound HTTP event | | Router | Plugin output matches a routing rule | | CLI | Manual `ductile run ` | ### 3.2 Job Model ```text { id: UUID plugin: string command: string (poll | handle) payload: JSON status: queued | running | succeeded | failed | timed_out | dead attempt: int (starts at 1) max_attempts: int (default 4) submitted_by: string (scheduler | webhook | route | cli) dedupe_key: string (optional) created_at: timestamp started_at: timestamp (null until running) completed_at: timestamp (null until terminal) next_retry_at: timestamp (null unless awaiting retry) last_error: text (null unless failed) parent_job_id: UUID (null unless created by routing) source_event_id: UUID (null unless created by routing) } ``` No `priority` field. Jobs are strictly FIFO. ### 3.3 Job State Machine ```text queued → running → succeeded → failed → queued (retry) → dead (max retries exceeded) → timed_out → queued (retry) → dead (max retries exceeded) ``` ### 3.4 Delivery Guarantee **At-least-once.** A job may run more than once (after crash, timeout, or retry). It will never be silently dropped. - Plugins MUST be idempotent, or use `state` to track what they've already processed. - The core provides an opt-in `dedupe_key` field. If a job is enqueued with a `dedupe_key` matching a job that succeeded within the effective dedupe window, it is not enqueued. The drop is logged at `INFO` with the `dedupe_key` and existing job ID. - `dedupe_ttl` is configurable (default 24h) and acts as the default dedupe window. Callers may set a per-enqueue dedupe TTL override when a narrower window is needed (for example, scheduler cadence). When this override is set, enqueue also guards against in-flight duplicates (`queued`/`running`) for that `dedupe_key`. ### 3.5 Dispatch **Bounded Worker Pool.** Ductile uses a global worker pool to process jobs in parallel. This ensures high throughput while preventing resource exhaustion. - **Global Limit:** Controlled by `service.max_workers` (defaults to `max(1, CPU-1)`). Operators can force whole-system serial dispatch by setting `service.max_workers: 1`. - **Plugin Parallelism:** Each plugin can define a `parallelism` limit in its configuration. The plugin manifest's `concurrency_safe` hint is the plugin author's declaration about whether same-plugin concurrent execution is safe; omitted means `true`. - **Smart Dequeue:** The scheduler and dispatcher skip jobs for plugins that have reached their active parallelism cap, ensuring the worker pool remains available for other tasks. Running counts and same-`dedupe_key` execution exclusion are derived from `job_queue`; dispatcher in-memory counters are local worker lifecycle coordination only. Revisit condition: sustained queue wait times exceed 60 seconds with all workers saturated. ### 3.6 Deduplication When a producer enqueues a job with a `dedupe_key`: 1. Determine effective dedupe TTL: per-enqueue override (if provided), otherwise service `dedupe_ttl`. 1. If a per-enqueue override is set, query for an existing `queued` or `running` job with the same `dedupe_key`. 1. Query for a `succeeded` job with the same `dedupe_key` completed within the effective TTL. 1. If either check finds a match: do not enqueue. Log at `INFO`: dedupe_key, existing job ID. 1. If no match is found: enqueue normally. During dispatch, a queued job with a `dedupe_key` is skipped while another job with the same `dedupe_key` **and the same target (`plugin` + `command`)** is `running`. The guard is per-target by design: a single source event that fans out to multiple distinct targets inherits one `dedupe_key`, and those distinct-target siblings must still run concurrently rather than serialise (and starve) behind each other. That execution serialisation is query-backed by `job_queue`, not a separate durable state table. ______________________________________________________________________ ## 4. Scheduler A single internal tick loop manages scheduled `poll` jobs. Each enabled plugin can define one or more schedule entries under `schedules:`. Plugins without schedules are ignored by the scheduler and can still be triggered via webhook, router, CLI, or API. For a full field-by-field reference and behavior details, see [SCHEDULER.md](https://ductile.run/SCHEDULER/index.md). ### 4.1 Schedule Entries Each schedule entry is independent and has its own ID (default: `default`), command, and payload: ```yaml plugins: withings: schedules: - id: hourly every: 1h command: poll payload: source: heartbeat ``` Supported schedule types: - `every`: Interval schedule (`5m`, `15m`, `30m`, `hourly`, `2h`, `daily`, `weekly`, `monthly`). - `cron`: Standard 5-field cron (`min hour dom month dow`). - `at`: One-shot RFC3339 timestamp. - `after`: One-shot delay from service start. ### 4.2 Time Controls Schedule execution can be constrained with time settings: - `jitter`: Random offset per scheduled run. - `only_between`: Time window string (e.g. `"08:00-22:00"`). - `timezone`: IANA timezone for cron/window evaluation. - `not_on`: List of weekdays to skip (`[saturday, sunday]` or `[0-6]`). `preferred_window` exists in config but is not enforced yet. Jitter is computed per scheduled run (not per tick): ```text next_run = last_successful_run + interval + random(-jitter/2, +jitter/2) ``` ### 4.3 Catch-up and Overlap Two per-schedule policies control missed ticks and concurrency: - `catch_up`: `skip` (default), `run_once`, `run_all`. - `if_running`: `skip` (default), `queue`, `cancel`. ### 4.4 Poll Guard The scheduler **must not enqueue** a new `poll` job if there is already a `queued` or `running` `poll` job for that plugin. Configurable per-plugin (default 1): ```yaml plugins: withings: max_outstanding_polls: 1 ``` ______________________________________________________________________ ## 5. Plugin System ### 5.1 Lifecycle: Spawn-Per-Command One process per job. No long-lived plugin processes. 1. Fork the plugin entrypoint. 1. Write JSON request to stdin. 1. Close stdin. 1. Read stdout until EOF or timeout. 1. Capture stderr. 1. Collect exit code. 1. Kill the process if it hasn't exited. Process spawn overhead is ~5ms on Linux — irrelevant when the shortest interval is 5 minutes. **Persistent connections (WebSockets, long-polling) are out of scope.** If needed, run as a separate service that pushes events into Ductile via the webhook endpoint. No streaming plugin mode — not now, not ever for this core. ### 5.2 Commands | Command | Purpose | When | | -------- | ------------------------------- | ------------------------------------- | | `poll` | Fetch data from external source | Scheduled by heartbeat | | `handle` | Process an inbound event | Routed from another plugin or webhook | | `health` | Diagnostic check | On-demand via `ductile status` | | `init` | One-time setup | On first discovery or config change | - `init` is not retried on failure — plugin is marked unhealthy. - `health` is not called on a schedule — it's a diagnostic tool for the operator. ### 5.3 Plugin Directory Structure ```text plugins/ ├── withings/ │ ├── manifest.yaml │ └── run.py ├── google-calendar/ │ ├── manifest.yaml │ └── run.py ├── notify/ │ ├── manifest.yaml │ └── run.sh └── lib/ # shared helpers (e.g. OAuth utilities) ``` ### 5.4 Manifest **Object format:** ```yaml manifest_spec: ductile.plugin manifest_version: 1 name: withings version: 1.0.0 protocol: 2 entrypoint: run.py description: "Fetch health data from Withings API" commands: poll: type: read description: "Fetch latest measurements from Withings API" sync: type: write description: "Push weight data to Withings API" oauth_callback: type: write description: "Handle OAuth2 callback and store tokens" health: type: read description: "Health check" config_keys: required: [client_id, client_secret] optional: [access_token] ``` **Command type semantics:** - `type: read` - No external side effects, idempotent (safe for automated retries) - Examples: poll, fetch, get, list, health - May emit a durable snapshot via `state_updates` (declared as a `fact_outputs` rule for append-only persistence; the compatibility view is updated automatically). - Cannot POST/PUT/DELETE to external APIs - `type: write` - Modifies external state, may not be idempotent - Examples: sync, send, notify, oauth_callback, delete - Default if type not specified (paranoid default) **Purpose:** Enables manifest-driven token scopes (`plugin:ro` vs `plugin:rw`) without hardcoding command knowledge in auth middleware. **Validation:** - `manifest_spec` — must be `ductile.plugin`. - `manifest_version` — must be `1`. - `protocol` — must match a version the core supports. Mismatch → plugin not loaded. - `entrypoint` — mandatory. Core constructs execution path relative to the discovered plugin directory. - `config_keys.required` — validated at load time. Missing keys → plugin not loaded, error logged. - `commands.*.type` — must be `read` or `write` if specified. Invalid type → plugin not loaded. See card #36 (Manifest Command Type Metadata). ### 5.5 Trust & Execution - Plugins MUST live under one of the configured plugin roots. Symlinks resolved, must resolve within an approved root. - `..` in `entrypoint` is rejected (path traversal prevention). - Entrypoint MUST be executable (`chmod +x`). Shebang line handles interpreter selection. - World-writable plugin directories are refused at load time. - Plugins run as the same OS user as the core. Use systemd `User=ductile` to limit blast radius. **Secret delivery is gated by attestation.** A plugin receives no secrets until it is *attested*: `ductile plugin lock` records a keyed fingerprint (keyed-BLAKE3 over the plugin's manifest and entrypoint bytes, keyed by the vault's `core` nonce). At spawn the core re-verifies that fingerprint; a mismatch is a **security event** — a possible binary swap — not a quiet denial: the job fails closed and the mismatch is escalated. Editing the manifest or entrypoint breaks the fingerprint and demands re-attestation. This is why attestation (`plugin lock`) is decoupled from config integrity (`config lock`): a lock taken for an unrelated config edit must never silently bless a swapped binary. Secrets themselves live in the **vault** — an age-encrypted store the daemon alone reads and writes (the sole-writer owner; see `docs/SECRETS.md` for the operator model and lifecycle). The daemon decrypts the blob **once** at startup into that single in-memory owner; boot integrity-verification, the attestation nonce, and secret delivery all read from that one snapshot — so a plugin's fingerprint is verified against the exact vault state that later releases its secrets, with no decrypt-twice window between the two. The trust idea: a secret is bound to an *attested plugin identity*, delivered out-of-band from config and environment, so a swapped or unattested binary cannot inherit credentials. Operators register a plugin as a **principal** and grant it named secrets; at spawn the core **Composes** that principal's authorized, active secrets and delivers them in the request envelope over **stdin only** (§6.1 `secrets`) — never via the environment (secrets are withheld from it; only `service.plugin_env_passthrough` names cross), never on argv. Plugins **consume** secrets; they never register, roll, or revoke them — that is the operator's `ductile vault` surface. Delivery fails closed by design: an *unknown* principal gets no secrets and runs normally, but a *revoked* principal fails the job rather than run secret-less. Every register/set/roll/revoke and every fail-closed denial is recorded in the `vault_audit` fact log (`ductile system vault-audit`). ### 5.6 Timeouts **Defaults:** | Command | Timeout | | -------- | ------- | | `poll` | 60s | | `handle` | 120s | | `health` | 10s | | `init` | 30s | **Enforcement:** 1. Core starts a deadline timer when spawning the process. 1. On timeout: `SIGTERM` to the process group. 1. 5-second grace period. 1. `SIGKILL` if still alive. 1. Job status → `timed_out`, follows retry policy. **Configurable per-plugin:** ```yaml plugins: slow-plugin: timeouts: poll: 300s handle: 300s ``` **Resource caps:** - Max stdout: 10 MiB captured. Exceeding this cap is a protocol/output failure; the captured prefix is kept for diagnostics. - Max stderr: 64 KiB captured for diagnostics. Excess stderr is truncated with a logged warning. ### 5.7 Retry & Backoff - Default: 4 attempts total (1 original + 3 retries). - Backoff: `base * 2^(attempt-1) + random(0, base)` where `base = 30s`. - Retry delays: ~30s, ~1m, ~2m (then dead). **Non-retryable conditions:** - Plugin exits with code `78` (EX_CONFIG from sysexits.h) — configuration error. - Plugin response may include `"retry": false`; core treats this as a compatibility signal, not plugin-owned policy. - All other failures are retried. **Configurable per-plugin:** ```yaml plugins: withings: retry: max_attempts: 5 backoff_base: 60s ``` ### 5.8 Circuit Breaker Configurable consecutive failure threshold per `(plugin, command)` pair. Applies to **scheduler-originated poll jobs only** — webhook-triggered `handle` jobs are not blocked by poll failures. - Default threshold: 3 consecutive failures. - Default reset: 30 minutes. - Manual reset: `ductile system reset `. - Inspect state and transition history: `ductile system breaker [--json]`. - States: `closed` -> `open` -> `half_open`. - When cooldown expires, scheduler allows a single half-open probe poll: - Success closes the circuit and resets failure count. - Failure reopens the circuit. ```yaml plugins: withings: circuit_breaker: threshold: 3 reset_after: 30m ``` ### 5.9 State Model **Config is static. Facts are durable. `plugin_state` is a compatibility view.** - `config` — from `config.yaml`, interpolated with env vars, read-only. Contains credentials, endpoints — things the operator sets. - Config paths (config dir, includes, backups) are local operator-controlled inputs; Ductile does not accept untrusted remote file paths. - `service.allow_symlinks` controls whether symlinks are permitted in config/plugin paths (warnings are always emitted when symlinks are detected). - `plugin_facts` — append-only record of durable plugin observations. Each row carries a stable snapshot the plugin emitted as `state_updates`, plus a `fact_type` declared in the plugin manifest's `fact_outputs` and a Ductile-owned monotonic `seq`. This is the durable record. See [PLUGIN_FACTS.md](https://ductile.run/PLUGIN_FACTS/index.md). - `plugin_state` — single JSON row per plugin maintained as a compatibility/cache view of the latest fact. Existing readers see the same shape they saw before facts existed. The view is rebuilt automatically by core when a fact lands, governed by the manifest's `compatibility_view` declaration (currently `mirror_object`). Plugins that have not declared `fact_outputs` still get write-through behaviour during the compatibility window; new plugins should declare `fact_outputs` rather than treating this row as their durable home. ```sql -- Append-only durable record (primary). plugin_facts ( id INTEGER PRIMARY KEY AUTOINCREMENT, seq INTEGER NOT NULL, -- Ductile-owned monotonic plugin_name TEXT NOT NULL, fact_type TEXT NOT NULL, job_id TEXT, command TEXT, fact_json JSON NOT NULL, created_at TEXT NOT NULL ); -- Compatibility/cache view of the latest fact (derived). plugin_state ( plugin_name TEXT PRIMARY KEY, state JSON NOT NULL DEFAULT '{}', updated_at TIMESTAMP ); ``` **Size limit:** 1 MB per `plugin_state` row. Exceeding this rejects the update and fails the job. The same limit constrains the snapshot a plugin emits, since the compatibility view mirrors it. ### 5.10 OAuth Plugins manage their own OAuth token lifecycle. The core does not understand OAuth. - `client_id`, `client_secret` → `config` (static, set by operator). - `access_token`, `refresh_token`, `token_expiry` → managed by the plugin and emitted as part of its `state_updates` snapshot. The plugin should declare a `fact_outputs` rule so each token-refresh observation is recorded append-only and the compatibility view stays current for downstream readers. - Plugin checks expiry on each invocation, refreshes if needed, returns new tokens via `state_updates`. - Shared OAuth helpers can live in `plugins/lib/`. ______________________________________________________________________ ## 6. Protocol (v2) ### 6.1 Request Envelope (core → plugin) Single JSON object written to plugin's stdin: ```json { "protocol": 2, "job_id": "uuid", "command": "poll | handle | health | init", "config": {}, "state": {}, "context": {}, "event": {}, "secrets": {}, "deadline_at": "ISO8601" } ``` - `event` — present only for `handle`. - `state` — the plugin's current compatibility-view row (the latest fact's snapshot, or write-through state for plugins not yet declaring `fact_outputs`). - `context` — shared metadata (Baggage) carried across the pipeline chain. - `secrets` — the plugin's composed, authorized secrets (a name→value map), present (`omitempty`) only when the plugin is an attested principal with active grants. Delivered here over stdin and nowhere else — distinct from `config`, never in the environment or argv. Read-only: plugins consume secrets; the operator manages them via `ductile vault` (see §5.5). - `deadline_at` — informational. Plugins MAY use it to abandon long-running work early. The core enforces the real deadline externally. ### 6.2 Response Envelope (plugin → core) Single JSON object written to plugin's stdout: ```json { "status": "ok | error", "result": "short human-readable summary", "error": "human-readable message (when status=error)", "retry": true, "events": [], "state_updates": {}, "logs": [] } ``` - `result` — required when `status=ok`. Summarizes what the plugin did. - `retry` — response-envelope compatibility signal. Defaults to `true` if omitted. Set `false` for permanent failures; core still owns the retry decision with exit status, attempts, and config as inputs. - `events` — array of event envelopes (see 6.3). - `state_updates` — the plugin's emitted snapshot. When the manifest declares a matching `fact_outputs` rule, core records this snapshot as an append-only `plugin_facts` row and rebuilds the compatibility view from it. Plugins without a declared `fact_outputs` rule get write-through into `plugin_state` directly during the compatibility window. - `logs` — array of `{"level": "info|warn|error", "message": "..."}`. Optional. Stored with the job record. ### 6.3 Event Envelope Every event emitted by a plugin in the `events` array: ```json { "type": "new_health_data", "payload": {}, "dedupe_key": "withings:weight:2026-02-08" } ``` - `type` — matches `event_type` in routing config. Exact string match. - `payload` — arbitrary JSON, passed to downstream plugin's `handle` command. - `dedupe_key` — optional. Downstream job inherits this as its `dedupe_key`. The core injects when creating downstream jobs: - `source` — plugin name. - `timestamp` — ISO8601. - `event_id` — UUID assigned by the core. ### 6.4 Framing Single JSON object on stdout. Not JSON Lines, not length-prefixed. One request, one response, process exits. ### 6.5 Protocol Mismatch If the request `protocol` field doesn't match what the plugin expects, the plugin SHOULD exit with code `78` (EX_CONFIG) and a clear error on stderr. The core refuses to load plugins whose manifest declares a `protocol` version it doesn't support. ______________________________________________________________________ ## 7. Routing Plugin chaining is declared in config, not by plugins. Plugins produce typed events; the config says where they go. ### 7.1 Config ```yaml routes: - from: withings event_type: new_health_data to: health-analyzer - from: health-analyzer event_type: alert to: notify ``` ### 7.2 Semantics - **Fan-out:** A single event can match multiple routes. All matching routes produce a job. - **No match:** Logged at DEBUG, dropped. Not an error. - **Matching:** Exact string match on `event_type` only. No wildcards, no regexes, no glob patterns. - **No conditional filters.** No `payload.severity == "high"`. If you need conditional logic, put it in the receiving plugin — it can inspect the payload and no-op. ### 7.3 Traceability When the router creates a downstream job from an event: - `parent_job_id` is set to the producing job's ID. - `source_event_id` is set to the core-assigned `event_id`. ______________________________________________________________________ ## 8. Pipelines (DSL) Pipelines provide a higher-level orchestration layer over raw routes, using a GitHub Actions-inspired notation. ### 8.1 Schema ```yaml pipelines: - name: youtube-summary on: discord.command.youtube # Trigger event type execution_mode: synchronous # Optional: async | synchronous timeout: 3m # Optional: duration (default 30s) steps: - id: download # Optional uses: youtube.download # plugin.command - id: summarize uses: fabric.summarize - id: notify uses: discord.respond ``` ### 8.2 Execution Modes - **async (default):** Fire-and-forget. The API returns `202 Accepted` with a `job_id` immediately. Dispatcher handles jobs as they come. - **synchronous (opt-in):** The API caller "stays on the line". The gateway waits for the entire execution tree (all steps) to reach a terminal state before responding with aggregated results. ### 8.3 Guarded Bridge The engine remains event-driven and asynchronous internally. Synchronous behavior is implemented as a "Guarded Bridge" at the API layer: 1. Dispatcher provides completion channels for job trees. 1. API handler blocks on these channels. 1. If `timeout` is exceeded, the bridge "breaks" and returns `202 Accepted` with the root `job_id`, allowing the client to poll for completion. ______________________________________________________________________ ## 9. API Endpoints The HTTP API allows external systems (LLMs, scripts, other services) to programmatically trigger plugin execution and retrieve job results. ### 9.1 Configuration ```yaml api: enabled: true listen: "localhost:8080" auth: tokens: - token: ${ADMIN_API_TOKEN} scopes: ["*"] ``` ### 9.2 Primary Trigger Endpoints The API exposes two first-class trigger paths: - `POST /plugin/{plugin}/{command}`: direct plugin execution (no pipeline routing), returns `202 Accepted`. - `POST /pipeline/{pipeline}`: explicit pipeline orchestration, returns `202 Accepted` by default and `200 OK` for synchronous pipelines. See `docs/API_REFERENCE.md` for full examples and response schemas. ### 9.3 GET /job/{job_id} Retrieves the status and results of a previously triggered job. **Request:** - URL param: `{job_id}` - UUID returned from one of the POST trigger endpoints - Header: `Authorization: Bearer ` **Response (200 OK - queued):** ```json { "job_id": "uuid-v4", "status": "queued", "plugin": "plugin_name", "command": "command_name", "created_at": "2026-02-09T10:00:00Z" } ``` **Response (200 OK - running):** ```json { "job_id": "uuid-v4", "status": "running", "plugin": "plugin_name", "command": "command_name", "started_at": "2026-02-09T10:00:05Z" } ``` **Response (200 OK - completed):** ```json { "job_id": "uuid-v4", "status": "completed", "plugin": "plugin_name", "command": "command_name", "result": { "status": "ok", "result": "Plugin executed successfully", "state_updates": {"last_run": "2026-02-09T10:00:10Z"}, "logs": [{"level": "info", "message": "Plugin executed successfully"}] }, "started_at": "2026-02-09T10:00:05Z", "completed_at": "2026-02-09T10:00:10Z" } ``` **Error Responses:** - `401 Unauthorized` - Missing or invalid token - `404 Not Found` - Job ID not found ### 9.4 Authentication & Authorization **Bearer token authentication** with scoped permissions. **Token registry** (`tokens.yaml`): - Multiple tokens with individual scope definitions - Each token references a scope file (JSON) - BLAKE3 hash ensures scope file integrity - Environment variable references for keys (never plaintext) **Scope types (current):** - `plugin:ro`, `plugin:rw` - Plugin and pipeline trigger permissions - `jobs:ro`, `jobs:rw` - Job read/write permissions - `events:ro`, `events:rw` - Event stream permissions - `*` - Full admin access **Example tokens.yaml:** ```yaml tokens: - name: admin-cli key: ${ADMIN_API_KEY} scopes_file: scopes/admin-cli.json scopes_hash: blake3:a3f8c2d9... - name: github-integration key: ${GITHUB_API_KEY} scopes_file: scopes/github-integration.json scopes_hash: blake3:b4e9d3c0... ``` **Example scope file (scopes/github-integration.json):** ```json { "scopes": [ "read:jobs", "read:events", "github-handler:rw", "withings:ro" ] } ``` **Authorization middleware:** 1. Extract bearer token from `Authorization` header 1. Lookup token in registry 1. Load and verify scope file (BLAKE3 hash check) 1. Normalize implied read-from-write scopes 1. Check if requested action matches any granted scope 1. Return 403 if denied, proceed if allowed Tokens should be stored in environment variables and interpolated (for example `${ADMIN_API_TOKEN}`). - All API requests must include `Authorization: Bearer ` header - Invalid or missing token returns `401 Unauthorized` **Rotation.** Secret values rotate through the vault — `ductile vault roll` (one secret) and `roll-principal` (every auto-pattern secret a principal holds); the at-rest age key rotates with `ductile vault rotate-key` (mint → re-encrypt → verify-before-retire, daemon stopped). See `docs/SECRETS.md`. **Two credential domains.** The `tokens.yaml` scoped tokens above authorize the *general* API. The vault management routes (`POST /vault/*`) are a **separate domain**: they authenticate against the **vault-resident admin token** (minted by `ductile vault init`, presented as `Authorization: Bearer ` / `DUCTILE_VAULT_TOKEN`), and the config `tokens.yaml` tokens — even `*` — are *rejected* there. Rationale: vault-write authority must not depend on a config-defined token an operator could grant by editing a file; it lives inside the encrypted store and is rotatable by a normal vault write, and a revoked admin token authenticates no one. ### 9.5 Resource Guarding (Synchronous Pipelines) To prevent HTTP worker exhaustion, synchronous pipelines are governed by a semaphore: - **api.max_concurrent_sync:** Max number of simultaneous blocking API calls (default 10). - **api.max_sync_timeout:** Hard limit on pipeline timeout to prevent zombie connections. ### 9.6 Use Cases - **LLM Tool Calling:** LLM agents can call `/plugin` for atomic actions and `/pipeline` for orchestrated workflows - **External Automation:** Scripts, cron jobs, or other services can trigger plugins programmatically - **Result Polling:** External systems can poll /job/{id} to wait for async plugin execution completion - **Manual Testing:** Developers can trigger plugins via curl without waiting for scheduler ______________________________________________________________________ ## 10. Webhooks For operator setup and example requests, see [WEBHOOKS.md](https://ductile.run/WEBHOOKS/index.md). ### 10.1 Listener ```yaml webhooks: listen: 127.0.0.1:8081 endpoints: - path: /hook/github plugin: github-handler secret_ref: github_webhook_secret signature_header: X-Hub-Signature-256 max_body_size: 1MB ``` ### 10.2 Security HMAC-SHA256 signature verification is **mandatory** for all webhook endpoints. 1. Read raw request body (up to `max_body_size`, default 1 MB). 1. Resolve `secret_ref` from tokens.yaml and compute `HMAC-SHA256(secret, raw_body)`. 1. Compare against the signature header (configurable name per endpoint). 1. Reject with `403` if invalid. No error details in response. 1. Reject with `413` if body exceeds `max_body_size`. No replay protection in V1. No rate limiting in V1 (proxy responsibility if fronted by reverse proxy). ### 10.3 Health Endpoint `/healthz` on the webhook listener port: ```json { "status": "ok", "uptime_seconds": 3600, "queue_depth": 2, "plugins_loaded": 5, "plugins_circuit_open": 0 } ``` No authentication. Localhost only. Useful for systemd watchdog and operator checks. ______________________________________________________________________ ## 11. Operations ### 11.1 Single-Instance Lock PID file with `flock(LOCK_EX | LOCK_NB)`: 1. Create/open `/ductile.lock`. 1. Acquire `flock`. Fail → log error, exit 1. 1. Write current PID. 1. Lock held for process lifetime. Kernel releases on crash/exit. ### 11.2 Crash Recovery On startup: 1. Open the SQLite database. 1. Acquire the exclusive lock. 1. Find all jobs with `status = running` — orphans from a prior crash. 1. For each orphan: increment `attempt`, set `status = queued` if under `max_attempts`, else `status = dead`. 1. Log each recovered job at WARN level. 1. Resume normal dispatch. ### 11.3 Config Reload Send `SIGHUP` to the running process (found via PID file) to reload config. On SIGHUP: 1. Parse new config. If invalid → log error, keep old config. 1. In-flight jobs continue with existing config snapshot. 1. Scheduler updates intervals/jitter for all plugins. 1. Router updates routing rules. 1. Plugin config changes take effect on next dispatch. 1. Newly added plugins discovered → `init` runs. 1. Removed/disabled plugins → queued jobs cancelled (status → `dead`), no new jobs enqueued. ### 11.4 Logging **Core logs:** JSON to stdout. Fields: `timestamp`, `level`, `component`, `plugin` (when relevant), `job_id` (when relevant), `message`. **Plugin stderr:** Captured. Always. Stored in `job_log` (capped at 64 KB). Logged at WARN to core log stream. **Plugin stdout:** Reserved exclusively for protocol response. Stored verbatim on completion in `job_log.result` (JSON). Non-JSON on stdout is a protocol error — job fails, stderr + stdout captured for debugging. **Redaction:** The core owns secret *handling*, not secret *hygiene in plugin output*. It withholds secrets from the plugin environment (delivery is stdin-only, §5.5/§6.1) and redacts secret values in `config show` / `/config/view`. But a plugin that echoes its own `secrets` to stdout/stderr defeats this — the core stores plugin output verbatim. So: don't log secrets *in the plugin*; the core will not scrub a plugin that leaks its own credentials. ### 11.5 Job Log Retention Pruned on every scheduler tick: ```sql DELETE FROM job_log WHERE completed_at < datetime('now', '-30 days') ``` Default 30 days. Configurable via `service.job_log_retention`. ### 11.6 CLI ```text ductile system start # run the service (foreground) ductile run # manually run a plugin once ductile status # show plugin compatibility views, queue depth, last runs # send SIGHUP to reload config without restart ductile system reset # reset circuit breaker for a plugin ductile plugins # list discovered plugins ductile logs [plugin] # tail structured logs ductile queue # show pending/active jobs ``` ### 11.7 CLI Principles To ensure predictability and safety for both human and LLM operators, all CLI commands MUST adhere to the standards defined in `docs/CLI_DESIGN_PRINCIPLES.md`. Core requirements: - **Hierarchy:** Strict **NOUN ACTION** pattern. - **Verbosity:** mandatory `-v` / `--verbose` flags. - **Safety:** mandatory `--dry-run` for mutations. - **Machine-Readability:** mandatory `--json` for status and inspection. ______________________________________________________________________ ## 12. Database Schema Ductile uses a persistent SQLite database for the work queue, append-only logs, metrics telemetry, and state store. ### 12.1 Tables ```sql -- Job queue (active and pending jobs) job_queue ( id TEXT PRIMARY KEY, -- UUID plugin TEXT NOT NULL, command TEXT NOT NULL, -- poll | handle payload JSON, status TEXT NOT NULL, -- queued | running | succeeded | failed | timed_out | dead attempt INTEGER NOT NULL DEFAULT 1, max_attempts INTEGER NOT NULL DEFAULT 4, submitted_by TEXT NOT NULL, -- scheduler | webhook | route | cli dedupe_key TEXT, created_at TEXT NOT NULL, -- ISO8601 started_at TEXT, completed_at TEXT, next_retry_at TEXT, last_error TEXT, parent_job_id TEXT, -- FK to job_queue.id source_event_id TEXT, -- UUID assigned by core event_context_id TEXT, -- ID of associated baggage row enqueued_config_snapshot_id TEXT, -- Config snapshot active at enqueue started_config_snapshot_id TEXT -- Config snapshot active at start ); -- Append-only job status transitions history job_transitions ( id INTEGER PRIMARY KEY AUTOINCREMENT, job_id TEXT NOT NULL, -- Joined to job_queue.id (no FK to allow pruning) from_status TEXT, to_status TEXT NOT NULL, reason TEXT, created_at TEXT NOT NULL -- ISO8601 ); -- Append-only job execution attempt log job_attempts ( id INTEGER PRIMARY KEY AUTOINCREMENT, job_id TEXT NOT NULL, -- Joined to job_queue.id (no FK to allow pruning) attempt INTEGER NOT NULL, created_at TEXT NOT NULL, -- ISO8601 UNIQUE(job_id, attempt) ); -- Config snapshots (durable record of active configurations) config_snapshots ( id TEXT PRIMARY KEY, config_hash TEXT NOT NULL, source_hash TEXT, source_path TEXT, source TEXT, reason TEXT NOT NULL, loaded_at TEXT NOT NULL, ductile_version TEXT, binary_path TEXT, snapshot_format INTEGER NOT NULL DEFAULT 1, semantics JSON, plugin_fingerprints JSON, sanitized_config JSON, secret_fingerprints JSON ); -- Append-only durable plugin record (primary). plugin_facts ( id TEXT PRIMARY KEY, seq INTEGER, -- Ductile-owned monotonic sequence plugin_name TEXT NOT NULL, fact_type TEXT NOT NULL, -- e.g. ".snapshot" job_id TEXT NOT NULL, command TEXT NOT NULL, fact_json JSON NOT NULL, created_at TEXT NOT NULL ); -- Compatibility/cache view of the latest fact (derived). -- One row per plugin. Existing readers see the same shape as before facts existed. plugin_state ( plugin_name TEXT PRIMARY KEY, state JSON NOT NULL DEFAULT '{}', updated_at TEXT ); -- Job log (completed jobs for audit/debugging) job_log ( id TEXT PRIMARY KEY, plugin TEXT NOT NULL, command TEXT NOT NULL, status TEXT NOT NULL, result TEXT, -- protocol response JSON attempt INTEGER NOT NULL, submitted_by TEXT NOT NULL, created_at TEXT NOT NULL, completed_at TEXT NOT NULL, last_error TEXT, stderr TEXT, -- capped at 64 KB parent_job_id TEXT, source_event_id TEXT ); -- Circuit breaker state for scheduler poll guard circuit_breakers ( plugin TEXT NOT NULL, command TEXT NOT NULL, -- poll state TEXT NOT NULL, -- closed | open | half_open failure_count INTEGER NOT NULL DEFAULT 0, opened_at TEXT, -- ISO8601 last_failure_at TEXT, -- ISO8601 last_job_id TEXT, -- latest processed scheduler poll job id updated_at TEXT NOT NULL, -- ISO8601 PRIMARY KEY(plugin, command) ); -- Append-only circuit breaker transition facts. -- circuit_breakers remains the current-state compatibility/cache row. circuit_breaker_transitions ( id TEXT PRIMARY KEY, plugin TEXT NOT NULL, command TEXT NOT NULL, from_state TEXT, -- closed | open | half_open | NULL to_state TEXT NOT NULL, -- closed | open | half_open failure_count INTEGER NOT NULL DEFAULT 0, reason TEXT NOT NULL, -- failure_threshold | success | cooldown_elapsed | manual_reset job_id TEXT, created_at TEXT NOT NULL -- ISO8601 ); ``` ______________________________________________________________________ ## 13. Configuration Reference Ductile uses a **Monolithic Runtime** compiled from a modular, **Tiered Directory** structure. ### 13.1 Overview For the complete configuration specification, including file formats, merge logic, and integrity verification rules, see:\ 👉 **[docs/CONFIG_REFERENCE.md](https://ductile.run/CONFIG_REFERENCE/index.md)** ### 13.2 Key Principles - **Include-Based Modularity:** Configuration is loaded from `config.yaml` plus any files or directories listed in `include:`. - **Multi-Root Plugin Discovery:** `plugin_roots` is the source of truth; roots are scanned in order and first match wins on duplicate plugin names. - **Pipeline Discovery Flow:** Pipelines are loaded from included YAML files (or include directories) that define `pipelines:` entries. - **Tiered Integrity:** High-security files (auth/webhooks) require a valid BLAKE3 hash in `.checksums` to start. Operational files (settings/routes) log warnings if hashes are missing or mismatched. - **Monolithic Grafting:** At runtime, all included files are merged into a single internal configuration object following strict precedence rules (later entries override earlier ones). - **Environment Interpolation:** Secrets are injected via `${VAR}` placeholders, which are interpolated after hash verification but before parsing. - **Default Permissions:** Config directories are created with `0700`. Config files and lock files default to `0600`; operators may relax permissions explicitly for shared environments. - **Secret Redaction:** CLI config inspection outputs redact token keys and webhook secrets; secrets are only shown at creation time. ## 14. Deployment ### 14.1 Systemd Unit ```ini [Unit] Description=Ductile After=network.target [Service] Type=simple ExecStart=/usr/local/bin/ductile system start --config /etc/ductile/config.yaml ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure User=ductile Group=ductile [Install] WantedBy=multi-user.target ``` ### 14.2 Development Run `ductile system start` directly. No systemd required. ______________________________________________________________________ ## 15. Project Layout ```text ductile/ ├── cmd/ │ └── ductile/ │ └── main.go ├── internal/ │ ├── config/ │ ├── queue/ │ ├── scheduler/ │ ├── dispatch/ │ ├── plugin/ │ ├── state/ │ ├── api/ │ ├── webhook/ │ └── router/ ├── plugins/ │ └── example/ │ ├── manifest.yaml │ └── run.py ├── config.yaml ├── go.mod ├── go.sum └── Makefile ``` ______________________________________________________________________ ## 16. Implementation Phases | Phase | Sprint | Scope | Status | | ------------------------- | ------ | ------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | | 1. Skeleton | 0 | Go scaffold, CLI, config loader, SQLite state, plugin discovery | ✅ Complete | | 2. Core Loop | 1 | Work queue, heartbeat scheduler with fuzzy intervals, dispatch loop, plugin protocol, crash recovery | ✅ Complete | | 3. API Triggers | 2 | HTTP server with chi router, POST /plugin and POST /pipeline, GET /job, Bearer token auth, job result storage | ✅ Complete | | 4. Routing | 3 | Config-declared event routing, downstream enqueuing, event_id traceability | ✅ Complete | | 5. Webhooks | 3 | HTTP listener, HMAC verification, /healthz, route inbound webhooks to plugins | ✅ Complete | | 6. Reliability Controls | 4 | Circuit breaker, retry with exponential backoff, deduplication enforcement | ✅ Complete | | 7. Pipeline Orchestration | 4 | Sync/Async execution modes, Guarded Bridge, YAML DSL, completion channels | ✅ Complete | | 8. CLI & Ops | 5 | Status/run/reload/reset/plugins/queue/logs commands, systemd unit | 🔄 In Progress (Status: ✅ Status implemented) | | 9. First Plugins | 6 | Port Withings & Garmin from existing Ductile, notify plugin | Planned | **Note:** Phase 3 (API Triggers) was prioritized before Routing and Webhooks to enable LLM-driven automation via curl-based triggers. This allows external systems to programmatically enqueue jobs and retrieve results immediately, accelerating the path to production use cases. ______________________________________________________________________ ## 17. Deferred Decisions | Topic | Rationale | | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Two-tier stderr/stdout caps (capture vs persistence) | Current spec is workable. Clarify post-V1 if storage becomes a concern. | | `protocol` field in response envelope | Accretive addition; back-compatible with plugins that omit it. | | Replay protection for webhooks | Provider-specific. Add per-plugin if a provider requires it. | | Rate limiting on webhook listener | Proxy responsibility. Core doesn't duplicate concerns it can't own. | | Secret redaction in plugin output | Core withholds secrets from env and redacts them in config views (§11.4); it does not scrub a plugin that echoes its own `secrets` to stdout/stderr — fix the plugin. | | Streaming / long-lived plugin mode | Out of scope permanently. If it needs to stream, it's not a plugin. | | Priority queues / multi-lane dispatch | Revisit only if daily jobs exceed 500 or median wait exceeds 30s. | | Router query language / payload filters | Put conditional logic in the receiving plugin. | # Ductile: Pipelines & Orchestration (DSL Reference) Ductile uses a YAML-based Domain Specific Language (DSL) to define event-driven workflows. Pipelines transform atomic **Connectors** into complex, multi-hop **Orchestrations**. ______________________________________________________________________ ## 1. Top-Level Structure A pipeline file (e.g., `pipelines.yaml`) contains an array of pipeline definitions. ```yaml pipelines: - name: my-workflow # Required: Unique identifier on: my.event.type # Required: Trigger event type execution_mode: async # Optional: async (default) | synchronous timeout: 30s # Optional: For synchronous execution steps: # Required: Sequential steps - uses: my-plugin ``` ______________________________________________________________________ ## 2. Pipeline Properties | Field | Type | Description | | ---------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | String | A unique name for the pipeline. Used for logging and API triggers. | | `on` | String | The event type that triggers this pipeline. Must match exactly. | | `on-hook` | String | Lifecycle signal that triggers this pipeline (`job.completed` / `job.failed` / `job.timed_out`). Mutually exclusive with `on`. | | `from_plugin` | String | **Optional source-plugin selector.** When set, the trigger or hook signal only matches when the upstream source plugin is exactly this plugin. Empty (default) preserves today's behaviour — match regardless of source plugin. See §2.2. | | `if` | Condition | **Optional pipeline-level trigger predicate.** Evaluated against the event's payload (and the upstream job's accumulated durable context, when available) after the trigger/hook name match; a false result skips dispatch entirely. Same shape as step-level `if:` (see §3.6). | | `max_depth` | Integer | **Optional author-set route depth cap.** Overrides the auto-computed cap. `0` means *unlimited*. Negative values are rejected at config load. | | `execution_mode` | Enum | `async` (fire-and-forget) or `synchronous` (API blocks for result). | | `timeout` | Duration | Max time to wait for a `synchronous` pipeline (e.g., `5s`, `2m`). | | `steps` | Array | The list of steps to execute in order. | ### 2.1 Pipeline-level `if:` vs. step-level `if:` Both `if:` blocks share the same predicate engine — atomic `path/op/value` plus `all/any/not`. They differ in *where* they evaluate: | Surface | Evaluated when | Scope | Effect on false | | ----------------------- | -------------------------------------------------- | ------------------------------------- | -------------------------------------------------------------------- | | Pipeline-level `if:` | Trigger/hook name has matched, before any dispatch | `payload`, `context` (when available) | No dispatch at all — no `core.switch`, no plugin spawn | | Step-level `if:` (§3.6) | At each step, after upstream steps run | `payload`, `context`, `config` | Step bypassed via internal `core.switch`; downstream steps still run | Use pipeline-level `if:` to **suppress dispatch** when an event isn't relevant to a pipeline at all. Use step-level `if:` to **gate a step** within a pipeline that is otherwise running. #### Context availability at trigger time `context.*` paths in a pipeline-level `if:` resolve against the upstream job's *accumulated durable context* — the same baggage view that downstream pipeline steps see. Context is available when the routed event was emitted by a plugin running inside an existing pipeline; it is empty for events from the scheduler tick, webhook ingress, and direct API triggers. A predicate that tests `context.*` against an absent context simply evaluates to false (no special-case error), the same way a payload predicate against an absent key returns false. For hook pipelines (`on-hook:`), context is **never** available at hook fire time: hooks fire only for root jobs that have no upstream context of their own. A `context.*` path in a hook pipeline's `if:` could therefore never match, so it is **rejected at config load** with `on-hook trigger predicate cannot reference context.* — context is not available at hook time`. Scope hooks with `payload.*` predicates (and `from_plugin:`, §2.2) instead. A pipeline may use **both** in the same definition. ```yaml - name: repo-changelog on: git_repo_sync.completed if: # pipeline-level: skip dispatch when no work path: payload.new_commits op: eq value: true steps: - id: changelog uses: changelog_microblog - id: commit uses: git_commit_push if: # step-level: only commit if the step before path: payload.changed # actually produced changes op: eq value: true ``` `max_depth` is a separate concern: it caps how many internal `core.switch` hops a pipeline may chain before the runtime considers the route exhausted. Author-setting it is rare; the auto-computed value is correct in almost all cases. Set `max_depth: 0` only when you have a deliberate need for unbounded recursion through `call:`, and you have read §6.4 of this doc. #### Hook-trigger predicate (`on-hook:` + `if:`) Lifecycle hook pipelines (`on-hook: job.completed | job.failed | job.timed_out`) fire for **every** matching lifecycle event across the whole runtime. Without a predicate, this is fundamentally noisy. A pipeline-level `if:` is the correct surface for scoping a hook pipeline: ```yaml - name: notify-on-real-failure on-hook: job.failed if: not: path: payload.plugin op: in value: [check_youtube, jina-reader] # known-noisy plugins steps: - uses: discord_notify ``` Hook predicates evaluate against the lifecycle event's payload, which includes the plugin name, status, attempt count, and other lifecycle fields documented in §9. ### 2.2 `from_plugin:` source-plugin selector Pipelines that fire from plugin-emitted events (`on:`) or lifecycle hooks (`on-hook:`) can be scoped to a single upstream plugin with the optional `from_plugin:` field. When set, the route only matches when the event's source plugin is exactly the named plugin. ```yaml - name: page-on-claude-failure on-hook: job.failed from_plugin: claude_harvest steps: - uses: pagerduty_notify ``` `from_plugin:` is a positive assertion: an empty source plugin (e.g. webhook ingress, scheduler tick) never matches a route that declares one. Use this to keep a hook pipeline silent for unrelated plugins without smuggling the filter through `if:` against `payload.plugin`. `from_plugin:` and `if:` compose. The selector is checked first; the predicate runs only when the source plugin matches. ```yaml - name: page-on-high-severity-claude-failure on-hook: job.failed from_plugin: claude_harvest if: path: payload.severity op: eq value: high steps: - uses: pagerduty_notify ``` For multi-plugin scoping, prefer either multiple narrowly-scoped pipelines or an `if:` predicate against `payload.plugin`. The single-plugin selector is intentional; a generic multi-value source matcher is deferred. #### Inspection The richer compiled-route shape (including `source_plugin` and `if`) is exposed via the `GET /config/view` API endpoint under the `compiled_routes` key, keyed by pipeline name. Operators can use this to answer: - what signal does this route match? - does it require a source plugin? - what predicate is evaluated? - what depth guard exists? ______________________________________________________________________ ## 3. Step Types Each step in a pipeline must perform exactly **one** of the following actions: ### 3.1 `uses` (Invoke Plugin) Calls a specific plugin or alias. This is the most common step. ```yaml steps: - id: download-step # Optional: Unique ID within the pipeline uses: youtube-dl ``` ### 3.2 `call` (Invoke Pipeline) Calls another pipeline by name, inheriting the current baggage. This promotes logic reuse. ```yaml steps: - call: standard-summarizer ``` ### 3.3 `steps` (Nested Sequence) Groups multiple steps together. Useful for organization or within a `split`. ```yaml steps: - steps: - uses: step-1 - uses: step-2 ``` ### 3.4 `split` (Parallel Fan-out) Executes multiple steps or sub-pipelines in parallel. Branches share the same **Baggage** but otherwise execute independently. Plugins that need per-branch filesystem isolation manage it themselves (e.g. via `mktemp -d`). ```yaml steps: - uses: processor - split: - uses: discord-notifier - uses: s3-archiver ``` ### 3.5 `relay` (Remote Event Relay) `relay` delivers a projected event to a named remote Ductile instance. The step is declarative: it refers to `relay-instances.yaml` by stable instance name and does not expose URLs or secrets in pipeline logic. ```yaml steps: - id: relay-to-lab relay: to: lab event: backup.ready dedupe_key: payload.archive_id with: archive_id: payload.archive_id archive_path: payload.archive_path checksum: payload.checksum baggage: trace_id: context.trace_id ``` Rules: - `to` is the outbound relay instance name. - `event` is the remote event type. - `dedupe_key` is optional and resolves from `payload.*` or `context.*`. - `with` is the remote event payload projection. If omitted, the current event payload is relayed. - `baggage` is an optional explicit projection into the relay envelope baggage. ### 3.6 `if` (Conditional Step Execution) A step may include an optional structured `if` object. Authored conditions compile into an internal `core.switch` hop. The switch evaluates the condition against the current scope and then either dispatches the gated step or bypasses it without spawning the gated plugin. `if` must be exactly one of: - atomic predicate: `path`, `op`, optional `value` - `all: [...]` - `any: [...]` - `not: ` Atomic example: ```yaml steps: - uses: discord-notify if: path: payload.status op: eq value: error ``` Composite example: ```yaml steps: - uses: long-video-handler if: all: - path: payload.kind op: eq value: video - path: payload.duration_sec op: gte value: 30 ``` Supported operators in v1: - `exists` - `eq` - `neq` - `in` - `gt` - `gte` - `lt` - `lte` - `contains` (case-insensitive string contains) - `startswith` (case-insensitive string prefix) - `endswith` (case-insensitive string suffix) - `regex` (Go regexp full-string match; use inline flags like `(?i)` for case-insensitive patterns) Path roots allowed in v1: - `payload.*` - `context.*` - `config.*` Semantics: - typing is strict - numeric operators require numeric operands - string operators require string path values and string comparison values - no implicit string-to-number coercion - missing paths resolve to absent for `exists`, otherwise compare as `null` - invalid conditions fail at pipeline load time - branch decisions are observable as internal `ductile.switch.true` / `ductile.switch.false` events - a `false` result bypasses the step and continues from the nearest downstream route ### 3.7 `with` (Payload Remap for `uses` Steps) `with` lets a `uses` step add or override top-level payload keys immediately before the plugin is spawned. ```yaml steps: - id: notify uses: discord_notify with: message: "{payload.stdout}" channel: "{context.origin_channel_id}" summary: "Build finished: {payload.status}" ``` Rules: - `with` is only valid on `uses` steps. - Each value is evaluated against a snapshot of the merged `payload.*` and `context.*` scope. - `context.*` values only exist if an upstream step claimed them with `baggage`. - A pure reference such as `{payload.count}` preserves the original type. - A mixed template such as `Build: {payload.status}` produces a string. - `with` entries do not see each other's output. They all read from the same pre-remap snapshot. - Invalid paths or malformed templates fail the job. Ductile does not silently substitute `null` or `""`. ### 3.8 `baggage` (Explicit Durable Context for `uses` Steps) `baggage` names the facts that should survive beyond the immediate plugin request. It is only valid on `uses` steps. Payload is per-hop. A plugin may emit useful fields, but those fields are not durable unless the pipeline author claims them with `baggage`. Plugin manifests help authors choose these mappings. Names-only `values.consume` says what request payload names a command consumes, and `values.emit` says what event payload names a command emits. The author still chooses durable names: ```yaml # plugin manifest commands: - name: handle type: write values: consume: - payload.url emit: - event: content_ready values: - payload.url - payload.content_hash - payload.truncated ``` ```yaml # pipeline steps: - id: summarize uses: fabric baggage: web.url: payload.url web.content_hash: payload.content_hash web.truncated: payload.truncated ``` ```yaml steps: - id: process uses: content_processor baggage: content.text: payload.content content.input_status: payload.status - id: notify uses: discord_notify baggage: processor.result: payload.result processor.exit_code: payload.exit_code with: message: "{payload.result}" ``` Rules: - `baggage` is only valid on `uses` steps. - Mapping keys are durable dotted paths such as `content.text` or `processor.result`. - Mapping values are source expressions resolved from `payload.*` or `context.*`. - Missing source paths fail the job or trigger. Ductile does not silently skip missing durable claims. - Durable context is deep-accreted. A downstream step may add new paths, but may not change an inherited path to a different value. - Repeating the same inherited value is allowed. Bulk import is available when an object should be promoted under a named namespace: ```yaml steps: - id: transcribe uses: whisper baggage: from: payload.metadata namespace: whisper ``` This imports `payload.metadata` as `context.whisper.*`. The namespace is required until plugin manifest default namespaces exist. Without a namespace, Ductile rejects the claim rather than placing generic keys at the durable root. Use `baggage` for durable facts and `with` for the next plugin request. These are separate concerns: ```yaml steps: - id: notify uses: discord_notify baggage: status.current: payload.status with: message: "Status changed to {payload.status}" ``` In this example, `status.current` is durable. `message` is just the request sent to `discord_notify`. ______________________________________________________________________ ## 4. How Data Flows ### 4.1 Filesystem (Plugin-managed) - Ductile core does not provision a workspace directory for jobs. - If Step A needs to hand a file to Step B, the producing plugin writes to a path it chooses (e.g. under `~/.cache//` or a `mktemp -d`) and the path is propagated as baggage via `with:`. - See `docs/PLUGIN_DEVELOPMENT.md` §9 for plugin-side guidance. ### 4.2 The Control Plane (Baggage) - Metadata (JSON) is stored in the `event_context` database table. - Every step receives durable context claimed by upstream steps. - New durable facts are claimed explicitly with `baggage`. - Existing durable paths are immutable: descendants may add new paths or repeat the same value, but may not rewrite inherited facts. - If a step does not declare `baggage`, it contributes no new durable facts. Its event payload is still the immediate input to downstream routing and plugin execution, but it is not written into `event_context` implicitly. ### 4.3 Results & Payloads - The event `payload` from Step A is passed to Step B as the immediate payload. - `with` can reshape that immediate payload before the plugin is spawned. - `baggage` can promote selected immediate payload fields into durable context. - In `synchronous` mode, the final API response aggregates the results from every step. - **Synthetic events:** If a pipeline step completes successfully but emits no events, Ductile routes a synthetic `ductile.step.succeeded` event to ensure downstream sequential steps are still triggered. ______________________________________________________________________ ## 5. Decision Making Ductile supports two kinds of decision making: ### 5.1 Native step gating with `if` Use `if` when you want to decide whether a step should run based on the current payload, accumulated context, or plugin config. Internally Ductile inserts a `core.switch` decision hop so the branch is explicit and observable. ### 5.2 Event-driven branching Ductile also supports **Event-Driven Branching**. A plugin decides the next path by choosing which event type to emit. 1. **Step 1:** Plugin `classifier` inspects data. 1. **Output:** Plugin emits `type: "image.detected"` or `type: "text.detected"`. 1. **Routing:** You define two pipelines—one `on: image.detected` and one `on: text.detected`. Use this when the plugin is making a domain decision about what happened. Use `if` when the pipeline is making a structural decision about whether a step should run. ______________________________________________________________________ ## 6. Dispatcher Preflight Before spawning a plugin process, the dispatcher runs a **preflight phase** for every job. Preflight separates orchestration decisions from plugin execution, ensuring consistent data-plane semantics regardless of whether a step is user-defined or an internal orchestration primitive such as `core.switch`. ### 6.1 Preflight Steps Preflight executes two operations in order: 1. **Load request context** — Fetches accumulated baggage from the `event_context` table (all upstream metadata for this job's execution tree). 1. **Prepare for execution** — User-defined `uses` steps may apply `with` remaps after the governance payload/context merge. Internal `core.switch` jobs evaluate the compiled condition and emit `ductile.switch.true` or `ductile.switch.false`. ### 6.2 Preflight Outcomes | Outcome | When | Effect | | -------- | ----------------------------------------------------------- | ---------------------------------------------------- | | **run** | Context loaded successfully | Plugin process or internal builtin executes normally | | **skip** | Reserved for explicit orchestration skip paths | Rare for authored `if:` pipelines | | **fail** | Context load, remap, or builtin evaluation returns an error | Job marked `failed`; no downstream routing | ### 6.3 Conditional Branch Routing When a compiled `if:` step is reached, the dispatcher runs the internal `core.switch` job. That job: 1. Evaluates the compiled condition against `payload.*`, `context.*`, and `config.*`. 1. Emits either `ductile.switch.true` or `ductile.switch.false`. 1. Lets the router dispatch either the gated step or the bypass path. Successor routing still happens before the deciding job is marked terminal, preventing synchronous callers from seeing the tree as complete before all children are enqueued. ### 6.4 Preflight Events The dispatcher emits a `job.preflight` event after preflight completes (or fails), with the following payload: ```json { "job_id": "uuid", "plugin": "plugin-name", "command": "command-name", "decision": "run | skip | fail", "reason": "" } ``` The `reason` field is empty for `run` decisions, contains the condition failure reason for `skip`, and contains the error message for `fail`. These events enable async consumers (TUI, event streams, monitoring) to distinguish orchestration decisions from plugin execution outcomes. ______________________________________________________________________ ## 8. Lifecycle Hooks (`on-hook`) Lifecycle hooks allow pipelines to trigger based on **system events** (e.g., job completion) rather than plugin-emitted events. Hook pipelines run as independent root jobs and do not inherit context from the job that triggered them. ### 8.1 DSL Syntax Use the `on-hook:` keyword instead of `on:`. These keywords are mutually exclusive. ```yaml pipelines: - name: notify-on-failure on-hook: job.completed steps: - uses: discord-notify if: path: payload.status op: neq value: succeeded ``` ### 8.2 Supported Signals | Signal | Triggered When | | --------------- | ------------------------------------------------------------------------------------ | | `job.completed` | A root job reaches a terminal state (`succeeded`, `failed`, `timed_out`, or `dead`). | ### 8.3 Opt-in Configuration To prevent accidental infinite loops and reduce noise, plugins must explicitly opt-in to lifecycle hooks in their configuration. ```yaml plugins: my-important-plugin: notify_on_complete: true # Required for on-hook: job.completed to fire ``` ______________________________________________________________________ ## 9. Failure States & Event Payloads When a job fails, times out, or becomes "dead" (exceeds retries), Ductile emits specialized events. These events include enhanced payloads to simplify downstream notifications. ### 9.1 Enhanced Payload Fields In addition to standard fields like `job_id` and `duration_ms`, failure events (`job.failed`, `job.timed_out`, `job.dead`) include: | Field | Description | Example | | --------- | -------------------------------------------------------------- | ----------------------------------------- | | `plugin` | The name of the plugin that failed. | `git-sync` | | `message` | A human-readable summary of the failure. | `Job failed [git-sync]: connection reset` | | `text` | An alias for `message` (convenience for notification plugins). | `Job failed [git-sync]: connection reset` | | `error` | The raw error message (if available). | `connection reset` | ### 9.2 Usage in Pipelines These fields enable simple notification steps without complex `if` logic or payload mapping: ```yaml pipelines: - name: failure-announcer on-hook: job.completed steps: - uses: discord-notify if: path: payload.status op: neq value: succeeded # discord-notify automatically uses payload.message if present ``` ______________________________________________________________________ ## 10. Validation Ductile performs several checks when loading pipelines: - **Cycle Detection:** Refuses to start if a pipeline calls itself (directly or indirectly). - **Shadowing:** Ensures two pipelines don't use the same name. - **Dangling Calls:** Ensures every `call` references a valid pipeline name. - **Condition Validation:** Verifies `if` trees have valid shape, supported operators, allowed roots, and safe depth/count limits. - **Schema Validation:** Verifies the YAML structure against the official [pipelines.json](https://ductile.run/PIPELINES/schemas/pipelines.schema.json). # Ductile — Routing & Orchestration Specification **Version:** 1.0\ **Date:** 2026-02-11\ **Model:** Governance Hybrid (DB-only) > **Note:** the original spec described a Data Plane consisting of core-managed workspace directories. The core no longer provisions per-job workspaces; filesystem state is the plugin's concern. Sections below referring to `workspace_dir` are retained for historical context but no longer describe runtime behaviour. ______________________________________________________________________ ## 1. Overview Ductile uses a **Graph-based Pipeline** model to orchestrate event flow. It separates **Governance** (metadata/context) from **Execution** (plugin-spawned subprocesses). ### 1.1 Core Components - **Control Plane (DB):** A SQLite ledger (`event_context`) that accumulates metadata ("Baggage") across hops. - **Filesystem (Plugin-managed):** Plugins that need a scratch path or persistent cache create and manage it themselves; the core does not provision a per-job directory. - **Orchestrator (DSL):** A YAML-based Pipeline DSL that supports nesting, branching, and single-root triggers. ______________________________________________________________________ ## 2. Pipeline DSL Pipelines are defined in YAML files referenced via `include:` in `config.yaml` (files or directories). ### 2.1 Syntax ```yaml pipelines: - name: wisdom-chain on: discord.video_link_received # The "Single Root" trigger steps: - id: downloader uses: yt-dlp-plugin - id: processing call: standard-audio-wisdom # Nested Pipeline call - id: delivery split: # Branching logic - uses: discord-notifier - steps: # Sequential branch - uses: s3-archiver - uses: db-indexer ``` ### 2.2 Functional Blocks - **uses:** Execute a specific plugin command. - **call:** Execute another named pipeline (reusable middleware). - **split:** Branch execution into multiple parallel paths. - **on:** The event that triggers the root of the pipeline. - **on-hook:** The lifecycle signal that triggers the root of the pipeline (e.g., `job.completed`). Mutually exclusive with `on`. ______________________________________________________________________ ## 2.3 Lifecycle Hooks Lifecycle hooks allow for out-of-band orchestration triggered by the **Dispatcher** rather than a plugin event. 1. **Opt-in:** A plugin must have `notify_on_complete: true` in its operator configuration. 1. **Signal:** When the job reaches a terminal state, the Dispatcher resolves any pipelines matching the signal (e.g., `job.completed`). 1. **Isolation:** Hook pipelines run as fresh root jobs with no context inheritance from the triggering job. ______________________________________________________________________ ## 3. The Control Plane (Baggage & Ledger) Every job in a pipeline is associated with an `event_context`. ### 3.1 `event_context` Schema ```sql CREATE TABLE event_context ( id TEXT PRIMARY KEY, -- UUID parent_id TEXT, -- FK for lineage pipeline_name TEXT, step_id TEXT, accumulated_json JSON NOT NULL, -- The "Baggage" created_at TEXT NOT NULL ); ``` ### 3.2 Explicit Context Accumulation Baggage is explicit: plugins emit event payloads; pipeline authors decide which values become durable. When Step A transitions to Step B: 1. Core reads `accumulated_json` from Step A's context. 1. If Step B declares `baggage`, Core evaluates those claims against the immediate event `payload.*` and inherited `context.*`. 1. Core deep-accretes the claimed values into a new `event_context` row for Step B. 1. Existing durable paths are immutable. A step may add a new path or repeat the same value, but may not rewrite an inherited path. Example: ```yaml steps: - id: fetch uses: web_fetch baggage: web.url: payload.url - id: summarize uses: summarizer baggage: web.content: payload.content web.status_code: payload.status_code ``` Bulk import is allowed only under an explicit namespace: ```yaml baggage: from: payload.metadata namespace: whisper ``` This imports `payload.metadata` as `context.whisper.*`. Omitting `namespace` is rejected until plugin manifest default namespaces exist. If a step declares no `baggage`, Core creates no new durable context for that hop beyond inherited baggage and control-plane fields. Immediate event payload still flows to downstream steps, but it is not promoted into `event_context` implicitly. ______________________________________________________________________ ## 4. Filesystem (Plugin-managed) The core does not provision per-job workspace directories. The previous "Data Plane" section described a hard-linked, janitor-pruned `/ws/` tree; that machinery has been removed. Plugins that need filesystem state are responsible for it: - **Ephemeral scratch:** `mktemp -d` (or language equivalent), cleaned up on exit. - **Persistent cache:** `~/.cache/ductile-/` or a path declared in plugin config and validated at startup. - **Step-to-step file passing:** the producing plugin writes to a path it chooses; the path is propagated as baggage via the pipeline's `with:` remap so the consuming plugin can read it. See `docs/PLUGIN_DEVELOPMENT.md` §9 for details. ______________________________________________________________________ ## 5. The Plugin Protocol (v2) Plugins receive the following via `stdin`: ```json { "protocol": 2, "job_id": "uuid-456", "context": { "origin_plugin": "discord", "channel_id": "123", "permission_tier": "WRITE" }, "event": { "type": "video_downloaded", "payload": { "filename": "lecture.mp4", "size_bytes": 10485760 } } } ``` ### 5.1 Plugin Responsibilities - **Metadata:** Read durable facts and routing info from `context`. - **Artifacts:** Read/write files at plugin-managed paths (see §4). - **Communication:** Emit event payloads for downstream steps. Payload is per-hop; values become durable only when a pipeline author claims them with `baggage`. ______________________________________________________________________ ## 6. Failure & Recovery ### 6.1 State Persistence Because the `event_context` is in SQLite, a crash is non-destructive for the control plane. * The **LLM Operator** can inspect the `event_context` to see exactly where a pipeline stalled. * The Core can "Replay" a step by creating a new job using the existing `event_context_id`. Plugin-managed filesystem state is the plugin's concern to recover. ### 6.2 Cycle Detection The Core maintains a `hop_count` in the `event_context`. If a pipeline exceeds 20 hops (or calls itself recursively too deep), the Core kills the chain to prevent infinite loops. ______________________________________________________________________ ## 7. CLI & Operations All orchestration-related CLI commands MUST support the following flags to ensure safety and observability: - **-v, --verbose:** Expose internal DAG resolution, baggage merging logic, and path calculations. - **--dry-run:** Preview the next steps of a pipeline without enqueuing jobs. ### 7.1 LLM Operator Affordances (RFC-004) The Routing system exposes specific "Admin Utilities" for the LLM: * `job inspect `: Returns the full Graph of what happened. * `pipeline visualize `: Returns a Mermaid.js diagram of the DSL. * `pipeline dry-run `: Executes the plugin in a sandbox; any filesystem isolation is the plugin's responsibility. ## 8. Branching & Decisions Ductile supports two models for decision making: **Step-Gating (DSL)** and **Multi-Event Branching (Plugin)**. ### 8.1 Step-Gating via `if` Pipelines can use the `if` keyword on any step to decide whether it should run based on the current payload, accumulated context, or plugin configuration. ```yaml - id: notifier uses: discord-notifier if: path: payload.status op: eq value: error ``` Authored `if:` conditions compile into an internal `core.switch` hop. That hop emits `ductile.switch.true` or `ductile.switch.false`, so the gated step only runs on the true branch while the false branch bypasses directly to the downstream route. ### 8.2 Multi-Event Branching For complex domain-level decisions, plugins are responsible for emitting specific **Event Types** to signal different outcomes. **Example Pipeline:** ```yaml - id: validator uses: schema-checker # The router matches the emitted event type to the next pipeline or step. ``` This pattern keeps the DSL declarative while offloading complex logic to the plugins. # Scheduler Detailed reference for Ductile's scheduler behavior and schedule configuration. ## Overview The scheduler runs a single heartbeat loop. On each tick it evaluates all enabled plugin schedules and enqueues due jobs. Each schedule entry is tracked independently in the `schedule_entries` table so catch-up and next-run behavior are stable across restarts. Jobs are always enqueued as the schedule's `command` (default: `poll`) and include the schedule's `payload`. ## Schedule Entry Fields ```yaml plugins: example: schedules: - id: hourly command: poll every: 1h jitter: 30s catch_up: run_once if_running: skip only_between: "08:00-18:00" timezone: "Australia/Sydney" not_on: [saturday, sunday] payload: source: scheduler ``` ### Common Fields - `id`: Unique schedule ID within the plugin (default: `default`). - `command`: Command to run (default: `poll`). - `payload`: JSON object merged into the command payload. ### Schedule Types Exactly one of the following should be set: - `every`: Interval schedule (supports `5m`, `15m`, `30m`, `hourly`, `2h`, `daily`, `weekly`, `monthly`). - `cron`: Standard 5-field cron (`min hour dom month dow`). - `at`: One-shot RFC3339 timestamp (UTC or offset). - `after`: One-shot delay from service start (duration). ## Time Constraints These constraints are applied before enqueueing a due job. - `jitter`: Random offset applied to interval schedules per run. - `only_between`: Time window in local schedule time (e.g. `"08:00-22:00"`). - Supports overnight windows such as `"22:00-06:00"`. - `timezone`: IANA timezone used for cron and time window evaluation. - `not_on`: Weekdays to skip (string names like `saturday` or integers `0-6`, `7` for Sunday). ## Catch-up Policy On startup, the scheduler can run missed ticks based on `catch_up`: - `skip` (default): Ignore missed intervals. - `run_once`: Enqueue a single catch-up job if any ticks were missed. - `run_all`: Enqueue one job per missed interval (bounded to 100 runs). Catch-up applies only to `every` schedules. Catch-up jobs use a `catchup`-scoped dedupe key to avoid duplication. ## Overlap Policy `if_running` controls what happens when a prior job is still in-flight: - `skip` (default): Do not enqueue a new job. - `queue`: Enqueue regardless of in-flight jobs. - `cancel`: Cancel outstanding jobs for the same plugin/command, then enqueue. ## Poll Guard A global per-plugin guard prevents multiple concurrent scheduled polls: ```yaml plugins: example: max_outstanding_polls: 1 ``` If a matching `queued` or `running` job exists, the scheduler skips enqueueing. ## Circuit Breaker Scheduler-originated polls respect the circuit breaker: - Opens after `threshold` consecutive failures. - Remains open for `reset_after`. - Half-open probe allows one poll; success closes the circuit. - Current state is stored in `circuit_breakers`; append-only history is stored in `circuit_breaker_transitions`. - Operators can inspect history with `ductile system breaker [--json]`. ```yaml plugins: example: circuit_breaker: threshold: 3 reset_after: 30m ``` ## State Tracking Schedule state is stored in `schedule_entries`: - `last_fired_at`: Last time the scheduler attempted to enqueue. - `last_success_at` / `last_success_job_id`: Latest successful run. - `next_run_at`: Next due timestamp. - `status`: `active`, `paused_invalid`, `paused_manual`, `exhausted`. One-shot schedules (`at`, `after`) transition to `exhausted` after firing. ## Examples ### Cron with timezone ```yaml plugins: reports: schedules: - id: weekdays-9am cron: "0 9 * * 1-5" timezone: "Australia/Sydney" ``` ### One-shot at ```yaml plugins: reminder: schedules: - id: send-once at: "2026-03-15T14:00:00Z" ``` ### Only between + not_on ```yaml plugins: poller: schedules: - every: 5m only_between: "08:00-18:00" not_on: [saturday, sunday] ``` # Plugin Facts This document is the canonical reference for durable plugin memory in Ductile. **The model:** durable plugin truth is the append-only `plugin_facts` stream. `plugin_state` is a compatibility/cache view of the latest fact, kept current automatically by core so that legacy readers see the same shape they always have. New plugins declare `fact_outputs` in their manifest and let the view come for free. ## 1. What A Plugin Does A plugin that needs to remember anything across invocations follows this pattern: 1. The plugin emits a successful, stable snapshot in `state_updates`. 1. The plugin manifest declares that snapshot as a fact output. 1. Core records the snapshot as an append-only row in `plugin_facts`, with a Ductile-owned monotonic `seq` and the declared `fact_type`. 1. Core rebuilds the compatibility `plugin_state` row from the newest fact according to the declared `compatibility_view` (currently `mirror_object`). This means: - `plugin_facts` is the durable record. - `plugin_state` is the compatibility/cache view. ```text Previously, plugins wrote durable truth directly into `plugin_state` via shallow merge of `state_updates`. That model is legacy; new plugins should always declare `fact_outputs`. Plugins still on direct write-through are running in a compatibility window. ``` ## 2. Migrated Plugins In-tree (codex repo): - `file_watch` `poll` → `file_watch.snapshot` - `folder_watch` `poll` → `folder_watch.snapshot` - `py-greet` `poll` → `py-greet.snapshot` - `ts-bun-greet` `poll` → `ts-bun-greet.snapshot` - `stress` `state` → `stress.state_snapshot` External plugins: - `gmail_poller` `poll` → `gmail_poller.snapshot` - `youtube_playlist` `poll` → `youtube_playlist.snapshot` - `jina-reader` `poll` → `jina-reader.snapshot` - `birdnet_firstday` `poll` → `birdnet_firstday.snapshot` - `sqlite_change` `poll` → `sqlite_change.snapshot` - `withings` `poll` and `token_refresh` → `withings.snapshot` `health` commands are intentionally **not** part of the durable fact flow. Health is diagnostic and should not mutate durable state. ## 3. Compliance Rules If you want a plugin to be compatible with this pattern, the plugin and core need a clear, defensible contract. ### Plugin-side rules - Emit facts only from commands that produce meaningful durable truth. - Prefer successful `poll` or equivalent snapshot-producing commands. - Do not use `health` or `init` as durable state — they should emit no `state_updates`. - Keep the emitted snapshot shape stable and explicit. - Return a full snapshot, not a partial patch. The compatibility view is rebuilt wholesale from the latest fact, so partial patches lose information. - Keep the snapshot JSON object-shaped and deterministic enough for operators to inspect; avoid non-deterministic ordering inside lists or maps. ### Core-side rules - Declare an explicit fact type in `manifest.yaml`. - Record each fact append-only in `plugin_facts`. - Declare how compatibility `plugin_state` is derived from that fact. - Add an operator-visible read path. - Add tests that prove both fact persistence and derived compatibility state. The smallest useful manifest shape is: ```yaml fact_outputs: - when: command: poll from: state_updates fact_type: file_watch.snapshot compatibility_view: mirror_object ``` ## 4. Recommended Fact Shape Use a fact when the plugin can answer: > "What is the current durable observed state of this plugin right now?" Good candidates: - watcher snapshots - cursors/checkpoints - discovered remote resource inventories - reducer-friendly state snapshots Poor candidates: - transient health checks - ephemeral timing/latency noise - values that are meaningful only to a single in-flight job For a first migration, prefer a **full snapshot** over incremental diffs. ## 5. Snapshot Examples ### `file_watch` `file_watch poll` returns a snapshot shaped like: ```json { "watches": { "single-file": { "exists": true, "fingerprint": "abc123", "size": 42, "mtime_ns": 1713740000000000000, "path": "/tmp/file.txt", "strategy": "sha256", "updated_at": "2026-04-22T01:02:03Z" } }, "last_poll_at": "2026-04-22T01:02:03Z" } ``` Core then: - stores that JSON in `plugin_facts.fact_json` - assigns a Ductile-owned `plugin_facts.seq` for new facts - tags it `file_watch.snapshot` - updates `plugin_state` for `file_watch` to the same snapshot shape This keeps legacy state readers working while giving operators an append-only history. ### `folder_watch` `folder_watch poll` returns the same top-level compatibility shape: ```json { "watches": { "docs": { "root": "/srv/content", "files": { "summary.md": "abc123" }, "snapshot_hash": "def456", "file_count": 1, "updated_at": "2026-04-22T01:02:03Z" } }, "last_poll_at": "2026-04-22T01:02:03Z" } ``` ### `py-greet` and `ts-bun-greet` The example greeting plugins emit a tiny full snapshot: ```json { "last_run": "2026-04-22T01:02:03Z", "last_greeting": "Hello, Ductile!" } ``` ### `stress` The `stress state` command emits the full compatibility snapshot for its only durable datum: ```json { "count": 42 } ``` ## 6. Migration Checklist For Another Plugin When migrating another plugin to `plugin_facts`, do all of the following: 1. Choose one command that produces durable truth. 1. Define one explicit fact type. 1. Make the plugin emit a stable object snapshot. 1. Ensure the snapshot is a full compatibility-state view, not just a partial patch. 1. Add `fact_outputs` to the plugin manifest. 1. Declare the compatibility view policy for `plugin_state`. 1. Add operator inspection support. 1. Add unit tests for persistence and derived state. 1. Add a Docker or similarly realistic fixture when runtime behavior matters. 1. Document the fact type, snapshot shape, and non-goals. ## 7. Questions To Resolve Before Adding A New Plugin Or Migrating One Before declaring `fact_outputs` for a plugin, answer: - What exact command owns durable truth? - Is the emitted JSON a full snapshot or only a delta? It must be a full snapshot — partial patches break the compatibility view. - Should the compatibility view mirror the newest fact exactly (`compatibility_view: mirror_object`), or does the plugin need a different reduction policy? Today only `mirror_object` is supported; a reducer-based policy would be a future extension. - What data should remain diagnostic only and stay out of durable storage? - How will an operator inspect recent facts? - What realistic test proves the fact path end to end? If those answers are vague, the plugin should remain on direct write-through (action-bookkeeping non-candidates) rather than declaring a half-thought fact contract. ## 8. Deployment Note For existing databases, apply required schema migrations before a normal deploy, then restart or deploy the updated binary. For non-empty existing databases, startup should validate and fail if required schema is missing. It should not silently add `plugin_facts`, `seq`, or related indexes during normal open. Startup errors should name the migration script needed for the current database shape. Existing rows without `seq` keep `seq` as `NULL`. Ductile does not backfill guessed order for legacy facts; new rows use `seq` for ordering, and legacy rows fall back to their previous timestamp order. # Building # Ductile Cookbook: Integration Patterns Practical recipes for wiring Ductile **Connectors** and **Orchestrations** to solve real-world problems. ______________________________________________________________________ ## Pattern: Automated Astro Staging Rebuild (Watch -> Trigger) **Use case:** Rebuild your Astro staging site automatically whenever new AI-generated summaries are added to a specific folder. ### 1) Configure the `folder_watch` Connector Set up a **Proactive Operation** (`poll`) to scan your content directory. ```yaml # ~/.config/ductile/plugins.yaml plugins: folder_watch: enabled: true schedules: - id: default every: 1m config: watches: - id: astro_summaries root: ${HOME}/site/src/content/summaries event_type: astro.summaries.changed recursive: true include_globs: ["**/*.md"] emit_mode: aggregate ``` ### 2) Define the Rebuild Orchestration (Pipeline) Create a **Pipeline** to respond to the `astro.summaries.changed` event. ```yaml # ~/.config/ductile/pipelines.yaml pipelines: - name: astro-rebuild-on-change on: astro.summaries.changed steps: - id: rebuild_staging uses: astro_rebuild_staging # A sys_exec connector clone ``` ### 3) Configure the Rebuild Connector ```yaml # ~/.config/ductile/plugins.yaml plugins: astro_rebuild_staging: enabled: true timeout: 15m config: command: "docker compose -f ${HOME}/admin/docker-compose.yml up -d --build" working_dir: "${HOME}/admin" ``` ______________________________________________________________________ ## Pattern: YouTube Playlist-to-Summary Pipeline **Use case:** Automatically fetch, transcribe, and AI-summarise new videos from a YouTube playlist, then write the result to disk. This is a multi-hop pipeline where each step passes its output (`result`) as the next step's `content` via baggage. ### 1) Configure the Playlist Watcher (Proactive) ```yaml # ~/.config/ductile/plugins.yaml plugins: youtube_playlist: enabled: true schedules: - every: 30m jitter: 2m timeout: 60s max_attempts: 2 config: playlist_url: "https://www.youtube.com/playlist?list=PL5Rty1LvKaJ5GI4nqEzvEPTdgobgODlkk" output_dir: "/home/matt/tmp/ductile-output" filename_template: "{video_id}.md" max_entries: 50 max_emit: 1 # Only process one new video per run emit_existing_on_first_run: false transcript_language: en ``` ### 2) Define the Processing Pipeline ```yaml # ~/.config/ductile/pipelines.yaml pipelines: - name: playlist-wisdom on: youtube.playlist_item steps: - id: transcript uses: youtube_transcript # fetches transcript, result = raw text - id: summarize uses: fabric # summarises transcript, result = markdown summary - id: write uses: file_handler # writes summary to disk ``` ### 3) Configure Supporting Plugins ```yaml # ~/.config/ductile/plugins.yaml plugins: youtube_transcript: enabled: true timeout: 60s max_attempts: 2 config: {} fabric: enabled: true timeout: 120s max_attempts: 2 config: FABRIC_DEFAULT_PATTERN: "summarize" file_handler: enabled: true timeout: 30s max_attempts: 1 config: allowed_write_paths: "/home/matt/tmp/ductile-output" default_output_dir: "/home/matt/tmp/ductile-output" ``` ### How it works 1. `youtube_playlist` polls the playlist every 30 min (with up to 2 min jitter). 1. For each new video, it emits a `youtube.playlist_item` event with `video_id` in the payload. 1. `youtube_transcript` fetches the transcript; its output (`result`) flows into the next step. 1. `fabric` summarises the transcript using the `summarize` pattern; its `result` becomes the markdown summary. 1. `file_handler` writes the summary to `{video_id}.md` in the output directory. ### Notes - Set `max_emit: 1` to process one new video per poll cycle — avoids burst load on initial run. - `emit_existing_on_first_run: false` means already-seen videos are skipped on restart. - `youtube_playlist` uses `yt-dlp --flat-playlist` internally; ensure yt-dlp is installed and on PATH. - For systemd services, add `~/.local/bin` to the service `Environment="PATH=..."` line. ______________________________________________________________________ ## Pattern: Discord Notifications via Incoming Webhook **Use case:** Send messages to a Discord channel from any pipeline step or scheduled trigger. The `discord_notify` plugin wraps Discord's incoming webhook API. It exposes two schedulable-friendly commands: - `handle` — called from pipeline steps (event-driven) - `poll` — identical behaviour, but allowed in `schedules:` blocks (ductile forbids `handle` in schedules) ### 1) Configure the Plugin ```yaml # ~/.config/ductile/plugins.yaml plugins: discord_notify: enabled: true timeout: 15s max_attempts: 2 config: webhook_url: "${DISCORD_WEBHOOK_URL}" # or hard-code in tokens.yaml default_username: "Ductile" ``` ### 2) Use in a Pipeline Step The plugin reads `message`, `content`, `result`, or `title` from the payload (in that order). ```yaml pipelines: - name: notify-on-build on: build.complete steps: - id: notify uses: discord_notify # payload.result from the previous step becomes the Discord message ``` Or pass a static message: ```yaml pipelines: - name: notify-on-error on: job.failed steps: - id: alert uses: discord_notify payload: message: "A job failed — check the dashboard." ``` ### 3) Use as a Scheduled Heartbeat `poll` is the schedulable alias for `handle`. Use it with `schedules:` blocks to send timed notifications. ```yaml plugins: discord_notify: schedules: # Daily 09:00 status ping - id: morning-ping cron: "0 9 * * *" command: poll payload: message: "Good morning — Ductile is running." not_on: [saturday, sunday] # Startup one-shot - id: boot-notify after: 30s command: poll payload: message: "Ductile started." ``` ### Notes - Discord hard-limits messages to 2000 characters; the plugin truncates automatically. - 4xx errors (bad webhook, forbidden) are not retried. 5xx and network errors retry per `max_attempts`. - `poll` and `handle` are identical in behaviour; the distinction is purely for ductile's scheduler validation. - Store the webhook URL in `tokens.yaml` and reference it via `${VAR}` interpolation to keep it out of operational config files. ______________________________________________________________________ ## Pattern: End-to-End: Playlist → Summary → Discord Notification **Use case:** Combine the two patterns above — automatically process new playlist videos and notify a Discord channel when a summary is written. ```yaml # ~/.config/ductile/pipelines.yaml pipelines: - name: playlist-wisdom on: youtube.playlist_item steps: - id: transcript uses: youtube_transcript - id: summarize uses: fabric - id: write uses: file_handler - id: notify uses: discord_notify # After file_handler, result contains the output path or confirmation. # Or set a static title: payload: title: "New summary ready" # message will fall through to context.result from the write step ``` This gives you an automated, end-to-end pipeline with Discord confirmation for every new video processed. ______________________________________________________________________ ## Pattern: Route YouTube vs Web URLs **Use case:** Use the `if` classifier to emit different event types based on URL content. ### 1) Configure the classifier instance ```yaml # ~/.config/ductile/plugins.yaml plugins: check_youtube: enabled: true timeout: 30s max_attempts: 1 config: field: text checks: - contains: "youtu.be" emit: youtube.url.detected - contains: "youtube.com" emit: youtube.url.detected - startswith: "http" emit: web.url.detected - default: text.received ``` ### 2) Use it in a pipeline ```yaml # ~/.config/ductile/pipelines.yaml pipelines: - name: ai-dispatch on: discord.ai.command steps: - id: classify uses: check_youtube - name: youtube-wisdom on: youtube.url.detected steps: - uses: youtube_transcript - uses: fabric - uses: file_handler - name: web-summarize on: web.url.detected steps: - uses: fabric ``` ### Notes - `default` is a final fallback; omit it if you want no-match to error. - The plugin passes the payload through unchanged. ______________________________________________________________________ ## Adding your own patterns Each recipe follows the same structure: configure a plugin, define a pipeline, wire the events. If you have a working integration worth sharing, add it here. # Plugin Development Guide Ductile is built on a **spawn-per-command** model. A plugin is any executable that reads one JSON request from `stdin`, writes one JSON response to `stdout`, and exits. There is no daemon, no shared memory, no in-process state. **Durable plugin memory is the append-only `plugin_facts` stream.** A plugin that needs to remember anything across invocations declares a `fact_outputs` rule in its manifest, returns a stable snapshot from its durable command, and lets core record that snapshot append-only and rebuild the compatibility view automatically. This guide treats the manifest as the contract that drives plugin quality — every directive is explained below and exists to push you toward the correct shape. If you find yourself wanting to do something the manifest doesn't sanction, that is usually a signal to step back rather than add a workaround. See [Plugin Facts](https://ductile.run/PLUGIN_FACTS/index.md) for the canonical reference and worked examples of the durability contract. ______________________________________________________________________ ## 1. The Lifecycle When a job is triggered (via scheduler, API, or webhook): 1. Ductile forks the plugin entrypoint as a fresh process. 1. The core writes a **request envelope** (JSON) to the plugin's `stdin`. 1. The plugin processes the command and writes a **response envelope** (JSON) to `stdout`, then exits. 1. Ductile captures `stderr` for logging and kills the process if it exceeds the timeout. Because every invocation is a fresh process, the plugin has no in-memory state across calls. Anything the plugin needs to remember must come back through the request envelope's `state` field on the next invocation. ______________________________________________________________________ ## 2. The plugin protocol ### 2.1 Request Envelope (Core → Plugin) ```json { "protocol": 2, "job_id": "uuid", "command": "poll | handle | health | init", "config": {}, "state": {}, "context": {}, "event": {}, "secrets": {}, "deadline_at": "ISO8601" } ``` | Field | What it is | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `protocol` | The wire-protocol version. Plugins declare which version they expect in `manifest.protocol`; mismatch refuses load. | | `job_id` | Ductile-assigned unique id for this invocation. Useful in logs and downstream events. | | `command` | The command the plugin is being asked to run. Always one of `poll`, `handle`, `health`, `init` (plus any plugin-declared command name). | | `config` | The static plugin config from the operator's YAML, with `${ENV}` interpolated. Read-only. | | `state` | The plugin's current compatibility-view row — i.e. the latest fact's snapshot for plugins that declare `fact_outputs`, or the direct-write `plugin_state` row for plugins that have not yet migrated. Treat it as *"what I knew last time."* | | `context` | Shared baggage carried across the pipeline chain. Operator-declared, immutable in the receiving plugin. | | `event` | Present only for `handle`. The triggering event envelope from upstream. | | `secrets` | A name→value map of the secrets the core composed for your plugin (present only when you are an attested principal with active grants). Read-only — distinct from `config`. | | `deadline_at` | Informational ISO8601 timestamp. Plugins may abandon long work early; core enforces the real deadline externally. | **Secrets reach your plugin here — over stdin, nowhere else.** They are *not* in your environment (the core strips it) and never on argv. Three rules: (1) you **consume** secrets (`secrets["GITHUB_TOKEN"]`); you never register, roll, or revoke them — that is the operator's `ductile vault` surface. (2) Your plugin must be attested with `ductile plugin lock ` before it receives *any* secret; editing the manifest or entrypoint breaks the fingerprint and requires re-`plugin lock`, or delivery fails closed at spawn. (3) Do **not** echo a secret to stdout/stderr — the core stores your output verbatim and will not scrub it. Full model: `docs/SECRETS.md`. ### 2.2 Response Envelope (Plugin → Core) ```json { "status": "ok | error", "result": "short human-readable summary", "error": "human-readable message (when status=error)", "retry": true, "events": [], "state_updates": {}, "logs": [] } ``` | Field | What it is | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `status` | `ok` for success, `error` for any failure. | | `result` | **Required when `status=ok`.** Short human-readable summary of what happened. Surfaces in `ductile job inspect`, the watch UI, and as the result for synchronous pipelines. | | `error` | **Required when `status=error`.** Human-readable diagnostic. | | `retry` | Response-envelope compatibility signal. Defaults `true` if omitted. Set `false` only when retrying the same request cannot succeed (configuration error, permanent input invalid). Core owns the final retry decision; this is a *fact about the failure*, not a policy instruction. | | `events` | Array of `{type, payload, dedupe_key?}` envelopes that drive downstream pipeline routing. | | `state_updates` | The plugin's emitted **snapshot** for this invocation. When the manifest declares a matching `fact_outputs` rule, core records this snapshot append-only as a `plugin_facts` row and rebuilds the compatibility view (`plugin_state`) from it. See §3.4. | | `logs` | Array of `{level, message}`. Stored with the job record. | ### 2.3 What `state_updates` Is, And What It Is Not `state_updates` is the snapshot of the plugin's **observed durable state at the end of this invocation**. It is not a partial patch and it is not a running diary of actions taken. A correct snapshot: - Is a full object representing the plugin's durable observed state. - Contains the same keys every invocation that command runs (presence-stable). - Is deterministic: the same observed inputs produce the same bytes out. - Has a clear cache-view story: a downstream reader of the latest snapshot understands what the plugin knows. An incorrect snapshot (anti-patterns — see §6): - A counter that increments each invocation (`executions_count`). - A timestamp that updates whether or not anything was observed (`last_run`). - A diff or patch (`{"new_id": "abc"}`). - An ordered set built from `set()` union (non-deterministic order). If a plugin emits action bookkeeping rather than observed state, it should emit no `state_updates` at all. Action bookkeeping belongs in `job_log`, which is captured automatically. ### 2.4 Framing And Errors - One JSON request on stdin → one JSON response on stdout. Not JSON Lines, not length-prefixed. - Exit code `78` (EX_CONFIG) marks a permanent configuration failure and is treated as non-retryable regardless of the `retry` field. - If the request `protocol` field doesn't match what the plugin expects, the plugin should exit `78` with a clear error on stderr. ______________________________________________________________________ ## 3. The Manifest (`manifest.yaml`) The manifest is the single source of truth for what the plugin is, what it does, and how its memory works. Treat reading this section top-to-bottom as a quality checklist for any new plugin. ### 3.1 Top-Level Fields ```yaml manifest_spec: ductile.plugin # required manifest_version: 1 # required name: my_plugin # required version: 0.1.0 # required protocol: 2 # required entrypoint: run.py # required description: "What this plugin does, in one sentence." # optional but recommended concurrency_safe: true # optional; default true commands: [...] # required, at least one fact_outputs: [...] # required for any plugin with durable memory config_keys: # optional; declares config contract required: [...] optional: [...] ``` #### `manifest_spec` (required) Must be the literal string `ductile.plugin`. Identifies this YAML as a ductile plugin manifest. Future manifest families (e.g. an event spec) would use a different identifier. #### `manifest_version` (required) Must be the integer `1`. Ductile uses this to evolve manifest semantics accretively without breaking existing plugins. #### `name` (required) The plugin's identity. Must be unique across all plugin roots — first plugin discovered with a given name wins; later duplicates are ignored. Use underscores or hyphens, no spaces. Pipelines, schedules, and routes refer to the plugin by this name. #### `version` (required) The plugin's release identity over time. Free-form string; prefer semver-compatible `MAJOR.MINOR.PATCH`. Bump when behaviour changes so operators can correlate facts and job logs to plugin version. #### `protocol` (required) Must be `2`. Declares the wire protocol version this plugin understands. Mismatch refuses load; do not lie about protocol support. #### `entrypoint` (required) Path to the executable, relative to the plugin directory. Must be marked executable (`chmod +x`). The shebang line picks the interpreter. No `..` allowed (path traversal prevention). Examples: `run.py`, `run.sh`, `./bin/dispatcher`. #### `description` (optional, recommended) Short human-readable summary of what the plugin does. Surfaces in operator inspection and LLM-driven tools. Treat it as the answer to *"what does this plugin do?"* in one sentence. #### `concurrency_safe` (optional, default `true`) Concurrency hint. `false` tells the runtime that the plugin is **not** safe to run two of in parallel — typically because it owns a single-writer durable resource (a SQLite DB it writes to, an OAuth token table) and parallel execution would race the writer. When `false`, runtime defaults to serial execution unless the operator explicitly overrides with `plugins..parallelism > 1`. If you have any doubt, set `false`. Concurrency-safe is a property the plugin author asserts and the runtime trusts. #### `commands` (required) Array of command declarations. Every command the plugin can be invoked with must be listed, with at least `name` and `type`. See §3.2. #### `fact_outputs` (recommended for any plugin with durable memory) Declares which commands emit durable facts and how the compatibility view is rebuilt. **If your plugin needs to remember anything across invocations, declare this.** See §3.4. #### `config_keys` (optional) Declares the static config contract: ```yaml config_keys: required: [client_id, client_secret, db_path] optional: [request_timeout, lookback_days] ``` `required` keys missing at load time refuse the plugin to load. `optional` keys are documented for operators but not enforced. Keep this list honest — it is the contract the operator's YAML satisfies. ### 3.2 The `commands` Array Each command is a pure function on `(config, state, context, event) → response`. The manifest declares the command's identity, its side-effect class, its input/output shape, and its retry properties. ```yaml commands: - name: poll type: read description: "Fetch latest detections; emit one event per first-of-day species." idempotent: true retry_safe: true input_schema: {} output_schema: status: string events: array state_updates: object values: consume: [] emit: - event: birdnet.firstday_species values: - payload.scientific_name - payload.common_name - payload.first_id - payload.detected_at ``` #### `name` (required) The command's identity inside this plugin. Standard names that the runtime recognises: `poll`, `handle`, `health`, `init`. Plugins may declare additional names (e.g. `token_refresh` in `withings`); those are invocable via API and schedules but do not get the standard-command convenience routing. | Standard name | Purpose | Typical `type` | | ------------- | ---------------------------------------------------------------------------------------------------- | ----------------- | | `poll` | Scheduled durable observation. Emits events on observed change; emits a snapshot in `state_updates`. | `read` | | `handle` | Event-driven response. Receives an upstream event, processes it, optionally emits downstream events. | `write` (usually) | | `health` | Diagnostic check. **Emits no `state_updates`.** | `read` | | `init` | Capability discovery / affordance bundle for LLM tools. **Emits no `state_updates`.** | `read` | #### `type` (required) `read` or `write`. This is about **external observable side effects**, not about whether the command emits durable facts: - `type: read` — no external POST/PUT/DELETE. Idempotent under retry. Examples: `poll`, `fetch`, `get`, `list`, `health`. A `read` command can still emit `state_updates` (the durable observation snapshot) and can still write to a local SQLite DB the plugin owns; the constraint is on external state. - `type: write` — modifies external state via the network. Examples: `sync`, `send`, `notify`, `oauth_callback`, `delete`. Default if `type` is omitted (paranoid default). `type` determines the token scope required to invoke the command (`plugin:ro` vs `plugin:rw`). #### `description` (optional) Short human-readable summary of what this command does. Critical for the TUI, the watch UI, and LLM operators reading capability discovery. #### `idempotent` (optional, boolean) Hint that calling this command N times produces the same observable result as calling it once, given identical inputs. Used by the runtime to make safer retry decisions. Be honest: a `sync` that posts measurements to a remote API is not idempotent unless the remote API deduplicates. #### `retry_safe` (optional, boolean) Hint that this command is safe to retry on transient failure. Stronger than `idempotent` in practice because it accounts for partial-side-effect risk during retry. Default to `false` if you are unsure. #### `input_schema` / `output_schema` (optional, legacy) Either a full JSON Schema object or a compact `field: type` map. Documents the request payload and response shape for API consumers and operators. The compact form expands automatically: ```yaml input_schema: url: string depth: integer ``` These remain useful as a typed surface but are not the durability contract — that is `values` plus `fact_outputs`. #### `values` (optional but recommended) Names-only payload contract — the *Hickey-faithful* successor to typed schemas for pipeline authoring. `values.consume` declares which payload names this command reads from the request. `values.emit` declares, per emitted event type, which payload names the event carries. ```yaml values: consume: - payload.url - payload.depth emit: - event: jina_reader.scraped values: - payload.url - payload.text - payload.content_hash ``` Rules: - Entries are payload **names**, not types. Format: `payload.` or `payload..` for nested keys; `payload.*` matches all. - Pipeline authors use `with:` to remap durable context into the request payload a downstream plugin expects, and `baggage:` to claim which event payload names become durable context. The plugin's `values` declaration is a sanity-aid for that authoring. - `values` does not decide durability. Durability is decided by pipeline `baggage:` (for event payloads becoming context) and by `fact_outputs` (for `state_updates` becoming `plugin_facts`). ### 3.3 `fact_outputs` — The Durability Declaration This is the directive that decides whether your plugin participates in the append-only fact model. ```yaml fact_outputs: - when: command: poll from: state_updates fact_type: my_plugin.snapshot compatibility_view: mirror_object ``` A `fact_outputs` rule says: *"when command `poll` succeeds, take its emitted `state_updates`, record it append-only as a `my_plugin.snapshot` fact, and rebuild the `plugin_state` row by mirroring the snapshot."* #### `when.command` (required) The command name whose successful response produces this fact. One rule per command-that-emits-durable-state. A plugin may declare multiple rules (e.g. `withings` declares one for `poll` and one for `token_refresh`, because both observe durable state). #### `from` (required) Currently must be the literal string `state_updates`. The fact is sourced from the plugin's emitted snapshot. Future protocol versions may add other sources (e.g. a structured `facts` field); they will be accretive additions, not breaking changes. #### `fact_type` (required) The fact's identity. Convention: `.`, where the noun describes the kind of observation. Most migrated plugins use `.snapshot`. Use a different noun only if the plugin emits distinct kinds of observation that downstream readers should differentiate. #### `compatibility_view` (optional, default `mirror_object`) How `plugin_state` is rebuilt from the latest fact. Currently the only supported value is `mirror_object`: replace `plugin_state.state` wholesale with the latest fact's `fact_json`. This is exactly what legacy readers expect, so the migration is transparent. Future view policies (e.g. a reducer that folds multiple facts) would be added as new enum values; today, `mirror_object` is the right answer. ### 3.4 The Plugin Memory Model In One Diagram ```text plugin emits state_updates snapshot │ ▼ ┌───────────────────────────────────────────────┐ │ core (manifest fact_outputs rule) │ └───────────────────────────────────────────────┘ │ ┌──────────────────────┴──────────────────────┐ ▼ ▼ plugin_facts (append-only, plugin_state (compatibility view, the durable record): rebuilt automatically): one row per invocation, one row per plugin, {seq, fact_type, fact_json, ...} {plugin_name, state, updated_at} ``` The plugin author writes only the snapshot. Core does the rest. The compatibility view exists so legacy readers (the request envelope's `state` field, operator inspection, schedules that read prior state) keep working without change. ______________________________________________________________________ ## 4. Worked Examples ### 4.1 Minimal Plugin (no durable memory) A plugin that emits a single event when its `health` is checked. No durable state, no `fact_outputs` needed. **`plugins/notify_echo/manifest.yaml`:** ```yaml manifest_spec: ductile.plugin manifest_version: 1 name: notify_echo version: 0.1.0 protocol: 2 entrypoint: run.sh description: "Emits an echo event when polled. Stateless." concurrency_safe: true commands: - name: poll type: read description: "Emits one notify_echo.tick event." idempotent: true retry_safe: true values: consume: [] emit: - event: notify_echo.tick values: - payload.message - payload.emitted_at - name: health type: read description: "Reports plugin reachability." idempotent: true retry_safe: true config_keys: optional: [message] ``` **`plugins/notify_echo/run.sh`:** ```bash #!/usr/bin/env bash set -euo pipefail REQUEST=$(cat) COMMAND=$(echo "$REQUEST" | jq -r '.command') MESSAGE=$(echo "$REQUEST" | jq -r '.config.message // "tick"') case "$COMMAND" in poll) cat < str: return datetime.now(timezone.utc).isoformat() def snapshot_state(*, last_result, last_checked_at, last_triggered_at): """Pure constructor for the full compatibility snapshot. Every field is required at every call site — the helper never inherits silently. Callers that don't observe a field this invocation pass the prior state value explicitly. """ return { "last_result": last_result, "last_checked_at": last_checked_at, "last_triggered_at": last_triggered_at, } def poll_command(request): config = request.get("config", {}) state = request.get("state", {}) # Observe durable state. with sqlite3.connect(config["db_path"]) as conn: result = conn.execute(config["query"]).fetchone() scalar = str(result[0]) if result else None timestamp = now_iso() triggered = scalar != state.get("last_result") events = [] if triggered: events.append({ "type": config["event_type"], "payload": { "result": scalar, "previous_result": state.get("last_result"), "detected_at": timestamp, }, }) # Build the snapshot. The full compatibility-view shape is emitted every # time, even for fields this invocation did not change — the helper # guarantees that. return { "status": "ok", "result": f"observed result={scalar} triggered={triggered}", "events": events, "state_updates": snapshot_state( last_result=scalar, last_checked_at=timestamp, last_triggered_at=timestamp if triggered else state.get("last_triggered_at"), ), "logs": [{"level": "info", "message": f"polled: {scalar}"}], } def main(): request = json.load(sys.stdin) command = request.get("command") if command == "poll": response = poll_command(request) elif command == "health": response = {"status": "ok", "result": "healthy"} else: response = {"status": "error", "error": f"Unknown command: {command}", "retry": False} json.dump(response, sys.stdout) if __name__ == "__main__": main() ``` Notice: - `fact_outputs` declares `sqlite_change.snapshot` mirrored from `poll`'s `state_updates`. That single declaration is what makes core record an append-only fact every poll and rebuild `plugin_state` automatically. - `concurrency_safe: false` because the plugin owns a single-writer observation cycle. - `snapshot_state` is a pure constructor. Every field is explicit at every call site — no sentinel-`None` overlay, no implicit inheritance from prior state. The caller carries `last_triggered_at` forward by reading it from `state` and passing it explicitly. - The snapshot has the **same three keys every invocation**. A no-change poll emits a snapshot byte-identical to the prior one, which keeps `plugin_facts` free of reordering noise. - `health` does not return `state_updates` and does not mutate durable state. ______________________________________________________________________ ## 5. The `health` And `init` Pattern `health` and `init` are diagnostic-only. Neither should emit `state_updates`. The reasons are concrete: - `health` runs from the watch UI, the operator's CLI, and circuit-breaker half-open probes. If `health` mutated durable state, every diagnostic click would create a new fact with no observed change. - `init` returns an LLM/tool affordance bundle for capability discovery. Its output is a function of static metadata, not observed state. If a plugin's `health` or `init` is currently emitting `state_updates`, that is a bug — remove the emission. The first post-deploy `poll` or `token_refresh` will replace `plugin_state` wholesale via `mirror_object`, sweeping any historical residue. ______________________________________________________________________ ## 6. What Does Not Belong In `state_updates` These are the explicit non-candidates for `state_updates` / `fact_outputs`. None of them should live in `state_updates` or in a `fact_outputs` rule. They belong in `job_log` (which captures all of them automatically) or nowhere at all. | Pattern | Why it's wrong | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------- | | `last_run`, `last_invoked_at` | Action trace. Updates whether or not anything was observed. Use `job_log` for run history. | | `executions_count`, `total_calls` | Monotonic counter of actions taken. Not observed durable state. | | `last_pattern`, `last_prompt`, `last_video_id` | Single-field "the most recent thing I did" markers. Diagnostic, not durable observation. | | `last_summary`, `last_error_message` | Action diagnostics. Belongs in logs. | | `last_health_check`, `last_init_at` | Diagnostic timestamps from non-mutating commands. | | Diff or partial-patch shapes | The compatibility view is rebuilt wholesale; partial patches lose information on the next mirror. | | Lists derived from `set()` union | Non-deterministic order produces a different snapshot on every poll even when nothing changed. | If your plugin has a candidate field and you're not sure whether it's observed state or action bookkeeping, ask: *"if a downstream reader reads this field, are they learning about an external observation, or about my plugin's own behaviour?"* External observation belongs in the snapshot. Plugin behaviour does not. ______________________________________________________________________ ## 7. Event Payload Convention Plugins should follow standard payload field conventions for interoperability. These are *event* payload conventions, not state conventions — they live alongside the durability model, not in conflict with it. ### 7.1 Standard Fields | Field | Type | Purpose | Used By | | ------------- | ------ | ----------------------------------- | --------------------------------------------------- | | `text` | string | Primary text content for processing | **Required** if producing text for downstream steps | | `result` | string | Final human-readable output | Terminal plugins (fabric, summarizers) | | `source_url` | string | Originating URL | Web scrapers, YouTube fetchers | | `source_type` | string | Content origin hint | All plugins | ### 7.2 Source Types - `web` — web page content (jina-reader) - `youtube` — YouTube video transcript - `file` — local file content - `llm` — LLM-generated content (fabric, claude, etc.) ### 7.3 Event Type Naming `.`. Examples: - `jina_reader.scraped` - `youtube_transcript.fetched` - `fabric.completed` - `file_handler.read` - `file_handler.written` ### 7.4 Pipeline Integration The core dispatcher automatically propagates these payload names from input events to output events: - `pattern`, `prompt`, `model` - `output_dir`, `output_path`, `filename` Plugins **do not** need to manually copy these fields — the dispatcher handles propagation. Just emit your event with the standard fields and the pipeline DSL takes care of the rest. ### 7.5 Baggage (Context) Fallback Only payload names claimed by a pipeline's `baggage:` declaration become durable context entries in the `event_context` ledger. Downstream plugins receive that accumulated baggage in `request.context`. If a field may be produced by an upstream step, prefer: 1. Read from `event.payload` (step-specific input). 1. Fall back to `request.context` for accumulated values. This makes pipelines resilient when intermediate plugins emit narrower payloads. ### 7.6 Example Event Payload ```python return { "status": "ok", "result": "Scraped https://example.com", "events": [{ "type": "jina_reader.scraped", "payload": { "url": "https://example.com", "source_url": "https://example.com", "source_type": "web", "text": "Scraped content here...", "content_hash": "abc123" } }] } ``` ______________________________________________________________________ ## 8. Built-in Plugin: `if` Classifier The `if` plugin is a general-purpose field classifier. It evaluates an ordered list of checks against a payload field and emits the **first** matching event type, with the payload unchanged. ### 8.1 Config (per instance) ```yaml plugins: check_youtube: enabled: true config: field: text checks: - contains: "youtu.be" emit: youtube.url.detected - contains: "youtube.com" emit: youtube.url.detected - startswith: "http" emit: web.url.detected - default: text.received ``` ### 8.2 Supported Checks - `contains`, `startswith`, `endswith`, `equals` (case-insensitive) - `regex` (Python `re.fullmatch` against the field value) - `default` (always matches if reached) ### 8.3 Semantics - Checks are evaluated in order; first match wins. - Missing fields are treated as empty strings. - No match + no default → `status: error` with `retry: false`. Core treats that as a v2 compatibility signal for a permanent failure. ### 8.4 Instance Naming Ductile uses manifest names as plugin identities. To create multiple instances of `if` (or any plugin), use plugin aliasing in `plugins.yaml`: ```yaml plugins: check_youtube: uses: if # inherit the if plugin's implementation enabled: true config: field: text checks: [...] ``` The aliased instance has its own config, its own facts, and its own compatibility-view row. ______________________________________________________________________ ## 9. Filesystem & Diagnostic Bundles Ductile does not provision a workspace directory for plugins. The core is dispatch, state, and routing; filesystem is the plugin's concern. - **If your plugin needs a scratch path,** create it yourself. For ephemeral work prefer `mktemp -d` (or the language equivalent) and clean up on exit. For persistent caches use `~/.cache/ductile-/` or a path declared in your plugin config and validated at startup. - **If your plugin needs an archive of its own stdout** (for offline debugging or external log shipping), tee it from inside the run script before writing the response envelope, e.g. `tee -a "$HOME/.cache/myplugin/stdout.log"`. Core does not write subprocess stdout to disk on your behalf; the operationally meaningful fragments are already captured in the database (`job_log`, `plugin_facts`, `event_context`). - **Cwd.** An *unconfined* plugin inherits the dispatcher's working directory. A **confined** plugin (privsep enforce) is given a cwd of its own account `state_dir` — see §10.6. Write state relative to cwd; do not assume the gateway's directory. ## 10. Security & Isolation - **Allowed paths.** Plugins should only read/write paths they themselves create (per the previous section) or paths explicitly named in their config. - **Execution.** Without privsep, plugins run as the same OS user as the gateway. Under **privsep enforce** each plugin is dropped to a distinct, unprivileged account uid — see §10.6 for what that guarantees and forbids. - **Trust.** Ductile refuses to load plugins with world-writable directories or `..` in their `entrypoint`. The entrypoint must be `chmod +x`. - **No persistent state outside what is declared.** Plugins must not write to their own plugin directory at runtime (under privsep it is read-only). ## 10.6 Runtime contract (privsep) Under privsep enforce the gateway drops the plugin to an unprivileged **account** uid and roots its runtime at that account's own `0700` `state_dir`. This is the authoritative contract — it supersedes any older "same OS user / inherits the dispatcher cwd" assumption above. Full rationale: [`docs/adr/confined-plugin-runtime-contract.md`](https://ductile.run/adr/confined-plugin-runtime-contract/index.md). Working reference: [`plugins/_template/`](https://ductile.run/plugins/_template/index.md) (copy it). **A confined plugin may rely on:** - `cwd`, `$HOME`, and `$XDG_CACHE_HOME` all `==` its `state_dir` — writable, private, owned by no other account. Write state with relative paths. - Secrets in `request["secrets"]` — never the environment, never argv. - A writable, shared `/tmp`. **A confined plugin must NOT** (each fails closed): write outside its `state_dir` (or `/tmp`); read `$HOME` dotfiles, `/home/`, or ambient host credentials; or **fetch dependencies at spawn** (`uv run --script`, `pip`/`npm install`). The blessed default is **Tier 1: stdlib, fetch nothing at spawn**; reach for spawn-time dependency resolution only when a third-party library is unavoidable, and prefer building/vendoring it ahead of time into the read-only plugin dir. ## 10.5 Resource & Execution Limits To ensure system reliability, prevent memory exhaustion, and avoid rogue plugin processes from consuming all supervisor resources, Ductile enforces strict limits: - **Stdout Cap (10 MiB)**: The supervisor captures at most 10 MiB of data written to `stdout`. Exceeding this cap is considered a protocol/output failure, the supervisor terminates the process, and the job is marked `failed`. - **Stderr Cap (64 KiB)**: The supervisor captures at most 64 KiB of data written to `stderr` for diagnostics (which is saved in the database `job_log.stderr`). Excess `stderr` output is truncated with a warning in the core log stream. - **State Update Size (1 MiB)**: The JSON payload returned in `state_updates` (which represents the facts the plugin wishes to record or its compatibility state view updates) is strictly limited to 1 MiB per row. Updates exceeding this size are rejected, and the job is marked `failed`. - **Command Timeouts**: Standard lifecycle commands are strictly timed out to prevent zombie processes. The defaults are: - `poll`: 60s - `handle`: 120s - `health`: 10s - `init`: 30s Timeouts can be overridden per plugin in `config.yaml` under `plugins..timeouts`. ______________________________________________________________________ ## 11. Quick Quality Checklist When you finish a new plugin, walk this list before merging: - `manifest_spec`, `manifest_version`, `name`, `version`, `protocol: 2`, `entrypoint` set. - `description` is a real one-sentence summary. - `concurrency_safe` is honestly set (`false` if the plugin owns a single-writer durable resource). - Every command has `name`, `type`, `description`, and honest `idempotent` / `retry_safe` flags. - Standard commands (`poll`, `handle`, `health`, `init`) follow the conventions in §3.2. - `health` and `init` emit no `state_updates`. - Each command declares `values.consume` / `values.emit` so pipeline authors can see the contract. - If the plugin remembers anything across invocations, it declares `fact_outputs` for the durable command(s). - The emitted snapshot is a full object, deterministic, and has the same keys every invocation of that command (presence-stable). - Nothing in `state_updates` matches the §6 anti-patterns. - `config_keys.required` is honest — required keys must actually be required. - Entrypoint is `chmod +x`. - Tests cover the snapshot constructor and the response shape. Runtime contract (privsep — §10.6): - **Tier 1 by default:** stdlib / system runtime, `#!/usr/bin/env python3` (or equivalent) — **no dependencies fetched at spawn**. Spawn-time resolution (`uv run --script`, `pip`/`npm install`) only when a third-party lib is unavoidable, and documented as such. - Writes go **only under cwd / `state_dir`** (or `/tmp`) — never `$HOME` dotfiles, `/home/`, sibling account dirs, or system paths. - Secrets are read from `request["secrets"]`, never the environment or argv — and never logged, echoed, or emitted. - If the plugin needs a secret, the operator config carries `requires_vault: true` + a `vault_principal` (fail-closed); the code does not assume ambient credentials. If every box ticks, the plugin is aligned with the durability model and should not need a future migration sprint to fix. ______________________________________________________________________ ## Stopwatch — timing is captured for you The dispatcher times every plugin invocation automatically. You do not need to wrap your handler in `time.now()` calls; the supervisor records a `stopwatch.Record` (plugin id, step, monotonic duration, status, etc.) to the `job_stopwatch` table in the ductile DB. Telemetry is system data; it lives in the supervisor's ledger, not in your request context or response. See [PLUGIN_DIAGNOSTICS.md](https://ductile.run/PLUGIN_DIAGNOSTICS/index.md) for the data shape and the `gateway_time` formula. ### Optional sub-spans for plugin-internal phases If you want to break down what your handler did internally (`db_query`, `http_call`, parsing, etc.), emit a list under `ductile_stopwatch_subs` at the top level of your response. Shape: ```json { "status": "ok", "result": "...", "ductile_stopwatch_subs": [ {"name": "fetch.http_get", "dur_ns": 31000000, "status": "ok"}, {"name": "fetch.body_read", "dur_ns": 11000000, "status": "ok", "bytes": 482103}, {"name": "fetch.decode", "dur_ns": 2400000, "status": "ok"} ] } ``` #### Field convention | Field | Required | Notes | | -------- | -------- | ---------------------------------------------------------------------------------------- | | `name` | yes | Dotted name `.` (e.g. `fetch.http_get`). Prefix enables filtering. | | `dur_ns` | yes | Monotonic duration in nanoseconds. Use `time.perf_counter_ns()` deltas. | | `status` | optional | `ok` / `err` / `skip`. Explains zero-duration or partial spans. | | `bytes` | optional | For I/O spans (body reads, file hashes). Quartile bytes vs. dur_ns to find slow servers. | | `count` | optional | For batch spans (files scanned, watches polled, retries attempted). | The dispatcher stores additional fields verbatim, but the convention above is what downstream tools query. New fields should be added by RFC, not by ad-hoc plugin choice — the convention is only as strong as its exemplars. See `plugins/_lib/_stopwatch.py` for a vendored Python helper that emits this shape from a context-manager API. The four exemplar plugins (`fetch`, `file_watch`, `folder_watch`, `sys_exec`) use it. #### Rules - **Sub-spans are advisory.** The dispatcher's own Record is always emitted regardless of whether you include sub-spans. A buggy or lying plugin poisons its own breakdowns only; the supervisor's timing is independent. - **The dispatcher caps the list at 32 entries per invocation by default.** If you emit more, the excess is dropped (head-keep — first 32 survive) and one warning is logged for the call. **Order matters:** put high-signal spans first. The default is appropriate for almost all plugins; see "Raising the cap" below before considering an override. - **Malformed entries are dropped silently.** Non-object items and non-list values do not raise. Defensive parsing is part of the contract. - **The dispatcher does not interpret sub-span fields beyond storing them.** Field semantics are the plugin's responsibility. Follow the convention above so quartile dashboards work across plugins. #### Raising the cap (rare) A plugin that legitimately produces more than 32 spans — typically a multi-stage pipeline coordinator with structurally distinct sub-phases — may declare a higher cap in its `manifest.yaml`: ```yaml stopwatch: max_subs: 64 ``` Range: `[1, 256]`. The hard upper of 256 is a system-wide invariant — every consumer of `subs_json` (DB row, API response, log line, future dashboards) needs a stable budget. A manifest declaring `max_subs > 256` is rejected at plugin load with a warn-level log; the plugin is not registered. Manifests omitting the field, or setting it to 0, use the default 32 — no behaviour change for existing plugins. Before adding a `stopwatch` block, ask: can the spans be aggregated? The default cap is a design forcing function. Aggregation patterns (above) will handle 95% of cases that initially look like "I need more spans." #### Aggregation patterns The 32 cap pushes you toward aggregation over per-event tracing. Patterns that work well within the cap: | Anti-pattern | Pattern | | ---------------------------------------- | ----------------------------------------------------------------------- | | One span per file in a 5000-file scan | One `.fingerprint_total` with `count=5000`, `bytes=...` | | One span per HTTP retry | One `.http_get` with `count=` annotation | | One span per loop iteration | Histogram buckets: `.loop.bucket_0_10ms`, `…bucket_10_100ms`, … | | One span per polled watch (many watches) | Aggregate totals, plus **outlier-only** per-watch spans (`> 50ms`) | For real per-event tracing, use an external tracing backend (OTel) — the stopwatch ledger is for quartile-grade aggregation, not timeline reconstruction. # Webhooks This document summarizes how webhooks work in Ductile and how to configure them safely. ## Overview - Webhooks run on a dedicated HTTP listener and enqueue plugin jobs. - Every webhook requires **HMAC-SHA256** signature verification. - Webhooks are configured in **`webhooks.yaml`** (a high-security file). - The webhook listener is configured via `webhooks.listen` in `config.yaml`. ## Configuration Model Only `config.yaml` is loaded automatically. Any webhook configuration must be referenced via the `include:` list: ```yaml include: - api.yaml - plugins.yaml - webhooks.yaml ``` If `webhooks.yaml` is not included, the listener will start **without endpoints**, and requests will return 404. ## File Layout Typical config root: ```text ~/.config/ductile/ ├── config.yaml ├── api.yaml ├── plugins.yaml ├── webhooks.yaml └── .checksums ``` ## webhooks.yaml Format ```yaml webhooks: endpoints: - name: astro_rebuild_staging path: /webhook/astro-rebuild-staging plugin: astro_rebuild_staging secret_ref: astro_webhook_secret signature_header: X-Ductile-Signature-256 max_body_size: 1MB # optional ``` Notes: - `secret_ref` is required; it resolves against the vault (preferred — `ductile vault set`) or a legacy `tokens.yaml` entry. - `signature_header` is mandatory. - `max_body_size` defaults to 1MB. ## Listener Port Set in `config.yaml`: ```yaml webhooks: listen: "127.0.0.1:8091" ``` ## Triggering a Webhook Example HMAC signature (GitHub-style `sha256=` prefix is supported): ```bash payload='{"payload":{"reason":"manual rebuild"}}' sig=$(printf '%s' "$payload" | \ openssl dgst -sha256 -hmac '' -hex | awk '{print $2}') curl -sS -X POST http://127.0.0.1:8091/webhook/astro-rebuild-staging \ -H "Content-Type: application/json" \ -H "X-Ductile-Signature-256: sha256=$sig" \ -d "$payload" ``` Response returns a `job_id` when accepted. ## Security & Integrity - `webhooks.yaml` is **high security** and must be sealed with `ductile config lock`. - In `strict_mode: true`, tampering will prevent startup. - Use `secret_ref` for secret management in production; hold the secret in the vault (`ductile vault set`) — `tokens.yaml` is the migrating-out legacy. See `docs/SECRETS.md`. ## Operational Checks - Webhook listener health: `GET /healthz` on the webhook listener port. - Verify endpoints are loaded: ```bash ductile config show --config-dir ~/.config/ductile | rg -n "webhooks" -C 3 ``` # YAML Tips for Ductile Configuration Practical techniques for keeping your `plugins.yaml` clean and maintainable. No prior YAML expertise assumed. ______________________________________________________________________ ## Anchors and Aliases: Stop Repeating Yourself ### The problem When you create multiple plugin instances that share the same base settings, you end up copy-pasting the same values over and over: ```yaml plugins: discord_test_cron: uses: discord_notify enabled: true timeout: 15s max_attempts: 2 schedules: - cron: "*/5 * * * *" command: poll config: webhook_url: "https://discord.com/api/webhooks/123/abc..." default_username: "Ductile" poll_message: "[T2] cron */5min" discord_test_window: uses: discord_notify enabled: true # repeated timeout: 15s # repeated max_attempts: 2 # repeated schedules: - every: 3m only_between: "07:00-22:00" command: poll config: webhook_url: "https://discord.com/api/webhooks/123/abc..." # repeated default_username: "Ductile" # repeated poll_message: "[T3] only_between 07-22" ``` If you want to change the timeout or webhook URL, you have to edit every block. That's fragile and error-prone. ### The solution: YAML anchors YAML has a built-in mechanism for this called **anchors** (`&`) and **aliases** (`*`). You define a block once with an anchor, then reference it by name anywhere else. The YAML parser expands the reference before any application ever reads the file — it's a pure YAML feature, not a ductile feature. #### Step 1 — Define an anchor An anchor is a name you attach to any YAML value using `&name`. It can go on a mapping (dict), a sequence (list), or a scalar (string/number). ```yaml # This defines an anchor called "discord-test" on this mapping block. # The key name ("x-discord-test") is arbitrary — pick something descriptive. x-discord-test: &discord-test uses: discord_notify enabled: true timeout: 15s max_attempts: 2 ``` #### Step 2 — Reference it with an alias Wherever you want to reuse that block, write `*anchor-name`. The YAML parser replaces it with the full content of the anchored block. ```yaml discord_test_cron: *discord-test # expands to: uses, enabled, timeout, max_attempts schedules: - cron: "*/5 * * * *" command: poll ``` But wait — you also need to *add* fields on top of the expanded block (like `schedules`). For that, use the **merge key** `<<:`. #### Step 3 — Merge with `<<:` `<<: *anchor-name` merges the referenced block's keys into the current mapping. Keys you define explicitly take priority over merged ones. ```yaml discord_test_cron: <<: *discord-test # merges: uses, enabled, timeout, max_attempts schedules: # adds this new key on top - cron: "*/5 * * * *" command: poll ``` This is equivalent to writing all four merged keys out explicitly, plus the `schedules` key. #### Step 4 — Anchors work on nested mappings too You can anchor the `config:` sub-block separately: ```yaml x-discord-config: &discord-config webhook_url: "https://discord.com/api/webhooks/123/abc..." default_username: "Ductile" ``` Then merge it inside the `config:` block of each plugin, adding only the field that differs: ```yaml discord_test_cron: <<: *discord-test schedules: - cron: "*/5 * * * *" command: poll config: <<: *discord-config # merges: webhook_url, default_username poll_message: "[T2] cron" # adds the unique field ``` ### Why this works with ductile There are two things worth understanding: **1. The YAML parser resolves anchors before ductile sees anything.** When ductile loads `plugins.yaml`, the YAML library reads the file and fully expands all anchors and merge keys first. By the time ductile's config loader processes the result, it just sees a normal, fully-populated config map. Ductile has no idea anchors were used. **2. Unknown top-level keys are silently ignored.** The anchor definitions live at the *top level* of the YAML file, outside the `plugins:` block. Ductile's config loader only reads keys it knows about (`plugins:`, `service:`, `pipelines:`, etc.). Anything else — including `x-discord-test:` and `x-discord-config:` — is silently ignored. This is why the anchor names are prefixed with `x-` by convention: it signals "this is application-level metadata, not ductile config". Any name works, but a consistent prefix avoids confusion. ### Full before/after example **Before** (80 lines, webhook URL repeated 5 times): ```yaml plugins: discord_test_cron: uses: discord_notify enabled: true timeout: 15s max_attempts: 2 schedules: - cron: "*/5 * * * *" command: poll config: webhook_url: "https://discord.com/api/webhooks/123/abc..." default_username: "Ductile Scheduler" poll_message: "[T2] cron */5min" discord_test_window: uses: discord_notify enabled: true timeout: 15s max_attempts: 2 schedules: - every: 3m only_between: "07:00-22:00" command: poll config: webhook_url: "https://discord.com/api/webhooks/123/abc..." default_username: "Ductile Scheduler" poll_message: "[T3] only_between 07-22" # ... and so on for each test instance ``` **After** (webhook URL in one place, instances are concise): ```yaml # Shared base — expanded by YAML parser, ignored as a key by ductile x-discord-test: &discord-test uses: discord_notify enabled: true timeout: 15s max_attempts: 2 x-discord-config: &discord-config webhook_url: "https://discord.com/api/webhooks/123/abc..." default_username: "Ductile Scheduler" plugins: discord_test_cron: <<: *discord-test schedules: - cron: "*/5 * * * *" command: poll config: <<: *discord-config poll_message: "[T2] cron */5min" discord_test_window: <<: *discord-test schedules: - every: 3m only_between: "07:00-22:00" command: poll config: <<: *discord-config poll_message: "[T3] only_between 07-22" ``` ### Caveats **Merge is shallow.** `<<:` merges the *top-level keys* of the anchored block. It does not deep-merge nested structures. If your anchor contains a `config:` block, and you also define a `config:` block in the instance, the instance's `config:` block wins entirely — the anchor's `config:` is not merged into it. This is why the example anchors `config` separately (as `*discord-config`) and merges it *inside* the `config:` block. **Explicit keys win over merged keys.** If a key appears in both the `<<:` block and the current mapping, the explicitly written key takes precedence. Use this to override specific values from the shared base. **Anchors are file-scoped.** An anchor defined in `plugins.yaml` cannot be referenced from `pipelines.yaml`. If you use modular config files, you need to repeat shared values across files or consolidate into a single file. **`config check` still validates the expanded result.** Anchors are transparent to ductile's validator. If a merged value is invalid, the error will point to the plugin instance, not the anchor definition. ______________________________________________________________________ ## Further reading - [YAML specification — anchors and aliases](https://yaml.org/spec/1.2.2/#anchors-and-aliases) - [YAML specification — merge keys](https://yaml.org/type/merge.html) - Ductile config reference: `CONFIG_REFERENCE.md` - Plugin instance aliasing (`uses:`): `COOKBOOK.md` # Operating # Ductile Deployment Guide This document describes how to deploy a host-local Ductile instance as a systemd user service. It reflects the first reference deployment on `matt-ThinkPad-T14s-Gen-1` (2026-02-22) and is the canonical procedure for repeating this on other hosts. See also: RFC-006 (local execution plane topology). ______________________________________________________________________ ## 1. Build the Binary Build from source and install to the user's local bin: ```bash cd /path/to/ductile go build -o ~/.local/bin/ductile ./cmd/ductile ``` Verify: ```bash ductile --version ``` The binary is self-contained — no additional runtime dependencies. ______________________________________________________________________ ## 2. Directory Layout Create a deployment root with separate `config/` and `data/` directories: ```text ductile-local/ ├── config/ │ ├── config.yaml # main config (includes others via `include:`) │ ├── api.yaml # API listen address + auth tokens │ └── plugins.yaml # plugin enable/config └── data/ ├── ductile.db # SQLite state DB (created on first start) └── outputs/ # write target for file_handler plugin ``` Create it: ```bash mkdir -p ~/admin/ductile-local/config ~/admin/ductile-local/data/outputs ``` ______________________________________________________________________ ## 3. Split Config Pattern Ductile supports modular ("grafted") configs via the `include:` key. The main config file sets global options and includes the others by relative path. ### config/config.yaml ```yaml log_level: info state: path: ./data/ductile.db plugin_roots: - /path/to/ductile/plugins include: - api.yaml - plugins.yaml ``` `plugin_roots` is a list of directories to scan for plugin executables at startup. Any plugin binary found here is *discovered*; only plugins listed in `plugins.yaml` are *configured* (and those not listed emit a warning but still load). ### config/api.yaml API bearer tokens are **vault-only**: a literal `token:` value in config is rejected at load (#94, credential-ladder ADR §8.5). The config references a vault secret by name; the bootstrap walk that mints it is [BOOTSTRAP.md](https://ductile.run/BOOTSTRAP/index.md) steps 4–6. ```yaml api: enabled: true listen: "127.0.0.1:8081" management_socket: /tmp/ductile-admin.sock # serves /vault/* while bootstrapping # No tokens yet = the bootstrap (management-only) posture when a vault exists. # AFTER minting, reference the secret and restart to activate the gateway: # auth: # tokens: # - secret_ref: core-api-token # scopes: ["*"] ``` Generate a value and mint it into the vault over the management socket: ```bash API_TOKEN=$(openssl rand -hex 32) printf '%s' "$API_TOKEN" | ductile vault set --api-url "unix:///tmp/ductile-admin.sock" \ --token "$ADMIN_TOKEN" --name core-api-token --pattern manual ``` Store the same value for API *clients* (the gateway itself never reads this): ```bash # ~/.zshrc export DUCTILE_LOCAL_TOKEN= ``` ### config/plugins.yaml ```yaml plugins: fabric: enabled: true timeout: 120s max_attempts: 2 config: FABRIC_DEFAULT_PATTERN: "summarize" file_handler: enabled: true timeout: 30s max_attempts: 1 config: allowed_read_paths: "${HOME}" allowed_write_paths: "${HOME}/ductile-local/data/outputs" jina-reader: enabled: true timeout: 30s max_attempts: 3 circuit_breaker: threshold: 3 reset_after: 5m config: {} ``` ______________________________________________________________________ ## 4. Validate Config Before starting the service, validate the configuration: ```bash cd ~/admin/ductile-local ductile config check --config config/config.yaml ``` Expected output: ```text Configuration valid (N warning(s)) WARN [unused] plugin "echo" discovered but not referenced in config ... ``` Warnings about undeclared plugins are expected if `plugin_roots` contains plugins you haven't explicitly configured. They are loaded but not usable without config entries. ______________________________________________________________________ ## 5. systemd User Service Create `~/.config/systemd/user/ductile-local.service`: ```ini [Unit] Description=Ductile Gateway (local prod) After=network.target [Service] Type=simple WorkingDirectory=${HOME}/ductile-local ExecStart=${HOME}/.local/bin/ductile system start --config config/config.yaml Restart=on-failure RestartSec=5s StandardOutput=journal StandardError=journal [Install] WantedBy=default.target ``` Enable and start: ```bash systemctl --user daemon-reload systemctl --user enable --now ductile-local ``` Check status: ```bash systemctl --user status ductile-local journalctl --user -u ductile-local -f ``` ______________________________________________________________________ ## 5b. Privsep — system service with account users (opt-in) The `--user` service above runs every plugin as your own uid (hygiene-only, ADR Layer 1a). To contain a *popped* plugin — so it cannot read the gateway's secrets or another plugin's memory — run the gateway as a **system service holding only `CAP_SETUID`+`CAP_SETGID`** and drop each plugin to an unprivileged **account** user (ADR Layer 1b). Templates live in [`deploy/systemd/`](https://ductile.run/deploy/systemd/index.md). **The model:** the binary is never setuid — privilege is *init-conferred*. The gateway runs as the unprivileged `ductile` account with exactly the two capabilities, just enough to setuid each plugin to its account at spawn. A cap-only gateway holds **no `CAP_CHOWN`**, so the accounts and their `0700` state dirs are provisioned by the init layer (`sysusers.d` + `tmpfiles.d`); the gateway only **verifies** them at boot and **refuses to start** if they are wrong (fail-closed). Install once, as root: ```bash sudo install -m0644 deploy/systemd/ductile-accounts.sysusers.conf /etc/sysusers.d/ductile-accounts.conf sudo install -m0644 deploy/systemd/ductile-accounts.tmpfiles.conf /etc/tmpfiles.d/ductile-accounts.conf sudo systemd-sysusers # create ductile + account users sudo systemd-tmpfiles --create /etc/tmpfiles.d/ductile-accounts.conf # create 0700 account dirs sudo install -m0644 deploy/systemd/ductile.service /etc/systemd/system/ductile.service sudo systemctl daemon-reload && sudo systemctl enable --now ductile ``` Match the config `accounts:` map to the provisioned uids/dirs: ```yaml accounts: default: { uid: 1001, gid: 1001, state_dir: /var/lib/ductile/accounts/default } untrusted: { uid: 1002, gid: 1002, state_dir: /var/lib/ductile/accounts/untrusted } plugins: sys_exec: { run_as: untrusted } # arbitrary-command → isolated tier # ungranted first-party plugins fall back to the shared `default` tier ``` > **One source of truth you hand-maintain (T11).** The same `{uid, gid, state_dir}` triple is written in **three** places that must agree exactly: the config `accounts:` map (above), `ductile-accounts.sysusers.conf` (which *creates* the users), and `ductile-accounts.tmpfiles.conf` (which *creates* the `0700` state dirs). There is no generator — you keep them in sync by hand. The safety net is that the gateway **verifies the coupling at boot and refuses to start** if it disagrees (a `run_as:` grant naming an account absent from the map fails even earlier, at config *load*). So a mismatch costs you a loud crash-loop, never a silent half-confined run — but the coupling itself is yours to maintain. Keep the three files adjacent in review for that reason. **The boot gate is fail-closed (ADR §5):** capability and a configured `accounts` map must agree. A privileged gateway with no accounts, or accounts configured on a host without the capability, **refuses to start** — never a silent run at gateway privilege. The one escape hatch is an explicit `service.unconfined: true`. The full truth table and the new boot-time refusals/warnings are in [§5c](#5c-privsep-config-reference). > **Dev gotcha — root with no accounts refuses to boot, by design (T13).** If you start the daemon as **root** (or via `sudo`) on a box with **no `accounts:` map** — the common shape when poking at it locally — the boot gate sees *capability held, nothing to drop to* and **refuses to start**: `privsep boot gate: gateway holds the uid-drop capability but no accounts are configured`. This is correct, not a regression — a privileged daemon with no wall to build is the worst case. The one-line fix for local/dev work is the explicit, loud escape hatch: > > ```yaml > service: > unconfined: true # run at the gateway uid on purpose; logged loudly at boot > ``` > > Or simply don't run the daemon privileged in dev — an ordinary `--user` service with no `accounts:` map runs `unconfined` quietly (the dev cell of the boot gate). Reserve root + `accounts:` for the real enforcing deployment. **Utility subcommands stay unprivileged:** `ductile config validate`, `secrets keygen`, etc. run as the *caller* (the binary is not setuid), so they cannot read the `0600` root-owned age key — exactly as intended. **macOS (launchd): enforce is Linux-proven, macOS-pending (T12).** Privilege-dropping enforce mode is proven only on a privileged **Linux** host so far; the launchd equivalent and the live rollout are still open — see `kanban/95-privsep-launchd-and-live-rollout.md` (and [BOOTSTRAP.md](https://ductile.run/BOOTSTRAP/index.md) plus `deploy/install-macos.sh` for the install shape). Until then a Mac runs **hygiene-only / `unconfined`** (no `accounts:` map, no drop capability under launchd), which the boot gate permits quietly. Do not assume a Mac gateway is confined. ### Docker / Unraid — hygiene-only by default (decision, card #89) **The Docker/Unraid image stays hygiene-only (ADR Layer 1a) — no account drop.** The image runs `USER ductile` (unprivileged), and adopting full privsep there would *raise* the container's privilege (run as root or with `SETUID`/`SETGID` caps), which is the worst trade on the one host where a privileged container is most costly. The ADR explicitly allows hygiene-only as a legitimate per-host default, and at one author it is the honest choice. You lose nothing silently: with **no `accounts:` map configured**, the boot gate runs the gateway `unconfined` — exactly today's behaviour — and 1a still bounds the spawn (env allowlist + secrets only over stdin). And the gate is **fail-closed against misconfiguration**: if you *do* add an `accounts:` map to a container that lacks the drop capability, the gateway **refuses to start** rather than presenting a wall it cannot enforce. So "hygiene-only" here is a deliberate, safe state, not a silent gap. *If you ever want full uid separation in a container* (e.g. running `sys_exec` on Unraid): run it `--cap-add=SETUID --cap-add=SETGID` (prefer caps over root), bake the account uids into the image, provision the per-account `0700` dirs on a persistent volume (the tmpfiles.d equivalent), and bind-mount the `0600` age key from the host (no TPM/Keychain in a container). That is opt-in and intentionally undocumented as a default — the homelab floor is hygiene-only. ______________________________________________________________________ ## 5c. Privsep config reference Lookup-only companion to the §5b how-to: every privsep config key, the boot-gate outcomes, the reserved tier names, and the failure modes — in one greppable place. For *why* it is shaped this way, see the ADR (`Ductile - PrivSec and Secrets.md`). ### Config keys | Key | Type | Meaning | | --------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `accounts:` | map (open) | Privsep account tiers, keyed by name. Each value is an account identity. Absent/empty map → boot gate decides posture (below). | | `accounts..uid` | int > 0 | Unprivileged user id the plugin is dropped to. Never `0`/root. Provisioned by `sysusers.d`; referenced here by number. No two accounts may share a uid (false isolation). | | `accounts..gid` | int > 0 | Unprivileged group id. | | `accounts..state_dir` | absolute path | The account's owned, persistent `0700` dir (created by `tmpfiles.d`, reconciled/verified at boot). | | `plugins..run_as` | string | The account this plugin is granted (names an `accounts:` entry). Empty/absent = no grant → falls back to the `default` tier, or `unconfined` if no `default` exists. The operator grants this — a plugin manifest hint is never trusted. | | `service.unconfined` | bool (default `false`) | Boot-gate escape hatch: run plugins at the gateway uid (no drop) even on a capable/configured host. Logged loudly. The only sanctioned way to opt out of enforcement. | ### Reserved tier keywords (T14) Two account names are matched **by name in code** — they are not ordinary tiers: | Name | Role | Absence behavior | | ----------- | ---------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `default` | The tier an **ungranted** plugin (no `run_as:`) falls back to. | No `default` tier → ungranted plugins run **`unconfined`** (gateway uid), and the gateway **warns at boot**. | | `untrusted` | The most-restricted tier a **fingerprint-mismatched** plugin is downgraded to (supply-chain-swap defence). | No `untrusted` tier → a fingerprint mismatch has no downgrade target, so that plugin's spawn is **terminal/no-retry** (`ErrNoDowngradeTarget`), and the gateway **warns at boot**. | `unconfined` is **not** an account name — it is the named *no-drop state* (gateway uid). Never define an account called `unconfined`; reach the state via an absent `accounts:` map or `service.unconfined: true`. ### Boot-gate outcomes (capability × accounts) Evaluated once at startup and on reload (`internal/dispatch/bootgate.go`). The drop **capability** is `CAP_SETUID`+`CAP_SETGID` (or root); **accounts configured** means a non-empty `accounts:` map. | | accounts configured | no accounts configured | | ------------------- | ----------------------------------------------------------- | --------------------------------------------------------------- | | **capability held** | **enforce** — drop each plugin to its account | **REFUSE to start** — privileged daemon with nothing to drop to | | **no capability** | **REFUSE to start** — a wall declared the host cannot build | **`unconfined`** — quiet info log (today's dev behaviour) | The single invariant: **capability and accounts-configured must agree.** Disagreement either direction is a fatal boot error. The one override is `service.unconfined: true`, which forces `unconfined` from any cell and is logged loudly. ### Failure modes (when privsep stops a boot or a spawn) | Condition | When it fires | Outcome | | --------------------------------------------------------------------------------------- | --------------------------- | --------------------------------------------------------------------------------- | | `run_as:` names an account absent from `accounts:` (typo, or no `accounts:` map at all) | config **load** | config rejected — daemon does not boot | | capability ↔ accounts disagree (the two REFUSE cells above) | boot gate | refuse to start (unless `service.unconfined: true`) | | secrets surface or an `accounts` `state_dir` is foreign-owned / un-tightenable | boot (filesystem reconcile) | refuse to start (all-or-refuse; never half-confined) | | no `default` tier defined while enforcing | boot | **warn**; ungranted plugins run `unconfined` | | no `untrusted` tier defined while enforcing | boot | **warn**; a later fingerprint mismatch has no downgrade target | | plugin fails fingerprint attestation, `untrusted` tier exists | spawn | downgraded to `untrusted` (runs, contained) | | plugin fails fingerprint attestation, no `untrusted` tier | spawn | `ErrNoDowngradeTarget` — **terminal, no retry** | | the uid/gid drop itself is refused by the kernel at spawn | spawn | `ErrAccountDropFailed` — **terminal, no retry** (never falls back to gateway uid) | ______________________________________________________________________ ## 6. Verification Checklist After starting the service, verify the following: ```bash # Health — no auth required curl http://localhost:8081/healthz # Expected: # {"status":"ok","uptime_seconds":N,"queue_depth":0,"plugins_loaded":5,"plugins_circuit_open":0} # Plugin list — requires auth curl -H "Authorization: Bearer $DUCTILE_LOCAL_TOKEN" http://localhost:8081/plugins # OpenAPI schema — no auth required curl http://localhost:8081/openapi.json | head -20 ``` Confirm: - [ ] `status: ok` in healthz - [ ] `plugins_loaded` > 0 - [ ] fabric, file_handler, jina-reader appear in `/plugins` - [ ] `/openapi.json` returns valid JSON ______________________________________________________________________ ## 7. RFC-006 Topology Notes RFC-006 defines two Ductile instance roles: | Role | Purpose | | ------------------- | ----------------------------------------------------------------- | | **Boundary node** | Public-facing gateway, handles external API calls, auth, routing | | **Host-local node** | Per-host execution plane, runs plugins with local resource access | This deployment is a **host-local node**: - Listens on `localhost` only (not exposed to LAN) - Token scoped to `["*"]` for local use - Plugins have access to local filesystem (`file_handler`) and local tools (`fabric`) - Receives work dispatched from a boundary node or local AgenticLoop agent The prod Unraid instance (`192.168.20.4:8888`) is the boundary node for this network. ______________________________________________________________________ ## 8. Updating the Binary When a new version is built: ```bash # Stop the service first (optional but clean) systemctl --user stop ductile-local # Rebuild cd /path/to/ductile go build -o ~/.local/bin/ductile ./cmd/ductile # Restart systemctl --user start ductile-local systemctl --user status ductile-local ``` Or just rebuild and restart in one shot — the service will pick up the new binary on next start: ```bash cd /path/to/ductile && go build -o ~/.local/bin/ductile ./cmd/ductile && systemctl --user restart ductile-local ``` ## 9. Schema Migrations Before Deploy If a release adds additive SQLite schema, apply the matching migration script to the existing state DB before the normal deploy/restart. This is especially relevant for instances that already have a populated database. The binary still carries the mono-schema for fresh databases, but the preferred operational path for existing databases is to run explicit migrations first so schema changes are intentional and visible in deployment steps. For non-empty existing databases, Ductile validates schema on startup instead of silently adding missing upgrade-era tables or indexes. If the DB is behind, startup should fail with a migration hint rather than mutating the schema implicitly. ## 10. Backups `ductile system backup` writes an atomic, point-in-time snapshot of the SQLite state DB plus selected runtime artefacts into a single `tar.gz` archive. The DB snapshot is taken via `VACUUM INTO`, which is safe under concurrent writers — no service stop required. ```bash ductile system backup --to [--scope SCOPE] [--config PATH] ``` The four scopes are a nested ladder; each level adds to the previous: | Scope | Contents | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `db` | `VACUUM INTO` snapshot of the state DB only | | `config` (default) | `db` + ductile config dir (`config.yaml`, `api.yaml`, `plugins.yaml`, `pipelines.yaml`, `webhooks.yaml`, `.checksums`) + the encrypted vault blob `vault.age` (the age key is **excluded** — out-of-band custody; restore needs both) | | `plugins` | `config` + every directory under `plugin_roots` (excludes `.git`, `node_modules`, `.venv`, `venv`, `__pycache__`, `.DS_Store`, `*.pyc`, `*.pyo`) | | `all` | `plugins` + every file referenced under `environment_vars.include` | Each invocation prints its INCLUDED / EXCLUDED list to stdout before doing the work and embeds a `BACKUP_MANIFEST.txt` inside the archive recording the same information plus ductile version, commit, hostname, source paths, source DB sha256, and any boundary warnings (e.g. `api.yaml` at `config` scope, env files at `all` scope). Refuses to overwrite an existing destination — operator owns naming and retention via shell glue. ### Scheduled backups systemd-timer (Thinkpad pattern) — `~/.config/systemd/user/ductile-backup.service`: ```ini [Unit] Description=Ductile backup snapshot [Service] Type=oneshot Environment=BACKUP_DIR=%h/admin/ductile-backups/thinkpad/auto ExecStart=/bin/sh -c 'mkdir -p "$BACKUP_DIR" && \ STAMP=$(date -u +%%Y%%m%%dT%%H%%M%%SZ) && \ %h/.local/bin/ductile system backup \ --to "$BACKUP_DIR/ductile-$STAMP.tar.gz" --scope config && \ find "$BACKUP_DIR" -name "ductile-*.tar.gz" -mtime +7 -delete' ``` Paired timer `~/.config/systemd/user/ductile-backup.timer`: ```ini [Unit] Description=Nightly ductile backup at 03:00 local [Timer] OnCalendar=*-*-* 03:00:00 Persistent=true [Install] WantedBy=timers.target ``` Enable: ```bash systemctl --user daemon-reload systemctl --user enable --now ductile-backup.timer ``` launchd (Mac pattern) — equivalent `LaunchAgent` plist with `StartCalendarInterval` runs the same command sequence; see existing `com.mattjoyce.ductile-local.plist` as a template for the `ProgramArguments` shape. Pre-migration backups before any breaking schema change are a separate manual invocation under `~/admin/ductile-backups//pre--/` — they sit outside the auto-rotation directory. ## 11. Deploying the Vault onto an Instance This is the runnable procedure for bringing up the **vault** (daemon-owned secret delivery + plugin attestation) on an existing host-local instance. It is *how-to* only — the steps to satisfy the need. For the **why** (the vault's mental model, the sole-writer split, compose-time attestation, spawn hygiene, the backup/key pairing) see [SECRETS.md](https://ductile.run/SECRETS/index.md); the short "Why this order" note at the end of this section covers only the deploy-specific theory. > First reference run: the Thinkpad (`matt-ThinkPad-T14s-Gen-1`), 2026-06-05. ### Order at a glance backup → `vault_audit` migration → age key + genesis → reconcile config.yaml → boot vault-operable posture + mint secrets over the admin socket → **`config lock` + `plugin lock --all`** → cutover (reload activates the gateway) → verify. **The two steps operators miss — both fail loud, both cost a crash-loop:** - **`plugin lock --all` is separate from `config lock`.** Plugin attestation fingerprints are *not* sealed by `config lock`. Without them, `verify_integrity_on_boot` rejects every plugin at startup. (They were deliberately decoupled — config integrity and plugin attestation are different concerns.) - **`validate_config_on_boot` surfaces dead config keys.** The strict admission gate rejects keys the lenient loader silently dropped — a misplaced `log_level`, a per-plugin `timeout`/`max_attempts` that belongs under `timeouts:`/`retry:`, a typo. Fix the keys (`config check` names them), or the daemon refuses to boot. ### Procedure ```bash # Build the new binary per §1/§8 but STAGE it — don't install yet; run the gates # against it while the old binary still serves. NEW=~/staging/ductile-new # freshly built branch binary CFG=~/.config/ductile KEY=~/.config/secrets/ductile/age.key # out-of-band; NOT inside $CFG # 1. Rollback point: back up DB + config, and snapshot the current binary. ductile system backup --to ~/backups/pre-vault-$(date -u +%Y%m%dT%H%M%SZ).tar.gz \ --scope config --config "$CFG" cp ~/.local/bin/ductile ~/backups/ductile-prev # 2. Schema: add the vault_audit table. Idempotent, hot-safe. This is OBSERVABILITY, # not a boot gate — the daemon boots without it; the audit writer just fails soft. python3 /path/to/ductile/scripts/migrate-add-vault-audit-table.py "$CFG/ductile.db" # 3. Age key (record the public recipient) + genesis. Daemon STOPPED for genesis. ductile secrets keygen --out "$KEY" # mode 0600 systemctl --user stop ductile-local "$NEW" vault init --vault "$CFG/vault.age" --key "$KEY" \ > ~/.config/secrets/ductile/genesis.out 2>&1 # admin token printed ONCE chmod 600 ~/.config/secrets/ductile/genesis.out # capture the token from here; it is the API credential # Capture-and-rotate hygiene: lift the token into 0600/0700 custody (or a password # manager), then SHRED genesis.out. If the token was ever exposed (shared channel, # client log, screen), roll it in place — no re-genesis — while the daemon is stopped: # ductile vault rotate-admin-token --config "$CFG" # mints + prints the NEW token once # The old token dies immediately; update DUCTILE_VAULT_TOKEN before any API write. # See SECRETS.md § "Rotating the admin token". # 4. Reconcile config.yaml, then validate. Set: # secrets.age_key_file: # api.management_socket: $CFG/vault-admin.sock # EXPLICIT — the built-in default # # lives beside the STATE DB (state/ or data/), not $CFG (#141) # service.admission: { verify_integrity_on_boot: true, fail_on_drift: true, # validate_config_on_boot: true, require_api_auth: true } # service.plugin_env_passthrough: [ ... ] # only env names a plugin actually reads "$NEW" config check --config "$CFG" # MUST be clean — resolve every "ignored config key". # Zero api tokens + genesis vault IS clean: the # doctor shares the daemon's bootstrap verdict (#133) # 5. Load secrets INTO the vault. There is no offline bulk import — the daemon is the # sole writer, so secrets enter THROUGH it. With NO api token in config yet, the # staged binary boots the vault-operable / ductile-closed posture: it serves /vault/* # on a local unix socket WITHOUT opening the public gateway. See the credential ladder # in docs/adr/vault-credential-ladder.md. SOCK="$CFG/vault-admin.sock" # MUST match api.management_socket from step 4 — # without that key the daemon serves the socket # beside the state DB and this dial refuses (#141) ADMIN=... # the admin token printed by genesis in step 3 "$NEW" system start --config "$CFG" & # serves management-only; stop it (kill %1) when done "$NEW" system status --config "$CFG" # expect: boot_posture: management-only (live) # Mint the API bearer token (operator-chosen value; read from stdin, never argv): printf '%s' "$API_TOKEN" | "$NEW" vault set --api-url "unix://$SOCK" --token "$ADMIN" \ --name core-api-token --pattern manual # Migrate each existing tokens.yaml secret the same way — one vault set per secret: # printf '%s' "$VAL" | "$NEW" vault set --api-url "unix://$SOCK" --token "$ADMIN" --name --principal

kill %1 # stop the management-posture daemon # Reference the api token in config so the NEXT boot resolves it and ACTIVATES the gateway: # api.auth.tokens: [ { secret_ref: core-api-token, scopes: [ "*" ] } ] # 6. Seal BOTH: config files AND plugin attestation. Attestation is keyed by the vault # nonce, so genesis (step 3) must already be done. "$NEW" config lock --config "$CFG" "$NEW" plugin lock --all --config "$CFG" # prints a confirm code "$NEW" plugin lock --all --config "$CFG" # commit with the code # 7. Cutover: install the new binary and restart. systemctl --user stop ductile-local cp "$NEW" ~/.local/bin/ductile systemctl --user start ductile-local # 8. Verify. journalctl --user -u ductile-local -n 40 # expect "compose-time attestation on"; no integrity/admission failure curl -s localhost:8081/healthz # status ok; plugins_loaded == your pre-deploy baseline ductile system vault-audit --config "$CFG" # genesis + secret sets recorded ``` ### Spawn-hygiene check before cutover (do not skip) Plugins no longer inherit the gateway environment ([SECRETS.md §4](https://ductile.run/SECRETS/index.md)). Before cutover, for each enabled plugin work out how it gets each secret **today**: - delivered via `${VAR}` interpolation into its `config:` block → unaffected (it travels over stdin); - read from the tool's own config (e.g. `fabric` reads `~/.config/fabric/.env`) → unaffected; - read **directly from the process environment** → add that name to `service.plugin_env_passthrough`, or move the secret into the vault. Catching this here is what prevents a fleet of plugins silently failing on a stripped environment after cutover. ### Rollback Stop the service, restore the previous binary (`~/backups/ductile-prev`), and — only if config changed — restore `config.yaml` and `.checksums` from the backup. The additive `vault_audit` table and the inert `vault.age` + age key are harmless to the prior binary, so a DB restore is normally unnecessary. Confirm green on `healthz`. ### Why this order (theory) The sequence is forced by dependency, not preference: - **Genesis before `plugin lock`** — plugin fingerprints are keyed by the vault nonce that genesis seeds; you cannot attest plugins until the vault exists. - **`config lock` ≠ `plugin lock`** — deliberately decoupled, so sealing config does not seal attestation and vice-versa. - **Migration is observability, not a gate** — `vault_audit` is additive and not a required table; run it so the audit log is complete from the first op, but the boot never hinges on it. - **The admission gates are independent levers** — `validate_config_on_boot` is the strict decode (it makes silently-dropped config keys loud); `verify_integrity_on_boot` is the `.checksums` + attestation preflight (it makes tamper/drift loud). Turning them on is what converts those silent failure modes into refuse-to-boot. For the full mental model — principals, the sole-writer split, compose-time delivery, attestation, and the backup/key pairing — see [SECRETS.md](https://ductile.run/SECRETS/index.md). # Operator Guide This guide is intended for system administrators and LLM operators managing a Ductile instance. It covers day-to-day operations, monitoring, and administrative safety. ______________________________________________________________________ ## 1. System Operations ### Starting the Service The primary way to run Ductile is in the foreground: ```bash ./ductile system start ``` For production environments, we recommend using a **systemd** unit. See [Architecture](https://ductile.run/ARCHITECTURE/#14-deployment) for an example configuration. ### Reloading Configuration You can reload the configuration without restarting the service by sending a `SIGHUP` signal or using the CLI: ```bash ./ductile system reload ``` ### Backups `ductile system backup` writes a point-in-time snapshot to a single `tar.gz` archive. The DB snapshot uses SQLite `VACUUM INTO`, so the gateway can stay running. ```bash ductile system backup --to /backups/ductile-$(date -u +%Y%m%dT%H%M%SZ).tar.gz \ --scope config ``` Scope is a nested ladder; each level adds to the previous: - `db` — DB snapshot only - `config` (default) — `db` + ductile config dir - `plugins` — `config` + every directory under `plugin_roots` - `all` — `plugins` + every file under `environment_vars.include` Each archive embeds a `BACKUP_MANIFEST.txt` recording ductile version, commit, hostname, source paths, source DB sha256, included items, excluded items with reasons, plugin-root mappings, and any boundary warnings (e.g. `api.yaml` appearing at scope `config`, env files appearing at scope `all`). Inspect with `tar -xzOf BACKUP_MANIFEST.txt` without re-extracting the rest. At scope `config` or higher the archive includes the encrypted vault blob (`vault.age`) so a restore is not secret-less, but the age key that decrypts it is **deliberately excluded** — custody it out-of-band (e.g. a password manager) and restore needs both. The manifest records the key as excluded with this pairing note. Restore = unpack the archive, write the age key file back from custody (mode 0600), then start. See `docs/SECRETS.md` §3 "Backup and restore" for the full steps and the `rotate-key`-destroys-the-old-key discipline. The command refuses to overwrite an existing destination — operator owns the naming pattern and retention. For a scheduled-backup setup (systemd timer or launchd), see `docs/DEPLOYMENT.md` §10. ### Vault operations The vault holds the secrets the core delivers to plugins (the `ductile vault` command family; the full model is in `docs/SECRETS.md`). Writes split into two classes by whether they touch the age key: - **Local, key-touching — daemon STOPPED** (`init`, `import`, `rotate-key`): they hold the age key and rewrite the blob, so they take the daemon's PID lock and refuse while it is running. - **Keyless API clients — daemon RUNNING** (everything else): they POST to the daemon (the sole writer) authenticated by the **vault admin token** (`--token` or `DUCTILE_VAULT_TOKEN`), not the config API tokens. ```bash # Genesis (once, daemon down): seeds the core principal, the fingerprint nonce, # and a one-time admin token — printed once, store it; it is the API credential. ductile vault init --vault vault.age --key age.key # Lifecycle (daemon up, admin token in DUCTILE_VAULT_TOKEN, --api-url omitted for brevity): ductile vault register-principal --name mailer --kind plugin printf '%s' "$SMTP_PW" | ductile vault set --name smtp_pw --principal mailer ductile vault roll --name smtp_pw # supersede the value (manual: stdin; auto: minted) ductile vault revoke --name smtp_pw # terminal; clears the value ductile vault revoke-principal --name mailer # stop delivery (fail closed) ductile vault purge-principal --name mailer # remove + strip its grants ``` `--pattern manual` (default) takes the value from stdin; `--pattern auto` has the daemon mint it. A plugin must be `plugin lock`-ed before it receives any secret. Audit every change with **`ductile system vault-audit [--principal NAME]`**. Rotating the at-rest key is below. **When a roll takes effect (freshness):** a **plugin** secret picks up a `roll` at its **next spawn** (re-resolved per job). A **webhook/relay** secret_ref is resolved at config **load**, so rolling it only takes effect on the running servers after a **`ductile system reload`** — roll, then reload. ### Rotating the vault key `ductile vault rotate-key` rotates the daemon's age identity: it mints a fresh key, re-encrypts the vault to it, and retires the old key — so the blob at rest is readable only by the new key. It is a **local, key-touching** operation and the daemon must be **stopped** (it refuses while the daemon holds the PID lock). Stop the service, rotate, start: ```bash # Stop the daemon via its service manager (there is no `system stop` command): # systemd: systemctl --user stop ductile-local # launchd: launchctl bootout gui/$UID/

` or `$DUCTILE_CONFIG_DIR` | | State DB | `/ductile.db` | SQLite, WAL mode | | API port | `127.0.0.1:8081` | from `service.api_port` in `config.yaml` | | Service supervisor | none enforced | launchd, systemd, docker compose, supervisord | | Auth token | none by default | from `tokens.yaml`, surfaced via env var of your choice | **Action for the operator setting this up.** Build your own runtime-context table for the gateways you operate — instance name, binary path, config dir, DB path, port, service supervisor, auth token env var. Keep it next to your deployment docs, not in this handbook. ______________________________________________________________________ ## CLI command reference Pattern: `ductile [flags]`. ### System ```bash ductile system start # Start gateway (foreground) ductile system status [--json] # Health: PID, state DB, plugins ductile system reload # Hot-swap config in a running gateway (SIGHUP) ductile system watch # Real-time TUI monitor ductile system reset # Reset circuit breaker ductile system skills [--config ] # Export LLM skill manifest (Markdown) ductile system selfcheck [--json] # Read-only integrity invariants ductile system backup --to # Atomic snapshot (VACUUM INTO) ductile system doctor # Startup and runtime health checks ``` ### Config ```bash ductile config check [--json] [--strict] # Validate syntax, policy, integrity ductile config lock # Authorize state (update .checksums) ductile config show [entity] # Show resolved config or entity ductile config get # Dot-notation read ductile config set = # Modify (use --dry-run to preview) ductile config init # Initialize config directory ductile config backup / restore # Archive / restore configuration ductile config token / scope # Manage API tokens and scopes ductile config plugin / route / webhook # Manage routing artefacts ``` ### Job ```bash ductile job inspect [--json] # Lineage, baggage, artifacts ductile job logs [--json] # Query stored job logs # Filters: --plugin --command --status --submitted-by # --from --to (RFC3339) --query --limit --include-result ``` ### Plugin ```bash ductile plugin list [--api-url URL] [--json] # Discover loaded plugins ductile plugin run # Manual execution ductile plugin lock # Attest plugin bytes — REQUIRED before it can receive vault secrets # (decoupled from `config lock`; re-run after any manifest/entrypoint edit) ``` ### Vault & Secrets The vault delivers secrets to attested plugins. Two op classes (full model: `docs/SECRETS.md`): **Onboarding a plugin to vault secrets is three ordered steps:** (1) `vault register-principal --name

--kind plugin`, (2) `vault set --name --principal

` to grant each secret, (3) `plugin lock

` to attest its bytes. All three are required — an unattested plugin has no recorded keyed fingerprint, so compose-time identity verification fails and it receives **nothing**. ```bash # Local, key-touching — daemon STOPPED (hold the age key; PID-lock guarded): ductile vault init --vault vault.age --key age.key # genesis: core principal + nonce + one-time admin token ductile vault rotate-key --config

# rotate the at-rest age key ductile vault rotate-admin-token --config # mint a fresh admin token in place # Keyless API clients — daemon RUNNING (admin token: --token / DUCTILE_VAULT_TOKEN). # From scratch (no api token yet) the daemon boots the vault-operable / ductile-closed # posture and serves /vault/* on a local unix socket — target it with --api-url unix://; # mint the api token there, then add its secret_ref to config and reload (credential-ladder ADR). ductile vault register-principal --name

--kind plugin|consumer|gateway ductile vault set --name --principal

# value from stdin (--pattern manual|auto) ductile vault roll --name # supersede value ductile vault revoke --name # terminal (clears value) ductile vault revoke-principal --name

# stop delivery (fail closed) ductile vault purge-principal --name

# remove + strip grants ductile vault roll-principal --name

# roll every auto secret it holds ductile secrets keygen|encrypt|rotate # age-at-rest tooling (config bundles, NOT the vault) ductile system vault-audit [--principal

] [--json] # the append-only audit log (names + outcomes, never values) ``` ### API (direct gateway calls) ```bash ductile api /jobs ductile api /plugin/echo/poll -f message="hello" ductile api /pipeline/youtube-wisdom -f url="…" ductile api /system/reload -X POST ductile api /healthz # Flags: -X METHOD, -f key=value, -H Header:val, -b BODY, --api-url, --api-key ``` ### Top-level ```bash ductile skills # Export capability registry as LLM Markdown ductile version # Version + commit + build time ``` ______________________________________________________________________ ## Universal flags | Flag | Purpose | | ---------------- | ----------------------------------------------- | | `--json` | Machine-readable output (all read commands) | | `-v, --verbose` | Internal logic, path resolution, baggage merges | | `--dry-run` | Preview mutations without committing | | `--config

` | Override config directory | ______________________________________________________________________ ## The lock rituals (`config lock` ≠ `plugin lock`) Locking is two decoupled acts — pick by what changed: ```bash # Config FILE edit (config.yaml, api.yaml, pipelines, webhooks): ductile config check # validate ductile config lock # authorize the file state (updates .checksums) ductile system reload # apply without restart # Plugin manifest/entrypoint edit (re-attest the bytes): ductile plugin lock # config lock does NOT do this ductile system reload ``` `config lock` re-hashes config files and *preserves* plugin fingerprints; it does not re-attest plugin bytes. A plugin whose bytes changed without a re-`plugin lock` is refused its vault secrets at spawn (fail closed) — a routine `config lock` will not fix it. A forgotten lock (either one) is a recurring root cause in incident response. ### Config integrity (tiered) | Tier | Files | On mismatch | | ------------- | --------------------------------------------------- | ---------------------------- | | High Security | `tokens.yaml`, `webhooks.yaml`, `scopes/*.json` | Hard fail (refuses to start) | | Operational | `config.yaml`, `plugins/*.yaml`, `pipelines/*.yaml` | Warn & continue | ______________________________________________________________________ ## Entity addressing Use `:` syntax with `config show/get/set`: ```bash ductile config show plugin:withings ductile config show pipeline:video-wisdom ductile config set plugin:withings.enabled=false ductile config show plugin:* # list all plugins ``` ______________________________________________________________________ ## Selfcheck — six read-only invariants 1. `config_discovery` — config dir resolves 1. `config_load` — config parses 1. `pid_lock` — PID file matches a running process 1. `db_integrity` — `PRAGMA integrity_check` 1. `db_schema` — required tables/columns/indexes match embedded baseline 1. `queue_terminal_freshness` — no stale terminal-state `job_queue` rows past retention **WAL safety**: when the gateway holds the PID lock, checks 4-6 are *skipped* with `detail: "skipped: active gateway holds PID lock — quiesce before selfcheck"`. The skip is correct behaviour, not a bug. Real-green pattern: run selfcheck **offline** against the new binary BEFORE installing. Once installed and running, expect "skipped" on 4-6 — the proof of correctness is that the gateway started at all, because the schema validator runs at startup and refuses to open the DB on mismatch. ______________________________________________________________________ ## Backup — atomic point-in-time snapshot ```bash ductile system backup --to [--scope SCOPE] [--config PATH] ``` Scopes (nested ladder; each adds to the previous): - `db` — DB snapshot only (SQLite `VACUUM INTO`, safe under concurrent writers) - `config` (default) — `db` + ductile config dir - `plugins` — `config` + every dir under `plugin_roots` - `all` — `plugins` + every file under `environment_vars.include` Each archive embeds `BACKUP_MANIFEST.txt` with version, commit, hostname, source paths, SHA256 of source DB, included/excluded items + reasons. Refuses to overwrite an existing `--to` destination. Inspect a manifest without re-extracting: ```bash tar -xzOf .tar.gz BACKUP_MANIFEST.txt ``` ______________________________________________________________________ ## Migrations & schema `internal/storage/schema.sql` is embedded in the binary; the schema validator runs at startup and refuses to open a DB missing any required table, column, or index. Schema changes ship as Python scripts at `scripts/migrate-*.py`, idempotent by design, run with the service quiesced. Always backup before migration: ```bash sqlite3 "PRAGMA wal_checkpoint(TRUNCATE);" && cp ``` ______________________________________________________________________ ## LLM capability discovery (`system skills`) Ductile is designed for LLM operation. Get the current live manifest: ```bash ductile system skills --config # or set DUCTILE_CONFIG_DIR and run: ductile system skills ``` Outputs Markdown listing all plugin commands with endpoints, schemas, and semantic anchors (`mutates_state`, `idempotent`, `retry_safe`) plus all configured pipelines. See [`DUCTILE_SKILLS_SCHEMA_V1.md`](https://ductile.run/DUCTILE_SKILLS_SCHEMA_V1/index.md) for the contract that output obeys. ______________________________________________________________________ ## Common workflows ### Trigger a pipeline via API ```bash curl -X POST http://:/pipeline/ \ -H "Authorization: Bearer $DUCTILE_TOKEN" \ -H "Content-Type: application/json" \ -d '{"payload": {"key": "value"}}' ``` ### Trigger a plugin directly (bypasses routing) ```bash curl -X POST http://:/plugin//poll \ -H "Authorization: Bearer $DUCTILE_TOKEN" \ -d '{"payload": {}}' ``` ### Inspect a failed job (routine — no incident analysis needed) ```bash ductile job inspect -v --json ``` ### Check gateway health ```bash curl http://:/healthz ``` ______________________________________________________________________ ## Architecture summary (operator view) - **Governance hybrid**: control plane is SQLite `event_context` baggage; filesystem state is plugin-managed. The core does not provision per-job workspaces. - **Spawn-per-command**: each plugin invocation is a fresh process (polyglot: bash, python, go, any executable). - **At-least-once**: jobs survive crashes and are recovered on restart. - **Immutable audit**: `origin_*` baggage keys can never be overwritten by plugins. ______________________________________________________________________ ## Job statuses `queued` → `running` → `succeeded` / `failed` / `timed_out` / `dead` If you see `dead` or persistent `failed`, treat as an incident: hand off to root-cause analysis (the `ductile-rca` skill) rather than continuing routine operation. ______________________________________________________________________ ## When to load other skills | Companion skill | When to load it | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `ductile-plugin-developer` | The work requires touching a plugin's code, manifest, or pipeline composition — not just its config. | | `ductile-rca` | Symptoms are not understood. *Stuck, hanging, tripped, missing, wrong.* Routine `job inspect` for a known-good system does **not** need RCA. | | `surface-contract` | Docs and code have drifted; you need to audit and re-align them. | Full ductile incident lifecycle (`ductile-rca` + this handbook + `ductile-plugin-developer`) is real and worth keeping in mind. ______________________________________________________________________ ## Reference docs In the same docs site: - [Architecture](https://ductile.run/ARCHITECTURE/index.md) — the technical deep dive - [Deployment](https://ductile.run/DEPLOYMENT/index.md) — host-local deployment + backup patterns - [Operator Guide](https://ductile.run/OPERATOR_GUIDE/index.md) — day-to-day commands with examples - [Health Check](https://ductile.run/HEALTH_CHECK/index.md) — invariants checked by `selfcheck` - [SQL Tightening Log](https://ductile.run/SQL_TIGHTENING_LOG/index.md) — schema-change audit trail - [Reload RCA](https://ductile.run/reload_rca/index.md) — canonical worked example of the reload deadlock RCA pattern In the repo: - [`AGENTS.md`](https://github.com/mattjoyce/ductile/blob/main/AGENTS.md) — the contributor contract; the design grounding behind these commands - [`CONSTITUTION.md`](https://github.com/mattjoyce/ductile/blob/main/CONSTITUTION.md) — the five pillars; this handbook is Pillar 1 (Run)