Ductile: Configuration Specification¶
Version: 1.1 (Tiered Directory Model)
Date: 2026-02-25
Status: Approved
This document defines the configuration structure, integrity verification, and runtime compilation behavior for Ductile.
1. Directory Structure¶
Ductile uses a configuration directory, typically located at ~/.config/ductile/. Only
config.yaml is implicitly loaded; all other files must be referenced via include:.
~/.config/ductile/
├── config.yaml # [Operational] Service-level settings
├── webhooks.yaml # [High Security] Webhook endpoints & secrets (include explicitly)
├── tokens.yaml # [High Security] API token registry (include explicitly)
├── relay-instances.yaml # [Operational] Outbound named relay targets (include explicitly)
├── relay-ingress.yaml # [Operational] Inbound trusted relay peers (include explicitly)
├── routes.yaml # [Operational] Global routing rules (include explicitly)
└── scopes/ # [High Security] Token scope definitions
├── admin-cli.json
└── github-integration.json
2. Tiered Integrity Preflight¶
Before starting, the system verifies all files against a monolithic .checksums manifest located in the configuration root. Integrity is enforced in two tiers:
| Tier | Files | Missing/Mismatch Behavior |
|---|---|---|
| High Security | tokens.yaml, webhooks.yaml, scopes/*.json |
Hard Fail: System refuses to start (EX_CONFIG). |
| Operational | config.yaml, routes.yaml, relay-instances.yaml, relay-ingress.yaml |
Warn & Continue: Logs a warning but loads the file (Unless service.admission.fail_on_drift: true is set, in which case it is a Hard Fail). |
2.1 The Seal (.checksums)¶
The .checksums file is a YAML manifest containing SHA-256 hashes indexed by the absolute path of every authorized file.
- System Lock-in: Moving the configuration directory breaks the seal.
- Authorization: The ductile config lock command is the only way to update the manifest.
3. Monolithic Compilation (Grafting)¶
At runtime, the gateway compiles all discovered files into a single, monolithic configuration object.
3.1 Merge Logic¶
- Root First:
config.yamlis loaded first as the base. - Explicit Includes: Additional files are loaded from the
include:list (and any directories listed there) in order. - Precedence: Later entries override earlier ones (n-1 branching).
- Matching Branches:
- Maps (e.g.,
plugins:): Keys are merged. Duplicate keys are overridden by the later file. - Arrays (e.g.,
pipelines:,routes:): Items are appended to the list. - Scalars: Later values replace earlier ones.
- Maps (e.g.,
3.2 Modular Example¶
config.yaml (Root)
pipelines.yaml
Resulting Monolith:
3.3 Directory includes¶
include: entries may point at directories. Ductile loads *.yaml files
from that directory (non-recursive) in alphabetical order and merges them
as if they were listed explicitly.
3.4 Naming convention for operator-facing instance identifiers¶
When config introduces an operator-facing identifier for a Ductile instance, peer, or similarly named runtime endpoint, use lower-case hyphenated names:
home-primarylabvps-backup
Do not use:
- underscores:
home_primary - spaces:
home primary - mixed case:
HomePrimary
Recommended pattern:
Rationale:
- reads cleanly in YAML and logs
- maps directly to URL path segments
- avoids competing conventions for operator-facing identities
- keeps names distinct from Go identifiers and internal field names
service.name is an operator-facing identity field and should follow
this convention when it names a concrete Ductile instance rather than a
generic service label.
4. File Formats¶
4.1 config.yaml (Service settings)¶
service:
name: ductile
tick_interval: 60s
log_level: info
log_format: json
dedupe_ttl: 24h
job_log_retention: 30d
job_queue_retention: 24h
# DB table history retention
job_transitions_retention: 30d # historical job state transitions
job_attempts_retention: 30d # job execution attempt logs
breaker_transitions_retention: 90d # circuit breaker state changes
# Concurrency & limits
# Omit to use the default: max(1, CPU-1). Set to 1 to force global serial dispatch.
max_workers: 4
hook_max_depth: 4 # max on-hook lifecycle chain depth, prevents loops (default: 4)
# Admission control: four independent gates the daemon applies at boot/reload.
# Each defaults to false (permissive). Enable only what you need.
admission:
verify_integrity_on_boot: true # run .checksums + fingerprint preflight at startup
fail_on_drift: true # operational config/routes drift -> reject (boot & reload)
validate_config_on_boot: true # require config validation to pass at startup
require_api_auth: true # reject an enabled API with no auth tokens
# strict_mode: true # DEPRECATED alias — enables all four admission gates above
allow_symlinks: false # permit resolving symbolic links under config/plugin roots
# Spawn-hygiene allowlist: extra env var NAMES passed through to plugin child
# processes, on top of the built-in minimal set (PATH, HOME, TZ, LANG, ...).
# Secrets do NOT go here — they reach plugins via the vault `secrets` envelope
# over stdin, never the environment.
plugin_env_passthrough: [MY_PLUGIN_FLAG]
plugin_roots:
- /opt/ductile/plugins
- /opt/ductile/plugins-private
api:
enabled: true
listen: 127.0.0.1:8080
management_socket: /tmp/ductile-admin.sock # Unix socket serving /vault/* in the
# management posture (BOOTSTRAP.md). Keep it
# under ~104 bytes. Default when omitted:
# vault-admin.sock beside the state DB.
allowed_origins: # List of allowed CORS origins (empty by default)
- http://localhost:3000
- https://my-dashboard.example
max_concurrent_sync: 10 # Simultaneous blocking synchronous request semaphore
max_sync_timeout: 5m # Hard cap on synchronous request timeout duration
state:
path: ./data/state.db
# Encryption at rest + the owned secret vault (see docs/SECRETS.md).
secrets:
age_key_file: ./age.key # age identity that decrypts encrypted config and the vault
vault_file: ./vault.age # the encrypted vault blob (default: <configDir>/vault.age)
# macOS-only. Each path is stat()-ed once on cold start (after PID lock,
# before "ductile running" log). Triggers any pending TCC popup for the
# Files-and-Folders service that gates the path. Runs synchronously while
# the operator is at the keyboard for the deploy. No-op on non-darwin and
# when the list is empty. Skipped on SIGHUP reload (binary cdhash
# unchanged → existing grants still valid).
#
# Configure local-volume paths only. An unreachable network mount blocks
# os.Stat for the filesystem-level timeout (seconds to minutes) and
# delays gateway readiness during the cold-start prewarm.
tcc_paths:
- /Users/me/Documents/Obsidian # triggers Documents grant
- /Volumes/Projects # triggers NetworkVolumes grant
4.1.1 Config Key Clarifications¶
- Database/State Alias: To support operator intuition,
database:can be used as an interchangeable alias forstate:. Both point to the SQLite DB: - Path Resolution: Relative paths (like
./data/state.db) are resolved against the directory containingconfig.yaml. A leading~or~/expands to the user's home directory in all path fields (state.path,plugin_roots,secrets.*,api.management_socket,tcc_paths,environment_vars.include); the~userform is not supported. - Deduplication Durations:
dedupe_ttlmatches terminal rows injob_queue, sojob_queue_retentionmust be at least as long asdedupe_ttl. The defaults are both 24h.
Note: the core does not provision per-job filesystem workspaces; the
workspace:config section has been removed. Plugins that need a scratch path manage it themselves — seedocs/PLUGIN_DEVELOPMENT.md§9.
service.plugin_env_passthrough is a list of env var names (not values) granted to plugin child processes on top of the built-in spawn-hygiene allowlist (PATH, HOME, TZ, LANG, …). Use it sparingly. Secrets must not travel this way — they are withheld from the plugin environment and delivered only through the vault secrets envelope over stdin (docs/ARCHITECTURE.md §5.5/§6.1).
secrets.age_key_file names the age identity (private key, mode 0600) used to decrypt encrypted config includes and the vault blob. Resolution order: DUCTILE_AGE_KEY_FILE env var → secrets.age_key_file (relative to configDir) → built-in default locations. secrets.vault_file names the encrypted vault blob; relative to configDir, defaulting to <configDir>/vault.age. An absent vault file means "no vault yet" (the migration/coexistence window) — not an error. The vault holds the owned secret store; see docs/SECRETS.md for the principal/secret model and the ductile vault lifecycle.
plugin_roots is the multi-root setting.
Discovery behavior: - Duplicate roots are ignored after first occurrence. - Roots are scanned in order; if duplicate plugin names exist across roots, the first discovered plugin is kept and later duplicates are ignored.
4.2 Plugin definitions (included file)¶
plugins:
echo:
enabled: true
parallelism: 1
notify_on_complete: true # Opt-in to job.completed lifecycle signals
schedules: # Optional; omit for event-driven plugins
- id: default
every: 5m
config:
message: "Hello"
4.2.1 Concurrency controls¶
service.max_workers: Global worker cap across all plugins. If omitted, Ductile usesmax(1, CPU-1). Set this to1to force whole-system serial dispatch.plugins.<name>.parallelism: Per-plugin concurrency cap.- Constraint:
1 <= parallelism <= max_workers.
Manifest interaction:
- Plugins may declare concurrency_safe: false in manifest.yaml; omitted
means true.
- The manifest hint is the plugin author's safety declaration. Operators use
plugins.<name>.parallelism to choose how much same-plugin concurrency to
allow within the global service.max_workers cap.
4.2.2 Inspecting effective plugin defaults¶
Per-plugin retry, timeouts, circuit_breaker, parallelism, and
max_outstanding_polls defaults live in code (single-sourced), and unset fields
resolve to those defaults at runtime — so a plugin stanza that omits them (or sets
only part of a block) does not show the values actually in force. Use the
effective view to see what runs, with each value tagged explicit (set in your
file) or default (inherited from code):
$ ductile config show --effective birda
plugin: birda
enabled: true (explicit)
retry.max_attempts: 4 (default)
retry.backoff_base: 30s (default)
timeouts.poll: 1m0s (default)
timeouts.handle: 5m0s (explicit)
...
parallelism: 8 (default) # inherits service.max_workers
ductile config get --effective birda.timeouts.handle resolves a single field
(5m0s (explicit)); add --json for { "value": ..., "source": ... }. Plain
config show (no --effective) is unchanged and renders the config as written.
4.3 webhooks.yaml (High Security)¶
Webhook definitions in standalone webhooks.yaml can be authored in two formats:
Option A: Documented Nested Form (Recommended)
webhooks:
endpoints:
- name: github
path: /webhook/github
plugin: github-handler
secret_ref: github_webhook_secret
signature_header: X-Hub-Signature-256
max_body_size: 1MB
Option B: Legacy Flat Form (Preserved for compatibility)
webhooks:
- name: github
path: /webhook/github
plugin: github-handler
secret_ref: github_webhook_secret
signature_header: X-Hub-Signature-256
max_body_size: 1MB
See WEBHOOKS.md for full configuration details, include-mode caveats, and signing examples.
4.4 tokens.yaml (High Security)¶
tokens:
- name: admin-cli
key: ${ADMIN_API_KEY}
scopes_file: scopes/admin-cli.json
scopes_hash: sha256:a3f8c2d9...
4.5 routes.yaml (Operational - Experimental)¶
[!IMPORTANT]
Global routing rules viaroutes.yamlare experimental. Most users should prefer thepipelinesDSL for orchestration.
4.6 relay-instances.yaml (Operational - Experimental)¶
relay-instances.yaml defines named outbound Remote Event Relay targets.
instances:
- name: lab
enabled: true
base_url: https://lab.example
ingress_path: /ingest/peer/home-primary
secret_ref: relay-lab-v1
key_id: v1
timeout: 10s
allow:
- backup.ready
- report.generated
Notes:
- name is the stable operator-facing alias used by sender-side config.
- base_url must be an absolute http or https URL.
- ingress_path is the receiver path that accepts the trusted relay request.
- secret_ref points at a tokens.yaml entry used as the shared HMAC secret.
- allow is an optional sender-side event-type allowlist.
4.7 relay-ingress.yaml (Operational - Experimental)¶
relay-ingress.yaml defines inbound trusted peers and the local acceptance policy for Remote Event Relay.
remote_ingress:
listen_path: /ingest/peer
max_body_size: 1MB
allowed_clock_skew: 5m
require_key_id: true
peers:
- name: home-primary
enabled: true
secret_ref: relay-lab-v1
key_id: v1
accept:
- backup.ready
baggage:
allow:
- trace_id
- requested_by
Notes:
- listen_path is the trusted relay ingress root mounted on Ductile's HTTP server.
- Relay ingress listens on api.listen; it does not introduce a separate listener address in Phase 1.
- allowed_clock_skew controls timestamp validation for replay-window hardening.
- require_key_id requires X-Ductile-Key-Id on inbound requests.
- peers[].accept is an optional receiver-side event-type allowlist.
- peers[].baggage.allow is a local policy for which remote baggage keys may seed new local root context.
Accepted relay requests are treated as fresh local root ingress events:
- the receiver performs normal local enqueue
- the receiver performs normal local exact-match routing
- no cross-instance event_context lineage is created
See REMOTE_EVENT_RELAY.md for a user-level guide and an end-to-end example.
5. Authentication Configuration¶
Ductile authentication is configured within the api section of the configuration (typically in config.yaml or a dedicated auth.yaml).
5.1 Scoped Tokens¶
For multi-user or production environments.
api:
auth:
tokens:
- token: admin_token
scopes: ["*"]
- token: readonly_token
scopes: ["plugin:ro", "jobs:ro", "events:ro"]
- token: operator_token
scopes: ["plugin:rw", "jobs:rw", "events:ro"]
5.2 Token Scopes¶
A scope is a role. Each API route asks the policy for an action (read or write) on a
capability (jobs/status/, plugin/invoke/, …); the capability table in
internal/auth/policy.go says which scope reaches it, and wider scopes reach it by implication.
The policy is Shearline over the
tarski engine; a refused request's 403 carries why
(see API_REFERENCE § Error Codes).
| Scope | Implies | Reaches |
|---|---|---|
* |
plugin:rw, jobs:rw, events:rw, system:rw |
everything |
plugin:rw |
plugin:ro, plugin:invoke:ro |
write-class plugin commands, pipeline triggers |
plugin:ro |
plugin:catalog:ro |
catalog reads only (P2-07) |
plugin:catalog:ro |
GET /plugin/{plugin} |
|
plugin:invoke:ro |
read-class POST /plugin/{plugin}/{command} |
|
jobs:rw |
jobs:ro |
(no write routes yet; the full jobs tree) |
jobs:ro |
jobs:status:ro, jobs:result:ro, jobs:logs:ro, jobs:tree:ro |
back-compat super-scope (D1) |
jobs:status:ro |
/job/{id}, /jobs, /scheduler/jobs, /analytics/*, /stopwatch/{plugin} |
|
jobs:result:ro |
result, error and stderr fields of a job response | |
jobs:logs:ro |
/job-logs |
|
jobs:tree:ro |
/job/{id}/tree |
|
events:rw |
events:ro |
|
events:ro |
events:meta:ro, events:payload:ro |
GET /events |
events:meta:ro, events:payload:ro |
reserved for per-subscriber shaping; grant nothing yet | |
system:rw |
system:ro |
POST /system/reload |
system:ro |
/topology, /config/view, /system/doctor, /system/selfcheck |
A token whose scope list names anything else fails config check and refuses to boot, with the
known scopes in the error. An empty scope list authenticates but reaches only the scope-free
discovery routes.
config check also reports the token policy's cost: every token is decided against every
capability once, and the costliest decision is printed against the policy engine's bounds
(token policy: worst decision ...). A policy whose costliest decision uses more than half of
either bound is refused at config check and at boot.
6. Environment Interpolation¶
Interpolation of ${VAR} syntax happens after integrity verification but before YAML parsing.
- Secrets must never be stored in YAML files; use environment variables.
- Interpolation is forbidden in file paths (e.g., include: or directory walking) to ensure a static, verifiable tree.
6.1 Environment file includes¶
You can preload env vars from .env files before interpolation:
Notes: - Paths are resolved relative to the file declaring the include. - Existing process environment variables are not overridden.