Handling secrets
How Secret fields keep credentials out of exported YAML and packaged zips, and how to resolve them at run time.
A plugin target can hold a credential in a plain field, SSHTarget.password
for instance. Left as a plain str, that value travels through every export
path unmarked: to_yaml, horus package, and anything downstream that reads
the dumped workflow dict would carry it verbatim.
horus_runtime.secrets marks such fields and redacts them wherever a workflow
is exported.
Marking a field
from horus_runtime.secrets import Secret
class SSHTarget(BaseTarget):
password: Secret | None = NoneSecret is a SecretStr subclass, so it keeps pydantic's masked repr/str/
JSON dump and format: "password" / writeOnly in the generated JSON schema.
A value built from a ${secret:<ref>} string is a reference: .ref is set,
and .resolve() looks the value up. Any other value is a literal: .ref is None, and .resolve() returns it as-is.
What gets redacted, and where
BaseWorkflow.to_yamlwalks the model withiter_secret_fields, thenredacts the dumped dict before writing. Each secret field becomes${secret:<ref>}, never the pydantic mask. A mask round-trips back in as a literal password on re-import, so this is why redaction targets a reference instead.packaging.package_workflowdoes the same for the workflow YAML it writes into the zip, sohorus package(see Packaging workflows) never ships a literal credential either.
iter_secret_refs(data) is the dict-side twin of iter_secret_fields: it walks
an already-dumped workflow dict (no models) and yields (path, ref) for every
${secret:<ref>} it finds. Use it wherever you only have the dumped dict, not
the validated model: a workflow stored as an opaque blob by whatever is
persisting it, for example.
Resolving a secret at run time
Secret.resolve() looks up the real value lazily:
HORUS_SECRET_<REF>(ref uppercased, non-alphanumerics replaced with_; seeenv_key_for_ref).- Failing that, the file named by
$HORUS_SECRETS_FILE: YAML, keyed by ref.
Neither is set: resolve() raises SecretResolutionError, naming the env var
and file it checked.
export HORUS_SECRET_SSH_PASSWORD=hunter2
# or
export HORUS_SECRETS_FILE=./secrets.yaml # { ssh_password: hunter2 }Secrets in a task's environment
A task can declare credentials its command needs as environment variables:
tasks:
- id: fetch_model
kind: horus_task
runtime:
kind: command
command: huggingface-cli download meta-llama/Llama-3.1-8B
secrets:
HF_TOKEN: "${secret:hf_token}"An entry can also carry a display name and a description. The key stays the env var name:
secrets:
HF_TOKEN:
value: "${secret:hf_token}"
name: Hugging Face token
description: Read access to gated model weightsEach entry is a TaskSecret whose value is a Secret, so it is redacted on
export like any other secret field (tasks.<N>.secrets.<NAME>.value). When the task
starts, every entry is resolved as described above; a missing one fails the
task before its command runs. The resolved values are set on every command the
task issues through BaseTarget.run_command, whichever executor and target run
it, and the caller's own env wins on a name clash. They are not set on the
next task.
Names
A name must match [A-Za-z_][A-Za-z0-9_]* and be at most 128 characters.
Names starting with HORUS_ are reserved, and so are PATH, HOME, USER,
LOGNAME, TMPDIR, TC_CACHE_DIR, LD_LIBRARY_PATH, LD_PRELOAD,
LD_AUDIT, IFS, BASH_ENV, ENV, CDPATH, SHELLOPTS, BASHOPTS and
PS4. A task with any other name fails validation, e.g.
Secret 'PATH': PATH is reserved. horus_runtime.secrets.env_name_error(name)
returns the same message, or None for a valid name.
Docker
The Docker executor passes each name as -e NAME, so the value is not on the
docker run command line. It is still in the container config, visible with
docker inspect while the container exists.
Detached local jobs
LocalTarget hands a detached job its environment through an env file in
the job directory. The file has mode 0600, and the job reads it and then
deletes it, so the values never appear on the job's command line.
Non-goals
Key management/rotation, encrypting the local secrets file, and per-run credential scoping are out of scope for this mechanism.
Follow-up
Lifting stored credentials out of an opaque workflow_data blob into a real
credentials store, with re-injection at run dispatch, is a separate,
not-yet-built piece. This mechanism only covers marking, redaction, and
env/file resolution.