Artifact
The Artifact System is part of the core execution mechanism of the Horus Runtime SDK.
What Is an Artifact?
An artifact represents a concrete unit of data produced or consumed by a task.
Every artifact is backed by a filesystem path. Regardless of what the artifact
logically represents (a Python dict, a trained model, a dataset) the runtime always
materializes it on disk as either a file or a directory. This is a fundamental
invariant: the path field is not optional metadata, it is the canonical identity of
the artifact: existence means the path exists on the target where the artifact
lives (checked through the ArtifactStore, not by the artifact
itself).
This constraint makes artifacts deterministic, cacheable, and transportable across machines without any special serialization protocol at the workflow level.
Examples of what artifacts can represent (and how they materialize):
- Local files and folders (stored as-is)
- Python dicts, lists, and other JSON-compatible objects (serialized to a
.jsonfile) - Arbitrary Python objects (serialized to a
.pklfile via pickle) - Datasets and model checkpoints (stored as files or directories)
- Remote objects such as S3 or HTTP resources (downloaded and cached locally before use)
Each artifact:
- Has a stable ID. The
ididentifies the artifact within its task and is the handle that workflow edges wire together (a producer output to a consumer input). Tasks are not linked by matchingids — an explicit edge does the wiring — so a producer output and the consumer input it feeds may have differentids. Within a single task, input and outputids must be unique (see DAG planning and edges). - Has a path identifying its location on disk
- Carries optional
labels: adict[str, str]of free-form metadata (labels: {subject: batch_017, role: measurement}). The runtime never interprets them and they play no part in caching; they exist for whatever reads the workflow afterwards, such as a UI grouping nodes or a lineage query. - Defines:
- How to read its contents back into a Python object
- How to write a Python object to its file representation
- How to package / unpackage itself for transport, as portable shell
commands (see
pack_command/unpack_commandbelow)
Filesystem lifecycle operations (existence checks, deletion, packaging) are
not performed by the artifact itself. They run through the
ArtifactStore against the target where the artifact
physically lives, so they behave identically whether that is the orchestrator
or a remote machine.
Defining Custom Artifacts
Artifacts inherit from BaseArtifact and are automatically registered using the
runtime registry mechanism.
from horus_runtime.core.artifact.base import BaseArtifact
class MyArtifact(BaseArtifact[str]):
kind = "my_artifact"
kind_name = "My Artifact"
kind_description = "A short, human-readable description of this kind."
def read(self) -> str:
with open(self.path) as f:
return f.read()
def write(self, value: str) -> None:
with open(self.path, "w") as f:
f.write(value)BaseArtifact API and Required Implementations
BaseArtifact[T] is a generic, file-backed artifact abstraction:
-
Generic type:
BaseArtifact[T]specifies the native Python type thatread()returns andwrite()accepts. -
Path normalization: Accepts both
strandPathforpath, always resolved to an absolutePath. -
ID logic: Each artifact has a unique
internal_id(UUID) and a user-friendlyid(auto-set if not provided). -
Event emission: Emits lifecycle events via the runtime event bus.
-
Required implementations:
read() -> T: Read and deserialize the artifact contents. Runs task-local, on the machine where the value is produced or consumed.write(value: T) -> None: Write the native representation to disk (also task-local).kind: str: Concrete discriminator value used for registry dispatch and type resolution.
-
Kind metadata (optional ClassVars):
kind_name: ClassVar[str]: short, human-readable name for the kind (e.g."File"), used by client UIs, registries, and logging.kind_description: ClassVar[str]: a longer description of the kind. Defaults to an empty string. For translatable text, prefer a plugin-scoped translator (see the SDK i18n guide).
-
Materialization hook (optional):
materialize() -> None: make the artifact's bytes available atpath. The default is a no-op — a file-backed artifact assumes a producer task, or an upload, already wrote it. Value artifacts override it to write their inlinevalue, which is what lets a workflow-owned root artifact run with no producer at all.
-
Transport hooks (override only when the artifact is not a single file):
pack_command(src, pkg) -> str | None: return a portable shell command that produces the single-file packagepkgfrom the artifact materialized atsrc, orNonewhen the artifact is already a single file (identity packaging — the base default).unpack_command(pkg, dest) -> str | None: return a portable shell command that materializes the artifact atdestfrom the packagepkg, orNonefor identity (the store simply moves the file into place).
Both commands are executed on the target where the artifact lives (or is being materialized) by the
ArtifactStore, so they must be POSIX-portable and reference only the given target-side paths.FileArtifactand other single-file artifacts inherit the identity defaults;FolderArtifactoverrides them withtarcommands.
ArtifactStore
An artifact is just a file-backed value description; where it lives, and how
to check, delete, or transport it there, is owned by a target. The
ArtifactStore (horus_runtime.core.artifact.store) is the mediator that binds
an artifact to a target and performs its lifecycle operations through that
target's filesystem primitives:
from horus_runtime.core.artifact.store import ArtifactStore
store = ArtifactStore(target)
await store.exists(artifact) # -> bool
await store.delete(artifact) # remove + emit delete event
package_path = await store.package(artifact) # build a single transferable file
await store.unpackage(artifact, package_path)Because every operation runs through the target rather than local pathlib
calls, the same code checks and moves an artifact whether it physically lives
on the orchestrator or on a remote (e.g. SSH) machine. package() builds the
package on the target and returns its path (or the artifact's own path
unchanged for single-file, identity artifacts); unpackage() materializes it on
the destination, or moves the file into place with the target's move_file for
identity artifacts. An artifact whose pack_format is "tar.gz" (such as
FolderArtifact) is packed and unpacked with the target's pack_folder /
unpack_folder; any other artifact runs its pack_command / unpack_command.
digest() calls the target's path_digest. A FolderArtifact subclass that
overrides pack_command / unpack_command sets pack_format = None to keep
them.
This package() is unrelated to the horus package CLI command, which zips a
whole workflow for transport — see
Packaging workflows.
ArtifactStore depends only on a small set of methods
(path_on_target, path_exists, remove, path_digest, pack_folder,
unpack_folder, move_file, run_command_sync, resolved_working_directory) that
BaseTarget satisfies. See
Target for those primitives.
Built-in Artifacts
The SDK provides the following artifact implementations:
FileArtifact
Reads and writes raw bytes. Suitable for any opaque file.
from horus_builtin.artifact.file import FileArtifactFolderArtifact
Represents a directory. It overrides the transport hooks so a whole directory
can move between targets as a single file: its pack_format is "tar.gz", so
the target's pack_folder archives the folder's contents with
tar czf … -C <src> . and unpack_folder extracts them with tar xzf into a
freshly recreated destination.
from horus_builtin.artifact.folder import FolderArtifactJSONArtifact[T]
Serializes any JSON-compatible Python object (dict, list, str, etc.) to a .json file.
The generic parameter T is used for type-checking only; no runtime validation is performed.
from horus_builtin.artifact.json import JSONArtifact
artifact = JSONArtifact[dict](path="/tmp/result.json")
artifact.write({"key": "value"})
data = artifact.read() # dict, cast to TPickleArtifact[T]
Serializes arbitrary Python objects using the pickle protocol.
from horus_builtin.artifact.pickle import PickleArtifact
artifact = PickleArtifact[list](path="/tmp/data.pkl")
artifact.write([1, 2, 3])
data = artifact.read() # listWarning: pickle is not secure against malformed or maliciously crafted data. Never unpickle data from untrusted sources.
Value artifacts
Most artifacts describe data that some task will produce. A value artifact describes data you author directly in the workflow — a sample count, a threshold, a feature flag — while still being a normal, file-backed artifact in every other respect.
ValueArtifact[T] adds one field to BaseArtifact[T]:
value: T— the artifact's authored value. It round-trips throughmodel_dump, so a workflow document can carry a value whose bytes have not been written yet.
and overrides materialize() to write that value to path. The authored value
is the source of truth: a file left by an earlier run with another value is
rewritten, so editing only a value re-runs its consumer. A file that already
holds the value is left untouched. Only a root is materialized, so a value a
producer task wrote is never overwritten.
The graph does not distinguish the two cases. A NumberArtifact produced by a
task is transferred to its consumer exactly like a NumberArtifact root whose
value you typed. The only runtime difference is that a root value artifact has
no producer, so transfer_artifacts calls materialize()
on it whenever a consumer needs it — and copies the root's value onto the
consumer's own input artifact, so that substitution sees the authored value
rather than the field default. When the consumer is of the same kind, the fields that
qualify the value (a boolean's on / off mapping) are copied along with it.
NumberArtifact
An int | float, materialized as a JSON number. Defaults to 0.
from horus_builtin.artifact.number import NumberArtifact
samples = NumberArtifact(id="samples", path="samples.json", value=10)
samples.materialize()
samples.read() # 10Optional limits: min and max are inclusive, and step counts from min
(or 0), with a small float tolerance so 0.3 fits a 0.1 step. step: 1
with an integer min allows only integers. An unset value starts at min.
exclude lists values that are never allowed, even inside the range.
- {id: temp, kind: number, path: temp.json, value: 300, min: 250, max: 400, step: 5}
- {id: offset, kind: number, path: offset.json, value: 1, exclude: [0]}BooleanArtifact
A bool, materialized as a JSON boolean (true / false). Defaults to False.
from horus_builtin.artifact.boolean import BooleanArtifact
verbose = BooleanArtifact(id="verbose", path="verbose.json", value=True)Mapping the states to other values
A boolean can hand its task a value of your choosing for each state instead of
true / false. Set both on and off to an Option ({value, label}, from
horus_builtin.artifact.options): value is what the task receives, label is
only what a UI shows (it falls back to the value). A "Use GPU" toggle:
artifacts:
- id: device
kind: boolean
path: roots/device.json
value: true # the state; stored as "gpu"
on: {value: gpu, label: GPU}
off: {value: cpu, label: CPU}
tasks:
- id: train
runtime:
kind: command
command: python train.py --device $device # --device gpuWith a mapping, value holds the value the task receives ("gpu" / "cpu"),
never the toggle state; the toggle is on when value == on.value. A bare
true / false given as value, passed to write(), or left on disk by a
producer that wrote the state is translated to its mapped value, so $device,
the materialized file (JSON, "gpu") and read() always agree.
from horus_builtin.artifact.boolean import BooleanArtifact
from horus_builtin.artifact.options import Option
device = BooleanArtifact(
id="device",
path="device.json",
value=True,
on=Option(value="gpu", label="GPU"),
off=Option(value="cpu", label="CPU"),
)
device.value # "gpu"
device.write(False)
device.read() # "cpu"Validation rejects setting only one of on / off, two equal values, a value
that is neither of them, and mapped values that are themselves true /
false (those already name the state). Without on / off, a boolean behaves
exactly as above.
StringArtifact
A str, materialized as plain text — the string as authored, with no quotes.
Defaults to "".
from horus_builtin.artifact.string import StringArtifact
label = StringArtifact(id="label", path="label.txt", value="baseline")Optional limits: pattern is a Python regex that must match the whole value
(single-quote it in YAML so backslashes survive), min_length and
max_length count characters, exclude lists values that are never allowed
(a placeholder like changeme), and multiline only switches the editor to a
text area.
- {id: pdb, kind: string, path: pdb.txt, value: 1abc, pattern: '[0-9][a-z0-9]{3}'}
- {id: notes, kind: string, path: notes.txt, value: "", max_length: 500, multiline: true}
- {id: token, kind: string, path: token.txt, value: abc123, min_length: 6, exclude: [changeme]}ChoiceArtifact
A value picked from a fixed list of options, each a {value, label} pair:
value is what the task receives, label is only what a UI shows (it falls
back to the value). value defaults to the first option. Strings are
materialized as plain text, like StringArtifact; other scalars as JSON.
from horus_builtin.artifact.choice import ChoiceArtifact
device = ChoiceArtifact(
id="device",
path="device.txt",
options=[{"value": "cpu", "label": "CPU"}, {"value": "gpu", "label": "GPU"}],
value="gpu",
)or in a workflow YAML:
- id: device
kind: choice
options:
- {value: cpu, label: CPU}
- {value: gpu, label: GPU}
value: gpuMultiChoiceArtifact
Any subset of options (same {value, label} rules as ChoiceArtifact).
The value is a list kept in option order, empty by default, materialized as a
JSON array. Reads and writes check every item, as for ChoiceArtifact.
min_picks and max_picks bound how many options are picked.
- id: elements
kind: multichoice
path: elements.json
options: [{value: H}, {value: C}, {value: O, label: Oxygen}]
value: [H, O]
max_picks: 2ListArtifact
A free list of strings or numbers, empty by default, materialized as a JSON
array. Optional limits: min_items and max_items bound its length, unique
rejects a repeated item, min and max bound every item (items must then be
numbers), and exclude lists items that are never allowed.
- {id: seeds, kind: list, path: seeds.json, value: [1, 2, 3]}
- {id: temps, kind: list, path: temps.json, value: [280, 300], min_items: 1, unique: true, min: 250, max: 400}Limits on value artifacts (the ones above, and the options of choice /
multichoice) are checked when the value is loaded
or authored. Values a producer task writes are not checked against them,
except for choice and multichoice. Quote option values YAML would read as
booleans ("yes", "on").
A command run --device $device renders run --device gpu — the value, never
the label. Validation rejects an empty options list, duplicate values
(compared by their rendered text, so true and "true" collide), and a
value that is not one of the options. write() and read() enforce the
same, so a producer task cannot emit an undeclared value.
ValueArtifact itself is abstract and is not registered as a kind; only
number, boolean, string and choice are. Subclass it to add your own — declare the
concrete value type and the on-disk serialization through read/write,
exactly as for any other artifact, and materialize() follows for free.
Value artifacts also change how $id renders in a command — see
Artifact substitution.
Registering custom artifacts
To register and discover artifact plugins within the Horus runtime use the following entry point:
[project.entry-points."horus.artifact"]For more details, refer to the AutoRegistry documentation.