Temple Compute Docs
Core

Artifact

The Artifact System is part of the core execution mechanism of the Horus Runtime SDK.

What Is an Artifact?

An artifact represents a concrete unit of data produced or consumed by a task.

Every artifact is backed by a filesystem path. Regardless of what the artifact logically represents (a Python dict, a trained model, a dataset) the runtime always materializes it on disk as either a file or a directory. This is a fundamental invariant: the path field is not optional metadata, it is the canonical identity of the artifact: existence means the path exists on the target where the artifact lives (checked through the ArtifactStore, not by the artifact itself).

This constraint makes artifacts deterministic, cacheable, and transportable across machines without any special serialization protocol at the workflow level.

Examples of what artifacts can represent (and how they materialize):

  • Local files and folders (stored as-is)
  • Python dicts, lists, and other JSON-compatible objects (serialized to a .json file)
  • Arbitrary Python objects (serialized to a .pkl file via pickle)
  • Datasets and model checkpoints (stored as files or directories)
  • Remote objects such as S3 or HTTP resources (downloaded and cached locally before use)

Each artifact:

  • Has a stable ID. The id identifies the artifact within its task and is the handle that workflow edges wire together (a producer output to a consumer input). Tasks are not linked by matching ids — an explicit edge does the wiring — so a producer output and the consumer input it feeds may have different ids. Within a single task, input and output ids must be unique (see DAG planning and edges).
  • Has a path identifying its location on disk
  • Carries optional labels: a dict[str, str] of free-form metadata (labels: {subject: batch_017, role: measurement}). The runtime never interprets them and they play no part in caching; they exist for whatever reads the workflow afterwards, such as a UI grouping nodes or a lineage query.
  • Defines:
    • How to read its contents back into a Python object
    • How to write a Python object to its file representation
    • How to package / unpackage itself for transport, as portable shell commands (see pack_command / unpack_command below)

Filesystem lifecycle operations (existence checks, deletion, packaging) are not performed by the artifact itself. They run through the ArtifactStore against the target where the artifact physically lives, so they behave identically whether that is the orchestrator or a remote machine.

Defining Custom Artifacts

Artifacts inherit from BaseArtifact and are automatically registered using the runtime registry mechanism.

from horus_runtime.core.artifact.base import BaseArtifact

class MyArtifact(BaseArtifact[str]):
    kind = "my_artifact"
    kind_name = "My Artifact"
    kind_description = "A short, human-readable description of this kind."

    def read(self) -> str:
        with open(self.path) as f:
            return f.read()

    def write(self, value: str) -> None:
        with open(self.path, "w") as f:
            f.write(value)

BaseArtifact API and Required Implementations

BaseArtifact[T] is a generic, file-backed artifact abstraction:

  • Generic type: BaseArtifact[T] specifies the native Python type that read() returns and write() accepts.

  • Path normalization: Accepts both str and Path for path, always resolved to an absolute Path.

  • ID logic: Each artifact has a unique internal_id (UUID) and a user-friendly id (auto-set if not provided).

  • Event emission: Emits lifecycle events via the runtime event bus.

  • Required implementations:

    • read() -> T: Read and deserialize the artifact contents. Runs task-local, on the machine where the value is produced or consumed.
    • write(value: T) -> None: Write the native representation to disk (also task-local).
    • kind: str: Concrete discriminator value used for registry dispatch and type resolution.
  • Kind metadata (optional ClassVars):

    • kind_name: ClassVar[str]: short, human-readable name for the kind (e.g. "File"), used by client UIs, registries, and logging.
    • kind_description: ClassVar[str]: a longer description of the kind. Defaults to an empty string. For translatable text, prefer a plugin-scoped translator (see the SDK i18n guide).
  • Materialization hook (optional):

    • materialize() -> None: make the artifact's bytes available at path. The default is a no-op — a file-backed artifact assumes a producer task, or an upload, already wrote it. Value artifacts override it to write their inline value, which is what lets a workflow-owned root artifact run with no producer at all.
  • Transport hooks (override only when the artifact is not a single file):

    • pack_command(src, pkg) -> str | None: return a portable shell command that produces the single-file package pkg from the artifact materialized at src, or None when the artifact is already a single file (identity packaging — the base default).
    • unpack_command(pkg, dest) -> str | None: return a portable shell command that materializes the artifact at dest from the package pkg, or None for identity (the store simply moves the file into place).

    Both commands are executed on the target where the artifact lives (or is being materialized) by the ArtifactStore, so they must be POSIX-portable and reference only the given target-side paths. FileArtifact and other single-file artifacts inherit the identity defaults; FolderArtifact overrides them with tar commands.

ArtifactStore

An artifact is just a file-backed value description; where it lives, and how to check, delete, or transport it there, is owned by a target. The ArtifactStore (horus_runtime.core.artifact.store) is the mediator that binds an artifact to a target and performs its lifecycle operations through that target's filesystem primitives:

from horus_runtime.core.artifact.store import ArtifactStore

store = ArtifactStore(target)

await store.exists(artifact)                 # -> bool
await store.delete(artifact)                 # remove + emit delete event
package_path = await store.package(artifact) # build a single transferable file
await store.unpackage(artifact, package_path)

Because every operation runs through the target rather than local pathlib calls, the same code checks and moves an artifact whether it physically lives on the orchestrator or on a remote (e.g. SSH) machine. package() builds the package on the target and returns its path (or the artifact's own path unchanged for single-file, identity artifacts); unpackage() materializes it on the destination, or moves the file into place with the target's move_file for identity artifacts. An artifact whose pack_format is "tar.gz" (such as FolderArtifact) is packed and unpacked with the target's pack_folder / unpack_folder; any other artifact runs its pack_command / unpack_command. digest() calls the target's path_digest. A FolderArtifact subclass that overrides pack_command / unpack_command sets pack_format = None to keep them.

This package() is unrelated to the horus package CLI command, which zips a whole workflow for transport — see Packaging workflows.

ArtifactStore depends only on a small set of methods (path_on_target, path_exists, remove, path_digest, pack_folder, unpack_folder, move_file, run_command_sync, resolved_working_directory) that BaseTarget satisfies. See Target for those primitives.

Built-in Artifacts

The SDK provides the following artifact implementations:

FileArtifact

Reads and writes raw bytes. Suitable for any opaque file.

from horus_builtin.artifact.file import FileArtifact

FolderArtifact

Represents a directory. It overrides the transport hooks so a whole directory can move between targets as a single file: its pack_format is "tar.gz", so the target's pack_folder archives the folder's contents with tar czf … -C <src> . and unpack_folder extracts them with tar xzf into a freshly recreated destination.

from horus_builtin.artifact.folder import FolderArtifact

JSONArtifact[T]

Serializes any JSON-compatible Python object (dict, list, str, etc.) to a .json file. The generic parameter T is used for type-checking only; no runtime validation is performed.

from horus_builtin.artifact.json import JSONArtifact

artifact = JSONArtifact[dict](path="/tmp/result.json")
artifact.write({"key": "value"})
data = artifact.read()  # dict, cast to T

PickleArtifact[T]

Serializes arbitrary Python objects using the pickle protocol.

from horus_builtin.artifact.pickle import PickleArtifact

artifact = PickleArtifact[list](path="/tmp/data.pkl")
artifact.write([1, 2, 3])
data = artifact.read()  # list

Warning: pickle is not secure against malformed or maliciously crafted data. Never unpickle data from untrusted sources.

Value artifacts

Most artifacts describe data that some task will produce. A value artifact describes data you author directly in the workflow — a sample count, a threshold, a feature flag — while still being a normal, file-backed artifact in every other respect.

ValueArtifact[T] adds one field to BaseArtifact[T]:

  • value: T — the artifact's authored value. It round-trips through model_dump, so a workflow document can carry a value whose bytes have not been written yet.

and overrides materialize() to write that value to path. The authored value is the source of truth: a file left by an earlier run with another value is rewritten, so editing only a value re-runs its consumer. A file that already holds the value is left untouched. Only a root is materialized, so a value a producer task wrote is never overwritten.

The graph does not distinguish the two cases. A NumberArtifact produced by a task is transferred to its consumer exactly like a NumberArtifact root whose value you typed. The only runtime difference is that a root value artifact has no producer, so transfer_artifacts calls materialize() on it whenever a consumer needs it — and copies the root's value onto the consumer's own input artifact, so that substitution sees the authored value rather than the field default. When the consumer is of the same kind, the fields that qualify the value (a boolean's on / off mapping) are copied along with it.

NumberArtifact

An int | float, materialized as a JSON number. Defaults to 0.

from horus_builtin.artifact.number import NumberArtifact

samples = NumberArtifact(id="samples", path="samples.json", value=10)
samples.materialize()
samples.read()  # 10

Optional limits: min and max are inclusive, and step counts from min (or 0), with a small float tolerance so 0.3 fits a 0.1 step. step: 1 with an integer min allows only integers. An unset value starts at min. exclude lists values that are never allowed, even inside the range.

- {id: temp, kind: number, path: temp.json, value: 300, min: 250, max: 400, step: 5}
- {id: offset, kind: number, path: offset.json, value: 1, exclude: [0]}

BooleanArtifact

A bool, materialized as a JSON boolean (true / false). Defaults to False.

from horus_builtin.artifact.boolean import BooleanArtifact

verbose = BooleanArtifact(id="verbose", path="verbose.json", value=True)

Mapping the states to other values

A boolean can hand its task a value of your choosing for each state instead of true / false. Set both on and off to an Option ({value, label}, from horus_builtin.artifact.options): value is what the task receives, label is only what a UI shows (it falls back to the value). A "Use GPU" toggle:

artifacts:
  - id: device
    kind: boolean
    path: roots/device.json
    value: true # the state; stored as "gpu"
    on: {value: gpu, label: GPU}
    off: {value: cpu, label: CPU}

tasks:
  - id: train
    runtime:
      kind: command
      command: python train.py --device $device # --device gpu

With a mapping, value holds the value the task receives ("gpu" / "cpu"), never the toggle state; the toggle is on when value == on.value. A bare true / false given as value, passed to write(), or left on disk by a producer that wrote the state is translated to its mapped value, so $device, the materialized file (JSON, "gpu") and read() always agree.

from horus_builtin.artifact.boolean import BooleanArtifact
from horus_builtin.artifact.options import Option

device = BooleanArtifact(
    id="device",
    path="device.json",
    value=True,
    on=Option(value="gpu", label="GPU"),
    off=Option(value="cpu", label="CPU"),
)
device.value  # "gpu"
device.write(False)
device.read()  # "cpu"

Validation rejects setting only one of on / off, two equal values, a value that is neither of them, and mapped values that are themselves true / false (those already name the state). Without on / off, a boolean behaves exactly as above.

StringArtifact

A str, materialized as plain text — the string as authored, with no quotes. Defaults to "".

from horus_builtin.artifact.string import StringArtifact

label = StringArtifact(id="label", path="label.txt", value="baseline")

Optional limits: pattern is a Python regex that must match the whole value (single-quote it in YAML so backslashes survive), min_length and max_length count characters, exclude lists values that are never allowed (a placeholder like changeme), and multiline only switches the editor to a text area.

- {id: pdb, kind: string, path: pdb.txt, value: 1abc, pattern: '[0-9][a-z0-9]{3}'}
- {id: notes, kind: string, path: notes.txt, value: "", max_length: 500, multiline: true}
- {id: token, kind: string, path: token.txt, value: abc123, min_length: 6, exclude: [changeme]}

ChoiceArtifact

A value picked from a fixed list of options, each a {value, label} pair: value is what the task receives, label is only what a UI shows (it falls back to the value). value defaults to the first option. Strings are materialized as plain text, like StringArtifact; other scalars as JSON.

from horus_builtin.artifact.choice import ChoiceArtifact

device = ChoiceArtifact(
    id="device",
    path="device.txt",
    options=[{"value": "cpu", "label": "CPU"}, {"value": "gpu", "label": "GPU"}],
    value="gpu",
)

or in a workflow YAML:

- id: device
  kind: choice
  options:
    - {value: cpu, label: CPU}
    - {value: gpu, label: GPU}
  value: gpu

MultiChoiceArtifact

Any subset of options (same {value, label} rules as ChoiceArtifact). The value is a list kept in option order, empty by default, materialized as a JSON array. Reads and writes check every item, as for ChoiceArtifact. min_picks and max_picks bound how many options are picked.

- id: elements
  kind: multichoice
  path: elements.json
  options: [{value: H}, {value: C}, {value: O, label: Oxygen}]
  value: [H, O]
  max_picks: 2

ListArtifact

A free list of strings or numbers, empty by default, materialized as a JSON array. Optional limits: min_items and max_items bound its length, unique rejects a repeated item, min and max bound every item (items must then be numbers), and exclude lists items that are never allowed.

- {id: seeds, kind: list, path: seeds.json, value: [1, 2, 3]}
- {id: temps, kind: list, path: temps.json, value: [280, 300], min_items: 1, unique: true, min: 250, max: 400}

Limits on value artifacts (the ones above, and the options of choice / multichoice) are checked when the value is loaded or authored. Values a producer task writes are not checked against them, except for choice and multichoice. Quote option values YAML would read as booleans ("yes", "on").

A command run --device $device renders run --device gpu — the value, never the label. Validation rejects an empty options list, duplicate values (compared by their rendered text, so true and "true" collide), and a value that is not one of the options. write() and read() enforce the same, so a producer task cannot emit an undeclared value.

ValueArtifact itself is abstract and is not registered as a kind; only number, boolean, string and choice are. Subclass it to add your own — declare the concrete value type and the on-disk serialization through read/write, exactly as for any other artifact, and materialize() follows for free.

Value artifacts also change how $id renders in a command — see Artifact substitution.

Registering custom artifacts

To register and discover artifact plugins within the Horus runtime use the following entry point:

[project.entry-points."horus.artifact"]

For more details, refer to the AutoRegistry documentation.

On this page