Temple Compute Docs

Fan-out and fan-in

A map runs one task body many times over a collection. It is an ordinary task (kind: horus_map) that reads one of its own inputs as a collection, clones itself once per item (score[0], score[1], ...), and runs the clones concurrently. Each clone is a plain horus_task that runs the map's own runtime and executor, so you write the body once and Horus dispatches it per item, at run time, once the collection is known.

Wiring a map into a pipeline is ordinary edge wiring: an upstream task feeds the map's collection input, and any downstream task consumes the map's folder output. The map itself only fans out. Producing the collection is whatever task writes the iterable artifact, and folding the results back into one value is whatever task consumes the folder.

What a clone sees

The body is written as if it handled a single element. That works because a clone carries every port the map declares, plus one addition and one rebinding:

  • A new artifact holding the item, created under the id you pick with over.as. In the body, $batch is one batch.
  • The folder output, rebound to that clone's own slot directory. In the body, $scored is this clone's directory, and Horus creates it before the clone runs, so the body never has to mkdir it.

Every declared input reaches every clone unchanged, the collection included. So a body can read its own element and the collection it came from: $batch is one batch and $batches is all of them. That is why the item gets an id of its own rather than replacing the collection under its id.

A map in YAML

Declare a kind: horus_map task. Its over field says which input carries the collection and what to call each item, and its runtime and executor are the body that runs once per item:

score_batches.yaml
name: score_batches
kind: horus_workflow

tasks:
  - id: split
    name: Split into batches
    kind: horus_task
    target: { kind: local, working_directory: "./horus-work" }
    runtime: { kind: command, command: "true" }   # produces ./horus-out/batches/*
    executor: { kind: shell }
    outputs:
      - { kind: folder, id: batches, path: "./horus-out/batches" }

  - id: score
    name: Score every batch
    kind: horus_map
    target: { kind: local, working_directory: "./horus-work" }
    over:
      input_id: batches
      as: batch
    inputs:
      - { kind: folder, id: batches, path: "./horus-out/batches_in" }
    outputs:
      - { kind: folder, id: scored, path: "./horus-out/scored" }
    runtime:
      kind: command
      command: "cp $batch/data.txt $scored/scored.txt"
    executor: { kind: shell }

  - id: summarize
    name: Summarize results
    kind: horus_task
    target: { kind: local, working_directory: "./horus-work" }
    runtime: { kind: command, command: "ls $scored > $summary" }
    executor: { kind: shell }
    inputs:
      - { kind: folder, id: scored, path: "./horus-out/scored_in" }
    outputs:
      - { kind: file, id: summary, path: "./horus-out/summary.txt" }

edges:
  - { source: split, source_output: batches, target: score, target_input: batches }
  - { source: score, source_output: scored, target: summarize, target_input: scored }

Run it triggered by the source task:

horus run score_batches.yaml --trigger split

The map task's fields

FieldMeaning
over.input_idthe id of one of the map task's own inputs, the one carrying the collection
over.asthe id each item is created under on its clone; must not collide with a declared input or output
runtime / executorthe body, run once per item
max_concurrencyoptional upper bound on how many clones are dispatched at once
on_item_failurefail (default) or drop; see When an item fails
chaintrue starts each item as soon as its source map has it; see Running items through maps
zipmore map outputs paired by item, [{input_id, as}]
drop_exit_codesexit codes (1 to 255) that drop an item instead of failing it

The map must declare exactly one output, and it must be a folder. That folder is the fan-in point: it holds one subdirectory per item, and it is what downstream tasks consume.

Inputs shared by every clone

A body usually needs more than its item. Declare the extra inputs on the map itself and wire them with ordinary edges. Each clone receives its own copy on its own target:

  - id: dock
    name: Dock every ligand
    kind: horus_map
    over:
      input_id: ligands
      as: ligand
    inputs:
      - { kind: folder, id: ligands, path: "ligands_in" }   # the collection
      - { kind: file, id: receptor, path: "rec.pdbqt" }     # shared by all clones
    outputs:
      - { kind: folder, id: complexes, path: "complexes" }
    runtime:
      kind: command
      command: "vina --ligand $ligand --receptor $receptor --out $complexes/complex.pdbqt"
    executor: { kind: shell }

An edge supplies receptor once to the map, and every clone gets it. Inside the body, $ligand is one ligand file, $ligands is still the whole folder, and $complexes is this clone's output directory.

Collections a map can fan out over

The input named by over.input_id must be an artifact kind that knows how to enumerate itself:

  • A folder: one item per child, sorted by name. Each item points directly at the child already on the map's target, so fanning out copies no data. A child directory is enumerated as a folder artifact and a child file as a file artifact, so each item packages and transfers correctly.
  • A JSON list: one item per element. Each element is written as a single-element JSON artifact into one <stem>.items directory next to the parent document (batches.items/00.json, batches.items/01.json, ...), so a long list does not bury the declared artifacts sharing that directory.

Custom artifact kinds can join this list; see Iterating a custom artifact kind below.

Slots and outputs

Slots are zero-padded indices, wide enough for the largest index, so they sort the same way lexically and numerically. Eleven items produce 00 through 10, not 0 through 10.

Each slot owns one clone and one directory:

  • The clone's id is <map id>[<slot>], for example score[00].
  • The clone's output directory is <folder output>/<slot>, for example scored/00.

So a downstream task that consumes the map's folder output sees one directory per item:

scored/
  00/scored.txt
  01/scored.txt
  02/scored.txt

Chained maps keep their slot names

When a map iterates another map's folder output (the edge feeding its over.input_id comes from a horus_map task), it reuses the upstream slot names instead of numbering its items again. report[02] processes what score[02] produced. Without gaps the names are the same as plain indices, so nothing changes for an existing workflow. With gaps, for example after score dropped slot 01, the downstream map has slots 00 and 02, and every later item keeps its slot, its clone id, and its cached result.

The rule depends on where the collection comes from, not on what the entries are called. A first map over a folder of files named 102202209, 102202300, ... still numbers its slots 0, 1, ...

Every map fed from one head map therefore shares one item space: slot 03 is the same item all the way down. If the head's collection changes (an item added or removed, or the count crosses 10, 100, ...), the later items get new slot names and run again down the chain.

Running items through maps

By default a map fed by another map waits until that map has finished. Three fields let items flow through a chain of maps instead.

chain: start each item when it is ready

With chain: true, clone B[X] starts as soon as A[X] is done, while A's other items still run. In a molecular dynamics workflow, each replica runs equilibrate, then produce, then analyze, and no replica waits for the slowest equilibration:

tasks:
  - id: equilibrate
    kind: horus_map
    over: { input_id: replicas_in, as: slot }
    # ... writes $equilibrated, one slot per replica

  - id: produce
    name: Run production MD per replica
    kind: horus_map
    chain: true
    over: { input_id: equilibrated_in, as: slot }
    inputs:
      - { kind: folder, id: equilibrated_in, path: "equilibrated_in" }
    outputs:
      - { kind: folder, id: produced, path: "produced" }
    runtime:
      kind: command
      command: "bash $slot/production.sh $slot $produced"
    executor: { kind: shell }

edges:
  - { source: equilibrate, source_output: equilibrated, target: produce, target_input: equilibrated_in }

A chained map is checked when the workflow loads:

  • over and every zip input must come from a map through an edge.
  • A command body reads a streamed input only through its as id, because the whole folder is not there yet. $<input id>, ${task.inputs} and ${task.inputs.<streamed id>} are rejected. Outputs, other inputs and other ${task...} names are fine.
  • A Python body is not checked. It must read only its as parameter.
  • An edge from a chained source cannot have a condition.
  • No other input of the map can take a chained source's whole output.

A chained map keeps each source map's machine until its own last item has read from it, so a slow consumer keeps a fast source's machine running.

zip: pair more inputs by item

zip binds slot X of another map's output in clone X, next to the over item. Each zip input must come from a map in the same item space. It works with or without chain:

  - id: report_qc
    name: Report QC per sample
    kind: horus_map
    over: { input_id: bams, as: bam }
    zip:
      - { input_id: stats, as: stat }
    inputs:
      - { kind: folder, id: bams, path: "bams" }
      - { kind: folder, id: stats, path: "stats" }
    outputs:
      - { kind: folder, id: reports, path: "reports" }
    runtime: { kind: command, command: "qc_report $bam $stat > $reports/report.txt" }
    executor: { kind: shell }

A slot missing on one side was dropped there, so that item does not run.

drop_exit_codes: drop an item on purpose

A clone that exits with one of these codes drops its item instead of failing, whatever on_item_failure says. It is never retried. Use it for a filter step:

  - id: filter
    kind: horus_map
    drop_exit_codes: [10]
    runtime: { kind: command, command: "check $sample || exit 10; ..." }
  • The clone ends skipped with reason dropped and the error Dropped: exit code N.
  • Clones downstream of a dropped item never run. They end skipped with reason upstream_dropped.
  • The map removes dropped slots from its output folder and completes, and the run still ends COMPLETED. Dropped items run again on the next run.
  • If every item was dropped, the map fails with All N items were dropped in <map>. and its consumers do not run.

Only command bodies match: a Python body that calls sys.exit(10) fails the item. Exit codes also depend on the backend. Slurm reports 1 for a timeout, out of memory, node failure or cancel, and Docker uses 125 to 127 for its own errors. Pick a code your command does not return for other reasons, such as 10.

Runtimes before 0.9.0 ignore these fields: they run the map after its source has finished, leave a zip's $<as> unrendered, and treat a drop code as a plain failure.

When an item fails

By default a failed clone fails the map. Under the default fail_fast failure policy, it also cancels the clones still running.

When the items are independent, for example one sample each, set on_item_failure: drop on the map so one bad item does not stop the rest:

  - id: score
    kind: horus_map
    on_item_failure: drop
    # ...

With drop:

  • Every clone runs to the end. A failure does not cancel the siblings, under either failure policy.
  • A failed clone stays failed, with its error, under its usual id (score[01]).
  • Its slot directory is removed from the map's folder output, so downstream tasks only see the items that succeeded. Logs survive as the clone's side artifacts and events; files the clone left in its slot do not.
  • The map ends completed and emits one event listing what it dropped, one line per item: the slot, the item's entry name, and the clone's error (at most 20 items, then +N more; each error cut to 500 characters).
  • If every item was dropped, the map fails as usual, so a downstream task never receives an empty folder.
  • A cancel is never dropped: cancelling the run cancels the map.

A dropped clone does not fail the run: a run whose only failures are dropped items ends COMPLETED. To tell such a run from a complete one, look for failed clones of a completed map: a dropped item is a clone <map>[<slot>] that ended failed while its map ended completed, a pair only drop produces. Any non-zero exit counts as an item failure, including a clone killed for running out of memory (exit 137), and so does a failure to place the clone or transfer its inputs.

Only use drop when a missing item is an acceptable result. A map whose outputs are all needed later, such as shards merged back into one file, should keep the default.

Retrying dropped items

A map that dropped items does not record its own manifest, and removes one left by an earlier run. The next run therefore runs the map again. When the working directory persists between runs (as with the horus CLI), the clones that succeeded skip on their own manifests, and only the dropped items run. Once nothing is dropped, the map records its manifest and skips as usual.

With the failure policy

on_item_failure: failon_item_failure: drop
fail_fastThe first failed clone cancels its siblings and fails the run.Clones run independently. A failed item is dropped, the map completes, and the run carries on. Only a map whose items all failed fails the run.
continueAll clones finish, then the map fails and its downstream tasks end skipped with reason upstream_failed.As with fail_fast. Failures elsewhere in the run still follow continue.

In Python

Construct a MapTask like any other task and put it in the workflow's task list (or add it with wf.add_task(...)):

from pathlib import Path

from horus_builtin.artifact.folder import FolderArtifact
from horus_builtin.executor.shell import ShellExecutor
from horus_builtin.runtime.command import CommandRuntime
from horus_builtin.task.horus_task import HorusTask
from horus_builtin.workflow.horus_workflow import HorusWorkflow
from horus_builtin.workflow.map import MapOver, MapTask
from horus_runtime.core.workflow.edge import WorkflowEdge

wf = HorusWorkflow(
    name="score_batches",
    tasks=[
        HorusTask(
            id="split",
            name="split",
            runtime=CommandRuntime(command="true"),
            executor=ShellExecutor(),
            outputs=[FolderArtifact(id="batches", path=Path("horus-out/batches"))],
        ),
        MapTask(
            id="score",
            name="Score every batch",
            over=MapOver(input_id="batches", item_id="batch"),
            runtime=CommandRuntime(command="cp $batch/data.txt $scored/scored.txt"),
            executor=ShellExecutor(),
            inputs=[FolderArtifact(id="batches", path=Path("horus-out/batches_in"))],
            outputs=[FolderArtifact(id="scored", path=Path("horus-out/scored"))],
        ),
    ],
    edges=[
        WorkflowEdge(
            source="split", source_output="batches",
            target="score", target_input="batches",
        ),
    ],
)

In Python the as field is spelled item_id, because as is a Python keyword. Both spellings validate, so a workflow authored in YAML and one built in Python round-trip through the same document.

Pass max_concurrency=N to cap how many clones the map dispatches at once, and on_item_failure="drop" to drop failed items. After the run, a dropped item is a clone in wf.tasks with status FAILED whose map has status COMPLETED.

Concurrency

Clones run concurrently with each other. Two knobs bound them: the map's own max_concurrency caps that map's fan, and the workflow's capacity gates every placement globally, clones included, so a large fan-out never oversubscribes a machine.

The workflow's max_concurrency does not count maps or their clones. A map starts without waiting for a slot and caps only its own clones, so six maps can run 48 clones at once. On a laptop or a shared host, set capacity to bound the total.

Every clone runs on its own copy of the map's target, which is what lets them run at the same time on a single declared target.

Skipping and resuming

A map memoizes like any other task. Its fingerprint covers its inputs and its own configuration, including the body, so editing the command invalidates the map's recorded result. A folder collection is fingerprinted by every file in it (by content, or by size and mtime under cache: fast), so when the upstream task rewrites the collection, the map runs again. Clones use the map's cache, and the map builds its folder digest from the slots its clones already digested instead of reading the folder again. When nothing changed and the folder output exists, the map skips entirely and does not fan out at all.

Once the map does run, each clone skips independently through ordinary skip_if_complete behaviour: a clone whose output and manifest are intact is left alone, and only the slots that are genuinely incomplete are rebuilt. A clone fingerprints its own item rather than the whole collection, so changing one item reruns only that item's clone. The exception is a body that can read the whole collection: a ${<collection>} or ${task...} placeholder, or any runtime other than a command (a Python function receives the collection by parameter name). Its clones keep the collection in their fingerprint and rerun whenever any item changes. The choice is stored on each clone as unfingerprinted_inputs. Setting skip_if_complete: false on the map propagates to every clone.

Clones are registered into the workflow's DAG as soon as the map builds them, so they serialize with the workflow document: a run resumed from a stored snapshot sees the previous run's clones and leaves the finished ones alone. Because slot names are stable, a re-run reuses the same clone ids and the same slot directories.

A folder output counts as complete as soon as it exists, so deleting one slot's contents does not invalidate the map itself. To force a partial rebuild, delete the map's own manifest (.horus/<map id>.json under its working directory) together with the manifest of the slot you want rebuilt. The map then re-expands, and every other slot skips.

Iterating a custom artifact kind

A map fans out over any artifact kind that mixes in IterableArtifact (horus_runtime.core.artifact.iterable). It is a plain ABC, not a registry root and not a kind of its own, so a plugin artifact opts in by inheriting it next to BaseArtifact:

from pathlib import Path

from horus_builtin.artifact.file import FileArtifact
from horus_runtime.core.artifact.base import BaseArtifact
from horus_runtime.core.artifact.iterable import IterableArtifact
from horus_runtime.core.target.base import BaseTarget


class ShardedDataset(BaseArtifact[str], IterableArtifact):
    kind: str = "sharded_dataset"

    async def items(self, target: BaseTarget) -> list[BaseArtifact]:
        entries = await target.list_dir(target.path_on_target(self))
        return [
            FileArtifact(id=f"{self.id}:{entry.name}", path=Path(entry.path))
            for entry in sorted(entries, key=lambda e: e.name)
            if entry.name.endswith(".shard")
        ]

The contract:

  • Return a deterministic list of real artifacts. The map turns each one into a clone input, so the order fixes the slot numbering. A map fed by another map uses the last part of each item's path as the slot name.
  • Each item must be materialized on target, at the path the returned artifact carries. The map hands the item to its clone as an already-placed input, so a co-located clone reads it in place and a remote one transfers it from there. Items belong next to the collection they came from, not in the consuming clone's working directory: items() runs before any clone exists, and an item reaches its clone through the ordinary transfer path like any other artifact.
  • Id each item f"{self.id}:{slot}" so it is traceable back to the collection it came from.
  • Raise ArtifactIterationError when the content cannot be enumerated, for example a JSON document that does not hold a list.

Enumeration should read through the target's channels (list_dir, get_file) rather than the local filesystem, so a collection that lives on a remote target can still be iterated.

Under the hood

The map is an ordinary DAG node that dispatches its own clones through the scheduler's machinery: the same transfer step, target pool, and placement manager that dispatch top-level tasks, with max_concurrency bounding this map's share. Each clone joins the live graph through the runtime expand(...) API behind an ordering-only edge from the map, which is why clones show up in the dashboard (nested under the map they belong to), in stored workflow documents, and in resumed runs, where a clone already in a terminal state is left alone instead of being redispatched.

Clone inputs are already on the map's target when the clone is built, either because the item was enumerated there or because it is a copy of an input the map already had transferred to it. The map therefore sources every clone input from itself: a co-located clone skips the transfer entirely, and a clone placed elsewhere gets a real transfer through the registered strategy.

For a loop that repeats while a predicate holds, see the Loops guide.

On this page