Fan-out and fan-in
A map runs one task body many times over a collection. It is an ordinary
task (kind: horus_map) that reads one of its own inputs as a collection,
clones itself once per item (score[0], score[1], ...), and runs the clones
concurrently. Each clone is a plain horus_task that runs the map's own
runtime and executor, so you write the body once and Horus dispatches it
per item, at run time, once the collection is known.
Wiring a map into a pipeline is ordinary edge wiring: an upstream task feeds the map's collection input, and any downstream task consumes the map's folder output. The map itself only fans out. Producing the collection is whatever task writes the iterable artifact, and folding the results back into one value is whatever task consumes the folder.
What a clone sees
The body is written as if it handled a single element. That works because a clone carries every port the map declares, plus one addition and one rebinding:
- A new artifact holding the item, created under the id you pick with
over.as. In the body,$batchis one batch. - The folder output, rebound to that clone's own slot directory. In the
body,
$scoredis this clone's directory, and Horus creates it before the clone runs, so the body never has tomkdirit.
Every declared input reaches every clone unchanged, the collection included.
So a body can read its own element and the collection it came from:
$batch is one batch and $batches is all of them. That is why the item
gets an id of its own rather than replacing the collection under its id.
A map in YAML
Declare a kind: horus_map task. Its over field says which input carries
the collection and what to call each item, and its runtime and executor
are the body that runs once per item:
name: score_batches
kind: horus_workflow
tasks:
- id: split
name: Split into batches
kind: horus_task
target: { kind: local, working_directory: "./horus-work" }
runtime: { kind: command, command: "true" } # produces ./horus-out/batches/*
executor: { kind: shell }
outputs:
- { kind: folder, id: batches, path: "./horus-out/batches" }
- id: score
name: Score every batch
kind: horus_map
target: { kind: local, working_directory: "./horus-work" }
over:
input_id: batches
as: batch
inputs:
- { kind: folder, id: batches, path: "./horus-out/batches_in" }
outputs:
- { kind: folder, id: scored, path: "./horus-out/scored" }
runtime:
kind: command
command: "cp $batch/data.txt $scored/scored.txt"
executor: { kind: shell }
- id: summarize
name: Summarize results
kind: horus_task
target: { kind: local, working_directory: "./horus-work" }
runtime: { kind: command, command: "ls $scored > $summary" }
executor: { kind: shell }
inputs:
- { kind: folder, id: scored, path: "./horus-out/scored_in" }
outputs:
- { kind: file, id: summary, path: "./horus-out/summary.txt" }
edges:
- { source: split, source_output: batches, target: score, target_input: batches }
- { source: score, source_output: scored, target: summarize, target_input: scored }Run it triggered by the source task:
horus run score_batches.yaml --trigger splitThe map task's fields
| Field | Meaning |
|---|---|
over.input_id | the id of one of the map task's own inputs, the one carrying the collection |
over.as | the id each item is created under on its clone; must not collide with a declared input or output |
runtime / executor | the body, run once per item |
max_concurrency | optional upper bound on how many clones are dispatched at once |
on_item_failure | fail (default) or drop; see When an item fails |
chain | true starts each item as soon as its source map has it; see Running items through maps |
zip | more map outputs paired by item, [{input_id, as}] |
drop_exit_codes | exit codes (1 to 255) that drop an item instead of failing it |
The map must declare exactly one output, and it must be a folder. That
folder is the fan-in point: it holds one subdirectory per item, and it is what
downstream tasks consume.
Inputs shared by every clone
A body usually needs more than its item. Declare the extra inputs on the map itself and wire them with ordinary edges. Each clone receives its own copy on its own target:
- id: dock
name: Dock every ligand
kind: horus_map
over:
input_id: ligands
as: ligand
inputs:
- { kind: folder, id: ligands, path: "ligands_in" } # the collection
- { kind: file, id: receptor, path: "rec.pdbqt" } # shared by all clones
outputs:
- { kind: folder, id: complexes, path: "complexes" }
runtime:
kind: command
command: "vina --ligand $ligand --receptor $receptor --out $complexes/complex.pdbqt"
executor: { kind: shell }An edge supplies receptor once to the map, and every clone gets it. Inside
the body, $ligand is one ligand file, $ligands is still the whole folder,
and $complexes is this clone's output directory.
Collections a map can fan out over
The input named by over.input_id must be an artifact kind that knows how to
enumerate itself:
- A folder: one item per child, sorted by name. Each item points directly at the child already on the map's target, so fanning out copies no data. A child directory is enumerated as a folder artifact and a child file as a file artifact, so each item packages and transfers correctly.
- A JSON list: one item per element. Each element is written as a
single-element JSON artifact into one
<stem>.itemsdirectory next to the parent document (batches.items/00.json,batches.items/01.json, ...), so a long list does not bury the declared artifacts sharing that directory.
Custom artifact kinds can join this list; see Iterating a custom artifact kind below.
Slots and outputs
Slots are zero-padded indices, wide enough for the largest index, so they
sort the same way lexically and numerically. Eleven items produce 00 through
10, not 0 through 10.
Each slot owns one clone and one directory:
- The clone's id is
<map id>[<slot>], for examplescore[00]. - The clone's output directory is
<folder output>/<slot>, for examplescored/00.
So a downstream task that consumes the map's folder output sees one directory per item:
scored/
00/scored.txt
01/scored.txt
02/scored.txtChained maps keep their slot names
When a map iterates another map's folder output (the edge feeding its
over.input_id comes from a horus_map task), it reuses the upstream slot
names instead of numbering its items again. report[02] processes what
score[02] produced. Without gaps the names are the same as plain indices,
so nothing changes for an existing workflow. With gaps, for example after
score dropped slot 01, the downstream map has slots
00 and 02, and every later item keeps its slot, its clone id, and its
cached result.
The rule depends on where the collection comes from, not on what the entries
are called. A first map over a folder of files named 102202209,
102202300, ... still numbers its slots 0, 1, ...
Every map fed from one head map therefore shares one item space: slot 03 is
the same item all the way down. If the head's collection changes (an item
added or removed, or the count crosses 10, 100, ...), the later items get new
slot names and run again down the chain.
Running items through maps
By default a map fed by another map waits until that map has finished. Three fields let items flow through a chain of maps instead.
chain: start each item when it is ready
With chain: true, clone B[X] starts as soon as A[X] is done, while A's
other items still run. In a molecular dynamics workflow, each replica runs
equilibrate, then produce, then analyze, and no replica waits for the slowest
equilibration:
tasks:
- id: equilibrate
kind: horus_map
over: { input_id: replicas_in, as: slot }
# ... writes $equilibrated, one slot per replica
- id: produce
name: Run production MD per replica
kind: horus_map
chain: true
over: { input_id: equilibrated_in, as: slot }
inputs:
- { kind: folder, id: equilibrated_in, path: "equilibrated_in" }
outputs:
- { kind: folder, id: produced, path: "produced" }
runtime:
kind: command
command: "bash $slot/production.sh $slot $produced"
executor: { kind: shell }
edges:
- { source: equilibrate, source_output: equilibrated, target: produce, target_input: equilibrated_in }A chained map is checked when the workflow loads:
overand everyzipinput must come from a map through an edge.- A command body reads a streamed input only through its
asid, because the whole folder is not there yet.$<input id>,${task.inputs}and${task.inputs.<streamed id>}are rejected. Outputs, other inputs and other${task...}names are fine. - A Python body is not checked. It must read only its
asparameter. - An edge from a chained source cannot have a
condition. - No other input of the map can take a chained source's whole output.
A chained map keeps each source map's machine until its own last item has read from it, so a slow consumer keeps a fast source's machine running.
zip: pair more inputs by item
zip binds slot X of another map's output in clone X, next to the over
item. Each zip input must come from a map in the same item space. It works
with or without chain:
- id: report_qc
name: Report QC per sample
kind: horus_map
over: { input_id: bams, as: bam }
zip:
- { input_id: stats, as: stat }
inputs:
- { kind: folder, id: bams, path: "bams" }
- { kind: folder, id: stats, path: "stats" }
outputs:
- { kind: folder, id: reports, path: "reports" }
runtime: { kind: command, command: "qc_report $bam $stat > $reports/report.txt" }
executor: { kind: shell }A slot missing on one side was dropped there, so that item does not run.
drop_exit_codes: drop an item on purpose
A clone that exits with one of these codes drops its item instead of failing,
whatever on_item_failure says. It is never retried. Use it for a filter step:
- id: filter
kind: horus_map
drop_exit_codes: [10]
runtime: { kind: command, command: "check $sample || exit 10; ..." }- The clone ends
skippedwith reasondroppedand the errorDropped: exit code N. - Clones downstream of a dropped item never run. They end
skippedwith reasonupstream_dropped. - The map removes dropped slots from its output folder and completes, and the
run still ends
COMPLETED. Dropped items run again on the next run. - If every item was dropped, the map fails with
All N items were dropped in <map>.and its consumers do not run.
Only command bodies match: a Python body that calls sys.exit(10) fails the
item. Exit codes also depend on the backend. Slurm reports 1 for a timeout,
out of memory, node failure or cancel, and Docker uses 125 to 127 for its own
errors. Pick a code your command does not return for other reasons, such as
10.
Runtimes before 0.9.0 ignore these fields: they run the map after its source
has finished, leave a zip's $<as> unrendered, and treat a drop code as a
plain failure.
When an item fails
By default a failed clone fails the map. Under the default fail_fast
failure policy, it also
cancels the clones still running.
When the items are independent, for example one sample each, set
on_item_failure: drop on the map so one bad item does not stop the rest:
- id: score
kind: horus_map
on_item_failure: drop
# ...With drop:
- Every clone runs to the end. A failure does not cancel the siblings, under either failure policy.
- A failed clone stays
failed, with itserror, under its usual id (score[01]). - Its slot directory is removed from the map's folder output, so downstream tasks only see the items that succeeded. Logs survive as the clone's side artifacts and events; files the clone left in its slot do not.
- The map ends
completedand emits one event listing what it dropped, one line per item: the slot, the item's entry name, and the clone's error (at most 20 items, then+N more; each error cut to 500 characters). - If every item was dropped, the map fails as usual, so a downstream task never receives an empty folder.
- A cancel is never dropped: cancelling the run cancels the map.
A dropped clone does not fail the run: a run whose only failures are dropped
items ends COMPLETED. To tell such a run from a complete one, look for
failed clones of a completed map: a dropped item is a clone <map>[<slot>]
that ended failed while its map ended completed, a pair only drop
produces. Any non-zero exit counts as an item failure,
including a clone killed for running out of memory (exit 137), and so does a
failure to place the clone or transfer its inputs.
Only use drop when a missing item is an acceptable result. A map whose
outputs are all needed later, such as shards merged back into one file,
should keep the default.
Retrying dropped items
A map that dropped items does not record its own manifest, and removes one
left by an earlier run. The next run therefore runs the map again. When the
working directory persists between runs (as with the horus CLI), the clones
that succeeded skip on their own manifests, and only the dropped items run.
Once nothing is dropped, the map records its manifest and skips as usual.
With the failure policy
on_item_failure: fail | on_item_failure: drop | |
|---|---|---|
fail_fast | The first failed clone cancels its siblings and fails the run. | Clones run independently. A failed item is dropped, the map completes, and the run carries on. Only a map whose items all failed fails the run. |
continue | All clones finish, then the map fails and its downstream tasks end skipped with reason upstream_failed. | As with fail_fast. Failures elsewhere in the run still follow continue. |
In Python
Construct a MapTask like any other task and put it in the workflow's task
list (or add it with wf.add_task(...)):
from pathlib import Path
from horus_builtin.artifact.folder import FolderArtifact
from horus_builtin.executor.shell import ShellExecutor
from horus_builtin.runtime.command import CommandRuntime
from horus_builtin.task.horus_task import HorusTask
from horus_builtin.workflow.horus_workflow import HorusWorkflow
from horus_builtin.workflow.map import MapOver, MapTask
from horus_runtime.core.workflow.edge import WorkflowEdge
wf = HorusWorkflow(
name="score_batches",
tasks=[
HorusTask(
id="split",
name="split",
runtime=CommandRuntime(command="true"),
executor=ShellExecutor(),
outputs=[FolderArtifact(id="batches", path=Path("horus-out/batches"))],
),
MapTask(
id="score",
name="Score every batch",
over=MapOver(input_id="batches", item_id="batch"),
runtime=CommandRuntime(command="cp $batch/data.txt $scored/scored.txt"),
executor=ShellExecutor(),
inputs=[FolderArtifact(id="batches", path=Path("horus-out/batches_in"))],
outputs=[FolderArtifact(id="scored", path=Path("horus-out/scored"))],
),
],
edges=[
WorkflowEdge(
source="split", source_output="batches",
target="score", target_input="batches",
),
],
)In Python the as field is spelled item_id, because as is a Python
keyword. Both spellings validate, so a workflow authored in YAML and one
built in Python round-trip through the same document.
Pass max_concurrency=N to cap how many clones the map dispatches at once,
and on_item_failure="drop" to drop failed items.
After the run, a dropped item is a clone in wf.tasks with status FAILED
whose map has status COMPLETED.
Concurrency
Clones run concurrently with each other. Two knobs bound them: the map's own
max_concurrency caps that map's fan, and the workflow's
capacity gates every placement globally,
clones included, so a large fan-out never oversubscribes a machine.
The workflow's max_concurrency does not count maps or their clones. A map
starts without waiting for a slot and caps only its own clones, so six maps
can run 48 clones at once. On a laptop or a shared host, set capacity to
bound the total.
Every clone runs on its own copy of the map's target, which is what lets them run at the same time on a single declared target.
Skipping and resuming
A map memoizes like any other task. Its fingerprint covers its inputs and its
own configuration, including the body, so editing the command invalidates the
map's recorded result. A folder collection is fingerprinted by every file in
it (by content, or by size and mtime under
cache: fast), so when the
upstream task rewrites the collection, the map runs again. Clones use the
map's cache, and the map builds its folder digest from the slots its clones
already digested instead of reading the folder again. When nothing changed and the folder output exists, the map skips
entirely and does not fan out at all.
Once the map does run, each clone skips independently through ordinary
skip_if_complete behaviour: a clone whose output and manifest are intact is
left alone, and only the slots that are genuinely incomplete are rebuilt. A
clone fingerprints its own item rather than the whole collection, so changing
one item reruns only that item's clone. The exception is a body that can read
the whole collection: a ${<collection>} or ${task...} placeholder, or any
runtime other than a command (a Python function receives the collection by
parameter name). Its clones keep the collection in their fingerprint and rerun
whenever any item changes. The choice is stored on each clone as
unfingerprinted_inputs.
Setting skip_if_complete: false on the map propagates to every clone.
Clones are registered into the workflow's DAG as soon as the map builds them, so they serialize with the workflow document: a run resumed from a stored snapshot sees the previous run's clones and leaves the finished ones alone. Because slot names are stable, a re-run reuses the same clone ids and the same slot directories.
A folder output counts as complete as soon as it exists, so deleting one
slot's contents does not invalidate the map itself. To force a partial
rebuild, delete the map's own manifest (.horus/<map id>.json under its
working directory) together with the manifest of the slot you want rebuilt.
The map then re-expands, and every other slot skips.
Iterating a custom artifact kind
A map fans out over any artifact kind that mixes in IterableArtifact
(horus_runtime.core.artifact.iterable). It is a plain ABC, not a registry
root and not a kind of its own, so a plugin artifact opts in by inheriting it
next to BaseArtifact:
from pathlib import Path
from horus_builtin.artifact.file import FileArtifact
from horus_runtime.core.artifact.base import BaseArtifact
from horus_runtime.core.artifact.iterable import IterableArtifact
from horus_runtime.core.target.base import BaseTarget
class ShardedDataset(BaseArtifact[str], IterableArtifact):
kind: str = "sharded_dataset"
async def items(self, target: BaseTarget) -> list[BaseArtifact]:
entries = await target.list_dir(target.path_on_target(self))
return [
FileArtifact(id=f"{self.id}:{entry.name}", path=Path(entry.path))
for entry in sorted(entries, key=lambda e: e.name)
if entry.name.endswith(".shard")
]The contract:
- Return a deterministic list of real artifacts. The map turns each one into a clone input, so the order fixes the slot numbering. A map fed by another map uses the last part of each item's path as the slot name.
- Each item must be materialized on
target, at the path the returned artifact carries. The map hands the item to its clone as an already-placed input, so a co-located clone reads it in place and a remote one transfers it from there. Items belong next to the collection they came from, not in the consuming clone's working directory:items()runs before any clone exists, and an item reaches its clone through the ordinary transfer path like any other artifact. - Id each item
f"{self.id}:{slot}"so it is traceable back to the collection it came from. - Raise
ArtifactIterationErrorwhen the content cannot be enumerated, for example a JSON document that does not hold a list.
Enumeration should read through the target's channels (list_dir,
get_file) rather than the local filesystem, so a collection that lives on a
remote target can still be iterated.
Under the hood
The map is an ordinary DAG node that dispatches its own clones through the
scheduler's machinery: the same transfer step, target pool, and placement
manager that dispatch top-level tasks, with max_concurrency bounding this
map's share. Each clone joins the live graph through the runtime expand(...)
API behind an ordering-only edge from the map, which is why clones show up in
the dashboard (nested under the map they belong to), in stored workflow
documents, and in resumed runs, where a clone already in a terminal state is
left alone instead of being redispatched.
Clone inputs are already on the map's target when the clone is built, either because the item was enumerated there or because it is a copy of an input the map already had transferred to it. The map therefore sources every clone input from itself: a co-located clone skips the transfer entirely, and a clone placed elsewhere gets a real transfer through the registered strategy.
For a loop that repeats while a predicate holds, see the Loops guide.