Transfer strategy
Transfer strategies move artifacts between targets so that each task can access its inputs regardless of where it runs.
Why Transfer Strategies Exist
When every task runs locally there is nothing to move: all tasks share a filesystem. As soon as a task runs on a remote target, its input artifacts may live somewhere the target cannot reach directly.
Rather than embedding movement logic inside the target or the workflow, Horus
separates the concern into a standalone BaseTransferStrategy. This keeps
targets focused on dispatch and workflow code focused on orchestration.
How Transfers Fit Into Execution
Before dispatching each task the workflow calls transfer_artifacts(), which:
- determines the source target for each input artifact
- looks up the registered strategy for the
(source, destination)target pair, falling back to the target-agnosticGenericTransferwhen none is registered - calls
strategy.transfer(artifact, source, destination)
Base Transfer Strategy
All strategies inherit from BaseTransferStrategy:
class BaseTransferStrategy[S: BaseTarget, D: BaseTarget](
AutoRegistryProduct,
AutoRegistry,
entry_point="transfer",
):
registry_key: ClassVar[str] = "transfer_key:handles_source.handles_destination"
transfer_key: str | None = None
handles_source: ClassVar[type[BaseTarget]]
handles_destination: ClassVar[type[BaseTarget]]
@final
async def transfer(
self,
artifact: BaseArtifact,
source: S,
destination: D,
) -> None:
...
@abstractmethod
async def _transfer(
self,
artifact: BaseArtifact,
source: S,
destination: D,
) -> None:
...Contract
handles_source: the target type this strategy reads fromhandles_destination: the target type this strategy writes totransfer_key: derived automatically; do not set manually- implement
_transfer(), nottransfer() transfer()is the publicfinalentry point and runsTransferMiddleware
transfer_key is composed automatically from handles_source.kind and
handles_destination.kind, joined with ..
Same-filesystem short-circuit
transfer() handles one case before any strategy runs: when source and
destination report the same location_id they share a filesystem, so
there is nothing to move. The artifact is repointed at its
path_on_target on the destination and transfer() returns
without invoking _transfer() (or the middleware). Because this lives at
the single entry point every transfer flows through, no individual strategy has
to re-implement it — a strategy's _transfer() only ever runs for a genuine
cross-location move.
Registration
BaseTransferStrategy uses AutoRegistryProduct so the registry key is a
composite of the two target kind defaults:
class MyTransfer(BaseTransferStrategy[LocalTarget, SSHTarget]):
handles_source = LocalTarget
handles_destination = SSHTarget
async def _transfer(self, artifact, source, destination) -> None:
...The registry key such as "local.ssh" is derived automatically at class
definition time.
Lookup
The workflow resolves a strategy at runtime using get_from_registry(). A
registered strategy for the exact (source, destination) pair always wins; when
none is found, the workflow falls back to GenericTransfer
rather than failing:
strategy = BaseTransferStrategy.get_from_registry(source_target, task.target)
if strategy is None:
strategy = GenericTransfer() # target-agnostic fallback
await strategy.transfer(artifact, source_target, task.target)Built-in Strategies
LocalNoOpTransfer
Handles LocalTarget -> LocalTarget transfers.
When both the producing task and the consuming task run on the same local machine, the artifact is already on a shared filesystem and no movement is needed.
from horus_builtin.transfer.local_noop import LocalNoOpTransfer
class LocalNoOpTransfer(BaseTransferStrategy):
handles_source = LocalTarget
handles_destination = LocalTarget
async def _transfer(self, artifact, source, destination) -> None:
passGenericTransfer
GenericTransfer (horus_runtime.core.transfer.generic) is the target-agnostic
fallback. It moves an artifact between any two targets using only the shared
filesystem primitives every target implements, so a new target kind can transfer
artifacts to and from anywhere without anyone writing a location-specific
strategy for it.
Its _transfer() only runs for a genuine cross-location move (the
same-filesystem case is handled upstream by
transfer()). It packages the artifact on
the source (via ArtifactStore), streams the
single package file through the orchestrator with get_file → put_file, and
unpackages it on the destination:
package → get_file → put_file → unpackageGenericTransfer is not registered by key (add_to_registry = False); the
workflow uses it directly only when no specific strategy is found. Registered
strategies (like LocalNoOpTransfer or a plugin's SSH transfer) therefore always
take precedence, so you only write a custom strategy when you need transport that
is faster or smarter than the generic package-and-stream path.
Transfer Events
Every input transfer started by transfer_artifacts() is reported on the
event bus as a TransferEvent
(horus_runtime.core.transfer.events), so a subscriber or UI can show data
moving along each workflow edge.
TransferEvent has event_type = "artifact_transfer" and is emitted at
DEBUG level, so it stays out of INFO logs.
| Field | Type | Meaning |
|---|---|---|
phase | "started" | "progress" | "finished" | "failed" | Point in the transfer lifecycle |
task_id | str | Consuming task whose input is being staged |
input_id | str | Consumer input id (the output id for a cache restore) |
source_task_id | str | None | Producing task, or None for a root artifact |
source_artifact_id | str | Artifact id on the source side |
strategy | str | None | Class name of the strategy that ran |
same_location | bool | Source and destination share a location_id (nothing is moved) |
is_value | bool | The artifact is a value artifact |
cache_restore | bool | Restore of one of task_id's own outputs from the artifact cache, not an edge |
stage | "packaging" | "transferring" | "uploading" | "unpacking" | None | Current stage (progress only) |
fraction | float | None | 0.0–1.0, only when the byte total is known |
bytes_done / bytes_total | int | None | Byte counts, when the strategy reports them |
task_id, input_id and source_task_id identify exactly one workflow edge.
Phases
The workflow wraps each strategy.transfer(...) call in
transfer_scope(template):
startedis emitted before the strategy runsprogressis emitted whenever the strategy callsreport_transfer_progressfinishedis emitted when the strategy returnsfailedis emitted when the strategy raises anException(the error is re-raised)
A cancelled transfer emits nothing after started; consumers should drop open
transfers when the run ends.
Each event is a distinct event with its own event_id and timestamp.
Reporting Progress From a Custom Strategy
Strategies do not need to know which task or edge they serve. Call
report_transfer_progress() from _transfer() and the active scope fills in the
rest:
from horus_runtime.core.transfer.events import report_transfer_progress
class LocalToSSHTransfer(BaseTransferStrategy[LocalTarget, SSHTarget]):
handles_source = LocalTarget
handles_destination = SSHTarget
async def _transfer(self, artifact, source, destination) -> None:
report_transfer_progress(stage="packaging")
package = await self._package(artifact)
total = package.stat().st_size
def on_chunk(sent: int) -> None:
report_transfer_progress(stage="uploading", done=sent, total=total)
await self._upload(package, destination, callback=on_chunk)
report_transfer_progress(stage="unpacking")
await self._unpack(artifact, destination)stage,doneandtotalare all optional;done/totalare bytes andfractionis derived only whentotal > 0- calls are throttled to roughly one event per 0.25 s per transfer; a stage
change or reaching
fraction == 1.0always emits, so it is safe to call from a per-chunk callback - outside a
transfer_scope(e.g. callingstrategy.transfer()directly) or without a bootedHorusContext, it is a no-op
The built-in GenericTransfer reports the packaging,
transferring and unpacking stages.
Silent Hand-offs
Only workflow edges are reported. Internal hand-offs that are not edges emit no
TransferEvent: a map task feeding its clones builds
its sources with EdgeSource(..., report=False), since one event per clone and
input would flood the bus.
Exceptions
| Exception | When raised |
|---|---|
TransferStrategyNotFoundError | No registered strategy handles the resolved (source, destination) target pair |
OrchestratorTargetNotSetError | A root input artifact needs a source but workflow.orchestrator_target is None |
Both are subclasses of TransferError.
Implementing a Custom Strategy
from horus_runtime.core.artifact.base import BaseArtifact
from horus_runtime.core.transfer.strategy import BaseTransferStrategy
from my_plugin.target.ssh import SSHTarget
from horus_builtin.target.local import LocalTarget
class LocalToSSHTransfer(BaseTransferStrategy[LocalTarget, SSHTarget]):
handles_source = LocalTarget
handles_destination = SSHTarget
async def _transfer(
self,
artifact: BaseArtifact,
source: LocalTarget,
destination: SSHTarget,
) -> None:
# Upload artifact.path to destination's working_directory over SCP
...Registering Custom Strategies
Expose the strategy through a horus.transfer entry point:
[project.entry-points."horus.transfer"]
local_to_ssh = "my_plugin.transfer.local_ssh"For more details, refer to the Auto-Registry documentation and the Auto-Registry Product documentation.