Results & artifacts

Results & artifacts#

Marking the headline result of a command and declaring the files it produces as exportable artifacts.

Mark an output as the result#

Flag the output that is what the experiment was for with –result, so knit describe highlights it.

APIs: knit_with_output, knit_output

A command may record several outputs, but usually one of them is the result — what the experiment was for. Append --result to its knit_with_output declaration to say so:

@command "measure" "Square a value and mark the result."
@with_required "x:integer" "The value to square."
@with_output   "square:integer" "0" "The squared value (the result)." --result
@with_output   "note:string"    ""  "An intermediate note (not a result)."
@with_table
_measure() {
    local x
    x="$(knit_get_parameter "x" "$@")"
    knit_output "square" "$(( x * x ))"        # the headline value result
    knit_output "note"   "squared x=${x}"      # recorded, but not a result
    printf 'square=%s\n' "$(( x * x ))"
}
@done

knit_register "measure" _measure "Square a value and mark the result."
knit_with_required "x:integer" "The value to square."
knit_with_output   "square:integer" "0" "The squared value (the result)." --result
knit_with_output   "note:string"    ""  "An intermediate note (not a result)."
knit_with_table
_measure() {
    local x
    x="$(knit_get_parameter "x" "$@")"
    knit_output "square" "$(( x * x ))"        # the headline value result
    knit_output "note"   "squared x=${x}"      # recorded, but not a result
    printf 'square=%s\n' "$(( x * x ))"
}
knit_done

--result is orthogonal to the output’s type and to recording: it is valid on any output (a scalar such as square here, or a file / directory), and it changes no value and moves no file. It is a declaration-time marker only, so knit describe can surface it: the flagged output carries a result tag in the human and Markdown views and a "result": true field in the JSON and YAML views (see Describe the command tree). An unflagged output such as note reports result false.

Set the value from the body with knit_output exactly as for any output (see Emit outputs from a command); the flag lives on the declaration, not the emission. For a file or directory you also want packaged for export, declare it with knit_with_output_artifact ... --result instead (see Declare and bind an artifact).

Declare and bind an artifact#

Declare a produced file with knit_with_output_artifact, write it under knit_artifact_dir, then bind it with knit_artifact.

APIs: knit_with_output_artifact, knit_artifact_dir, knit_artifact

An artifact is a file or directory a command produces that you want kept for export — a dataset, a plot, a captured config. Declare it with knit_with_output_artifact "name:type" "description" (the type must be file or directory), write it under the artifacts root reported by knit_artifact_dir, then bind it from the body with knit_artifact:

@command "tabulate" "Write a data table as an artifact."
@with_output   "rows:integer" "0" "How many rows were written." --result
@with_output_artifact "table:file" "The data table (CSV)." --result
@with_table
_tabulate() {
    # knit_artifact_dir is the artifacts/ root: write into it, then declare.
    local out
    out="$(knit_artifact_dir)"
    mkdir -p "${out}"
    printf 'i,i2\n1,1\n2,4\n3,9\n' > "${out}/table.csv"
    knit_artifact "table" "table.csv"          # already inside artifacts/
    knit_output   "rows" "3"
    printf 'wrote %s\n' "${out}/table.csv"
}
@done

knit_register "tabulate" _tabulate "Write a data table as an artifact."
knit_with_output   "rows:integer" "0" "How many rows were written." --result
knit_with_output_artifact "table:file" "The data table (CSV)." --result
knit_with_table
_tabulate() {
    # knit_artifact_dir is the artifacts/ root: write into it, then declare.
    local out
    out="$(knit_artifact_dir)"
    mkdir -p "${out}"
    printf 'i,i2\n1,1\n2,4\n3,9\n' > "${out}/table.csv"
    knit_artifact "table" "table.csv"          # already inside artifacts/
    knit_output   "rows" "3"
    printf 'wrote %s\n' "${out}/table.csv"
}
knit_done

The <linked-path> you bind is always inside the artifacts root, given relative to it (table.csv) or as an absolute path within it; a path outside the root is fatal. knit records the artifact’s value as that artifacts-relative path, so the database holds no absolute machine path and the record stays relocatable. Add --result (as on table here) to mark the artifact as the headline result too (see Mark an output as the result). To give the artifact a semantic kind — so a consumer can require a CSV table rather than merely a file — register the kind first and name it here (see Declare an artifact kind).

knit records each artifact as one row in the framework-owned artifacts table — its path, name, type, content checksum, and result flag — not as a column of the producing command’s own table, and links that row to the producing invocation with a produced provenance edge. The content digest is always recorded (there is no --no-checksum opt-out for an artifact). Because each artifact is its own node, which invocation produced this file? is a reverse walk of the produced edge (see Trace an artifact back to its producer), and knit describe lists artifacts in a dedicated Artifacts section, separate from the value outputs. The entry must exist and match its declared type when you bind it. Artifacts are write-once: binding the same path twice is fatal, so a command re-run that produces a fixed-named artifact needs a distinct <linked-path> per run. To create the entry from a file that lives elsewhere, use the --link-from / --copy-from shortcuts (see Link or copy an artifact into place).

Fan out a variadic output#

Declare a *-quantified output artifact and bind the same name many times, so one command produces a whole collection.

APIs: knit_with_output_artifact, knit_artifact_dir, knit_artifact

A scalar artifact binds once. Add a * (zero or more) or + (one or more) to the kind and the name becomes a collection: the body may bind it any number of times, and each binding is its own artifacts row with its own produced edge. Use it when one run scatters a whole set of files — shards, frames, per-seed outputs — that you do not know the count of up front:

# A "*" (zero or more) output is a COLLECTION: the body may bind the same name
# any number of times, and each binding is its own artifacts row with its own
# "produced" edge. (Use "+" for one-or-more, which is fatal if the body binds
# nothing.) Here one run scatters `n` CSV shards under the artifacts root.
@command "shard" "Fan out a range into several CSV shards."
@with_optional "n:integer" "3" "How many shards to write."
@with_output_artifact "shards:csvfile*" "The CSV shards (zero or more)."
@with_table
_shard() {
    local n out i
    n="$(knit_get_parameter "n" "$@")"
    out="$(knit_artifact_dir)"
    mkdir -p "${out}"
    for (( i = 1; i <= n; i++ )); do
        printf 'id,sq\n%d,%d\n' "${i}" "$(( i * i ))" > "${out}/shard-${i}.csv"
        knit_artifact "shards" "shard-${i}.csv"    # bind one member of the collection
    done
    printf 'wrote %s shard(s)\n' "${n}"
}
@done

# A "*" (zero or more) output is a COLLECTION: the body may bind the same name
# any number of times, and each binding is its own artifacts row with its own
# "produced" edge. (Use "+" for one-or-more, which is fatal if the body binds
# nothing.) Here one run scatters `n` CSV shards under the artifacts root.
knit_register "shard" _shard "Fan out a range into several CSV shards."
knit_with_optional "n:integer" "3" "How many shards to write."
knit_with_output_artifact "shards:csvfile*" "The CSV shards (zero or more)."
knit_with_table
_shard() {
    local n out i
    n="$(knit_get_parameter "n" "$@")"
    out="$(knit_artifact_dir)"
    mkdir -p "${out}"
    for (( i = 1; i <= n; i++ )); do
        printf 'id,sq\n%d,%d\n' "${i}" "$(( i * i ))" > "${out}/shard-${i}.csv"
        knit_artifact "shards" "shard-${i}.csv"    # bind one member of the collection
    done
    printf 'wrote %s shard(s)\n' "${n}"
}
knit_done

The quantifier is the only difference from a scalar declaration. * accepts an empty fan-out (a run that produced nothing is not an error); + requires at least one binding and is fatal after the body if none was made, so it states at least one of these must exist. The write-once rule is unchanged: each member still needs a distinct artifacts-relative path (here shard-${i}.csv), so a loop that reuses one path is rejected. Every member is recorded exactly as a scalar artifact is — same path/name/kind/checksum columns, same produced edge — so the collection needs no special handling to trace, remove, or query; it is simply several artifact rows sharing one declared name. Consume the whole set at once with a variadic input (see Consume many artifacts with a glob).

Trace an artifact back to its producer#

Recover which invocation produced a file by walking the produced edge from its artifacts row back to the producing command.

APIs: query:graph, query:sql, knit_with_output_artifact

Each artifact is recorded as one row in the artifacts table (its path, name, type, checksum, and result), linked to the invocation that made it by a produced provenance edge — not as a column of the producing command’s own table (see Declare and bind an artifact). So which invocation produced this file? is a reverse walk of that edge, keyed on the artifacts-relative path.

With query graph the walk is a Cypher match. The producing node needs no label — read the producer off the edge, so the same query works no matter which command made the file:

$ ./exp.sh query graph --format column --header --exec \
    "MATCH (t)-[e:produced]->(a:artifacts)
       WHERE a.path = 'table.csv'
       RETURN e.source_name, e.source_id"

e.source_name is the producing command and e.source_id is its invocation id, which joins back to that command’s own recorded row for the rest of the provenance (its parameters, its call edge to a parent, and so on).

The same lookup in query sql joins the artifacts row to the __provenance__ edge on the artifact id:

$ ./exp.sh query sql --format column --header --exec \
    "SELECT p.source_name, p.source_id
       FROM artifacts a
       JOIN __provenance__ p ON p.target_id = a.id AND p.edge_type = 'produced'
      WHERE a.path = 'table.csv'"

Both directions are open: forward (RETURN a.path for a given producer, or a join the other way) lists every file an invocation produced. See Query the provenance graph for the Cypher subset and Run raw SQL for the shared --format / --header options.

Walk an artifact’s full lineage#

Walk producer –produced–> artifact –used_by–> consumer in one query by joining the two provenance edges on the artifact’s row.

APIs: query:graph, query:sql, knit_with_input_artifact, knit_with_output_artifact

An artifact sits between the command that made it and the commands that read it. The producer leaves a produced edge into the artifact’s row (see Trace an artifact back to its producer); each consumer leaves a used_by edge out of it (see Consume an artifact by kind). Because both edges meet at the same artifacts node, one query walks the whole chain producer --produced--> artifact --used_by--> consumer.

With query graph the chain is a single Cypher path through the artifact node, keyed on its artifacts-relative path:

$ ./exp.sh query graph --format column --header --exec \
    "MATCH (p)-[pr:produced]->(a:artifacts)-[ub:used_by]->(c)
       WHERE a.path = 'table.csv'
       RETURN pr.source_name, a.kind, ub.target_name"

The producing and consuming nodes need no label — name them off the edges (pr.source_name, ub.target_name), so the same query works whatever commands sit at the ends. To list everything that read a given artifact, keep only the used_by half (RETURN ub.target_name); to list every artifact a command consumed, match the used_by edge into it.

The same walk in query sql joins the two __provenance__ edges to the artifacts row on the artifact id:

$ ./exp.sh query sql --format column --header --exec \
    "SELECT pr.source_name AS producer, ub.target_name AS consumer
       FROM artifacts a
       JOIN __provenance__ pr ON pr.target_id = a.id AND pr.edge_type = 'produced'
       JOIN __provenance__ ub ON ub.source_id = a.id AND ub.edge_type = 'used_by'
      WHERE a.path = 'table.csv'"

See Query the provenance graph for the Cypher subset and Run raw SQL for the shared --format / --header options.

Declare an artifact kind#

Give an artifact a semantic kind with knit_register_artifact, then produce it by naming the kind in knit_with_output_artifact.

APIs: knit_register_artifact, knit_with_output_artifact, knit_artifact

An artifact’s type is its physical form — file or directory. Its kind is what it means: a CSV table, a checkpoint, a mesh. A kind is a semantic label backed by exactly one physical type, so a consumer can require a CSV table rather than merely a file. Declare a kind once at the top level with knit_register_artifact "kind:type" "description", then a producer names the kind in place of the bare physical type:

# A kind is a semantic label for an artifact, backed by exactly one physical
# type (file or directory). Declare it once at the top level; a producer then
# names the kind in place of the bare physical type.
@artifact "csvfile:file" "A tabulated result in CSV format."

@command "tabulate" "Write a data table as a csvfile artifact."
@with_output_artifact "table:csvfile" "The data table (CSV)." --result
@with_table
_tabulate() {
    # knit_artifact_dir is the artifacts/ root: write into it, then bind.
    local out
    out="$(knit_artifact_dir)"
    mkdir -p "${out}"
    printf 'i,i2\n1,1\n2,4\n3,9\n' > "${out}/table.csv"
    knit_artifact "table" "table.csv"          # recorded with kind=csvfile
    printf 'wrote %s\n' "${out}/table.csv"
}
@done

# A kind is a semantic label for an artifact, backed by exactly one physical
# type (file or directory). Declare it once at the top level; a producer then
# names the kind in place of the bare physical type.
knit_register_artifact "csvfile:file" "A tabulated result in CSV format."

knit_register "tabulate" _tabulate "Write a data table as a csvfile artifact."
knit_with_output_artifact "table:csvfile" "The data table (CSV)." --result
knit_with_table
_tabulate() {
    # knit_artifact_dir is the artifacts/ root: write into it, then bind.
    local out
    out="$(knit_artifact_dir)"
    mkdir -p "${out}"
    printf 'i,i2\n1,1\n2,4\n3,9\n' > "${out}/table.csv"
    knit_artifact "table" "table.csv"          # recorded with kind=csvfile
    printf 'wrote %s\n' "${out}/table.csv"
}
knit_done

file, directory, and the dir alias are builtin kinds — they are their own physical type and need no declaration, so "table:file" keeps working unchanged (see Declare and bind an artifact). A named kind is declared once; redefining any kind (a builtin included) is fatal, and the physical type must itself be file or directory.

The kind is recorded in the kind column of the framework-owned artifacts table, next to the physical type; a bare table:file records type=file, kind=file, while table:csvfile records type=file, kind=csvfile. knit describe shows the kind (not the bare type) for a declared output artifact. The kind is the contract a consumer checks against — see Consume an artifact by kind.

Consume an artifact by kind#

Require a produced artifact of a given kind with knit_with_input_artifact, resolve it with knit_input_artifact_path, and record a used_by edge.

APIs: knit_with_input_artifact, knit_input_artifact_path

A command consumes an artifact another command produced by declaring knit_with_input_artifact "name:kind" "description". This registers a required string parameter whose value is the artifacts-relative path of a recorded artifact of that kind. In the body, knit_input_artifact_path resolves that path to the on-disk file:

# A consumer requires an artifact of a given kind. The parameter value is the
# artifacts-relative path of a recorded artifact; Knit resolves it, refuses a
# missing artifact or a kind mismatch, and --- with --verify-checksum ---
# re-hashes the bytes and refuses a table that changed since it was produced.
@command "summarize" "Consume a csvfile artifact and count its data rows."
@with_input_artifact "table:csvfile" "Artifacts-relative path of the CSV to read." --verify-checksum
@with_output "rows:integer" "0" "Number of data rows in the consumed table." --result
@with_table
_summarize() {
    # Resolve the recorded artifacts-relative path to the on-disk file.
    local csv rows
    csv="$(knit_input_artifact_path "$(knit_get_parameter table "$@")")"
    rows=$(( $(wc -l < "${csv}") - 1 ))       # one header line; the rest are data
    knit_output "rows" "${rows}"
    printf 'summarize: %s data rows\n' "${rows}"
}
@done

# A consumer requires an artifact of a given kind. The parameter value is the
# artifacts-relative path of a recorded artifact; Knit resolves it, refuses a
# missing artifact or a kind mismatch, and --- with --verify-checksum ---
# re-hashes the bytes and refuses a table that changed since it was produced.
knit_register "summarize" _summarize "Consume a csvfile artifact and count its data rows."
knit_with_input_artifact "table:csvfile" "Artifacts-relative path of the CSV to read." --verify-checksum
knit_with_output "rows:integer" "0" "Number of data rows in the consumed table." --result
knit_with_table
_summarize() {
    # Resolve the recorded artifacts-relative path to the on-disk file.
    local csv rows
    csv="$(knit_input_artifact_path "$(knit_get_parameter table "$@")")"
    rows=$(( $(wc -l < "${csv}") - 1 ))       # one header line; the rest are data
    knit_output "rows" "${rows}"
    printf 'summarize: %s data rows\n' "${rows}"
}
knit_done

Before the body runs, Knit resolves the path to its artifacts row and refuses the run when the value is empty, when no artifact is recorded at that path, or when the recorded artifact’s kind is not the required one — so the body never sees the wrong kind of input. Add --verify-checksum (as here) to also re-hash the bytes and refuse a run when the artifact changed since it was produced; the digest recorded at production time is the reference. The kind must be registered (builtin or via Declare an artifact kind).

Consuming an artifact records a used_by provenance edge from the artifact’s row to the consuming invocation, the mirror of the produced edge the producer left. Together they complete the lineage producer --produced--> artifact --used_by--> consumer, which a single query walks end to end (see Walk an artifact’s full lineage). A plain file input that is not a recorded artifact takes a bare path parameter instead and leaves no edge — reach for knit_with_input_artifact only when the input is a tracked artifact whose lineage you want.

Consume many artifacts with a glob#

Declare a +/ input artifact and read every member with knit_input_artifact_paths, passing a comma list or a glob.*

APIs: knit_with_input_artifact, knit_input_artifact_paths

A * (zero or more) or + (one or more) on an input artifact lets one parameter stand for a whole set. Its argument is a comma-separated list of artifacts-relative paths, and any element holding a glob metacharacter (*, ?, [) is expanded against the artifacts root — so a single --shards 'shard-*.csv' gathers a whole fan-out. knit_input_artifact_paths fills a bash array with the resolved on-disk paths, in order, de-duplicated:

# A "+" (one or more) input requires at least one member. Its argument is a
# comma-separated list of artifacts-relative paths, and any element holding a
# glob metacharacter (*, ?, [) is expanded against the artifacts root --- so one
# `--shards 'shard-*.csv'` gathers the whole fan-out. knit_input_artifact_paths
# fills a bash array with the resolved on-disk paths, in order, de-duplicated.
@command "merge" "Merge every shard matched by a glob into one row count."
@with_input_artifact "shards:csvfile+" "Artifacts-relative glob of the shards to merge (one or more)."
@with_output "rows:integer" "0" "Total data rows across the merged shards." --result
@with_table
_merge() {
    local -a paths=()
    knit_input_artifact_paths paths "$(knit_get_parameter "shards" "$@")"
    local total=0 p
    for p in "${paths[@]}"; do
        total=$(( total + $(wc -l < "${p}") - 1 ))     # drop each shard's header line
    done
    knit_output "rows" "${total}"
    printf 'merged %s shard(s), %s data row(s)\n' "${#paths[@]}" "${total}"
}
@done

# A "+" (one or more) input requires at least one member. Its argument is a
# comma-separated list of artifacts-relative paths, and any element holding a
# glob metacharacter (*, ?, [) is expanded against the artifacts root --- so one
# `--shards 'shard-*.csv'` gathers the whole fan-out. knit_input_artifact_paths
# fills a bash array with the resolved on-disk paths, in order, de-duplicated.
knit_register "merge" _merge "Merge every shard matched by a glob into one row count."
knit_with_input_artifact "shards:csvfile+" "Artifacts-relative glob of the shards to merge (one or more)."
knit_with_output "rows:integer" "0" "Total data rows across the merged shards." --result
knit_with_table
_merge() {
    local -a paths=()
    knit_input_artifact_paths paths "$(knit_get_parameter "shards" "$@")"
    local total=0 p
    for p in "${paths[@]}"; do
        total=$(( total + $(wc -l < "${p}") - 1 ))     # drop each shard's header line
    done
    knit_output "rows" "${total}"
    printf 'merged %s shard(s), %s data row(s)\n' "${#paths[@]}" "${total}"
}
knit_done

Every resolved member is validated before the body runs — containment, existence, and recorded kind (plus the checksum with --verify-checksum) — and the first bad element is fatal and names itself, so the body never sees a partial or wrong-kind set. + refuses an empty result (the parameter is required, and a glob matching nothing is fatal); * accepts it (the parameter is optional and defaults to empty, and the array simply comes back empty). Quote the glob so your shell does not expand it before Knit does. The consumer’s own row stores the raw argument you passed (the pattern), while one used_by edge is recorded per resolved member — so the lineage carries the concrete set, not the pattern. For a single artifact, use the scalar knit_input_artifact_path instead (see Consume an artifact by kind).

Discover and merge a fan-out#

Pair a *-output producer with a +/ -input consumer so one command scatters many artifacts and the next gathers them by glob.*

APIs: knit_with_output_artifact, knit_artifact, knit_with_input_artifact, knit_input_artifact_paths

Variadic artifacts compose into a discover-and-merge pattern across two commands: a producer fans out a * collection, and a consumer gathers the whole set through a + input with one glob argument — without either side hard-coding how many members there are. The producer binds one collection name per member:

# A "*" (zero or more) output is a COLLECTION: the body may bind the same name
# any number of times, and each binding is its own artifacts row with its own
# "produced" edge. (Use "+" for one-or-more, which is fatal if the body binds
# nothing.) Here one run scatters `n` CSV shards under the artifacts root.
@command "shard" "Fan out a range into several CSV shards."
@with_optional "n:integer" "3" "How many shards to write."
@with_output_artifact "shards:csvfile*" "The CSV shards (zero or more)."
@with_table
_shard() {
    local n out i
    n="$(knit_get_parameter "n" "$@")"
    out="$(knit_artifact_dir)"
    mkdir -p "${out}"
    for (( i = 1; i <= n; i++ )); do
        printf 'id,sq\n%d,%d\n' "${i}" "$(( i * i ))" > "${out}/shard-${i}.csv"
        knit_artifact "shards" "shard-${i}.csv"    # bind one member of the collection
    done
    printf 'wrote %s shard(s)\n' "${n}"
}
@done

# A "*" (zero or more) output is a COLLECTION: the body may bind the same name
# any number of times, and each binding is its own artifacts row with its own
# "produced" edge. (Use "+" for one-or-more, which is fatal if the body binds
# nothing.) Here one run scatters `n` CSV shards under the artifacts root.
knit_register "shard" _shard "Fan out a range into several CSV shards."
knit_with_optional "n:integer" "3" "How many shards to write."
knit_with_output_artifact "shards:csvfile*" "The CSV shards (zero or more)."
knit_with_table
_shard() {
    local n out i
    n="$(knit_get_parameter "n" "$@")"
    out="$(knit_artifact_dir)"
    mkdir -p "${out}"
    for (( i = 1; i <= n; i++ )); do
        printf 'id,sq\n%d,%d\n' "${i}" "$(( i * i ))" > "${out}/shard-${i}.csv"
        knit_artifact "shards" "shard-${i}.csv"    # bind one member of the collection
    done
    printf 'wrote %s shard(s)\n' "${n}"
}
knit_done

The consumer discovers them with a glob and merges what it finds:

# A "+" (one or more) input requires at least one member. Its argument is a
# comma-separated list of artifacts-relative paths, and any element holding a
# glob metacharacter (*, ?, [) is expanded against the artifacts root --- so one
# `--shards 'shard-*.csv'` gathers the whole fan-out. knit_input_artifact_paths
# fills a bash array with the resolved on-disk paths, in order, de-duplicated.
@command "merge" "Merge every shard matched by a glob into one row count."
@with_input_artifact "shards:csvfile+" "Artifacts-relative glob of the shards to merge (one or more)."
@with_output "rows:integer" "0" "Total data rows across the merged shards." --result
@with_table
_merge() {
    local -a paths=()
    knit_input_artifact_paths paths "$(knit_get_parameter "shards" "$@")"
    local total=0 p
    for p in "${paths[@]}"; do
        total=$(( total + $(wc -l < "${p}") - 1 ))     # drop each shard's header line
    done
    knit_output "rows" "${total}"
    printf 'merged %s shard(s), %s data row(s)\n' "${#paths[@]}" "${total}"
}
@done

# A "+" (one or more) input requires at least one member. Its argument is a
# comma-separated list of artifacts-relative paths, and any element holding a
# glob metacharacter (*, ?, [) is expanded against the artifacts root --- so one
# `--shards 'shard-*.csv'` gathers the whole fan-out. knit_input_artifact_paths
# fills a bash array with the resolved on-disk paths, in order, de-duplicated.
knit_register "merge" _merge "Merge every shard matched by a glob into one row count."
knit_with_input_artifact "shards:csvfile+" "Artifacts-relative glob of the shards to merge (one or more)."
knit_with_output "rows:integer" "0" "Total data rows across the merged shards." --result
knit_with_table
_merge() {
    local -a paths=()
    knit_input_artifact_paths paths "$(knit_get_parameter "shards" "$@")"
    local total=0 p
    for p in "${paths[@]}"; do
        total=$(( total + $(wc -l < "${p}") - 1 ))     # drop each shard's header line
    done
    knit_output "rows" "${total}"
    printf 'merged %s shard(s), %s data row(s)\n' "${#paths[@]}" "${total}"
}
knit_done

Run them in sequence — shard --n 3 then merge --shards 'shard-*.csv' — and the glob resolves to whatever the fan-out produced. The provenance keeps each member distinct on both sides: the producer leaves one produced edge per bound member, and the consume leaves one used_by edge per resolved member, so the lineage producer --produced--> member --used_by--> consumer holds for every file in the set. One join over the two edge kinds recovers the whole group, and what merged this shard? or which shards fed this merge? are the two ends of that same walk (see Walk an artifact’s full lineage). This scales a sweep without a fixed member count: add more shards and the same glob gathers them.