Provenance

Provenance#

Linking recorded rows across tables through the __provenance__ graph of call and used_by edges.

Understand call and used_by edges#

The __provenance__ table links recorded rows with directed call and used_by edges, so a query can trace what produced what.

APIs: knit_with_setup

Each command records its own rows in its own table (see Record invocations in a table). The provenance graph is what links those rows across tables: a single __provenance__ table whose every row is one directed edge between two invocations, written as source --edge_type--> target. An edge stores the (id, name) pair of its source and target — the id joins to a row, and the name identifies which table that row lives in — so a query can hop from a row in one table to a related row in another.

There are two kinds of edge:

  • call — the source invoked the target. Every time a participating command runs another command, knit records a call edge from caller to callee, and stamps it with the call’s start_time and end_time. This is how a job’s row links to the knit run rows it launched, and those to the app rows the run recorded. A call edge may also carry an alias naming the call site (see Distinguish repeated calls).

  • used_by — the target references a setup that an earlier invocation built. When a job (or other command) declares knit_with_setup (see Depend on a setup), knit records a used_by edge from the setup (the source, the antecedent) to the invocation that uses it (the target). It has no duration, so its timestamps are NULL.

The direction is consistent: the source is always the antecedent (the caller, or the setup that was built first) and the target the dependent (the callee, or the consumer). Which invocations become nodes is governed by participation — see Opt in/out of the provenance graph — and the edges themselves are queried with knit query graph.

Opt in/out of the provenance graph#

Force a command in or out of the provenance graph with knit_with_provenance and knit_without_provenance.

APIs: knit_with_provenance, knit_without_provenance

Alongside data rows, knit records a provenance graph: each participating invocation is a node, and calling one command from another records a “call” edge, so a query can trace which run produced which result. Whether a command participates is orthogonal to whether it records a table (see Record invocations in a table) — a command can do either, both, or neither.

By default participation follows visibility: visible commands participate; hidden commands (knit_hidden, e.g. internal workers) are transparent. Override that default explicitly.

Use knit_without_provenance to keep a visible command out of the graph. A read-only fan-in command that only reads back existing rows is the typical case — it should not add call edges of its own:

@command "aggregate" \
    "Fan-in: total the inside metric across every recorded render."
@without_provenance
_aggregate() {
    # Read back what the renders recorded. Each `knit run -- render` wrote one row
    # in the `render` table (rank 0 only), so one SELECT sees every image the
    # experiment has produced, across every job submission.
    local count total
    count=$(knit query sql --exec "SELECT count(*) FROM render;")
    total=$(knit query sql --exec "SELECT sum(inside) FROM render;")

    printf 'Summed inside=%s over %s render(s).\n' "${total:-0}" "${count:-0}"
}
@done

knit_register "aggregate" _aggregate \
    "Fan-in: total the inside metric across every recorded render."
knit_without_provenance
_aggregate() {
    # Read back what the renders recorded. Each `knit run -- render` wrote one row
    # in the `render` table (rank 0 only), so one SELECT sees every image the
    # experiment has produced, across every job submission.
    local count total
    count=$(knit query sql --exec "SELECT count(*) FROM render;")
    total=$(knit query sql --exec "SELECT sum(inside) FROM render;")

    printf 'Summed inside=%s over %s render(s).\n' "${total:-0}" "${count:-0}"
}
knit_done

Use knit_with_provenance for the opposite: force a command into the graph even when it would otherwise be transparent (for example a hidden command you still want to see as a node).

Either mark propagates to unmarked lexical descendants, so marking a parent command governs its whole colon-nested subtree unless a child overrides it. A transparent command is skipped when a command it invokes resolves its parent: the callee links to the nearest participating ancestor instead.

Distinguish repeated calls#

Name a call with knit_as so repeated invocations of the same command can be told apart in a provenance query.

APIs: knit_as

When one command invokes another more than once, each call is a separate edge in the provenance graph — but by default the edges look alike, so a later query cannot tell one call apart from another. knit_as names a single call: it records an alias on that call’s provenance edge, then runs the command:

knit_as fast run --procs 8 -- mcrank
knit_as slow run --procs 1 -- mcrank

Used as knit_as <alias> <cmd> …, it is equivalent to knit <cmd> … except that the delegated invocation’s call edge carries <alias>. A query can then address each call independently — “the fast run” versus “the slow run” — instead of seeing two indistinguishable edges. Without knit_as a call edge’s alias is NULL.

The alias is one-shot: it lands only on the directly named call’s edge, never on the nested edges that call’s own body records. It is validated at the call site — it must be non-empty, must not be a registered table name (which would collide with a node label in a query), and must not already have been used within the current invocation (two edges sharing an alias would be indistinguishable).

Consume an artifact by kind#

Require a produced artifact of a given kind with knit_with_input_artifact, resolve it with knit_input_artifact_path, and record a used_by edge.

APIs: knit_with_input_artifact, knit_input_artifact_path

A command consumes an artifact another command produced by declaring knit_with_input_artifact "name:kind" "description". This registers a required string parameter whose value is the artifacts-relative path of a recorded artifact of that kind. In the body, knit_input_artifact_path resolves that path to the on-disk file:

# A consumer requires an artifact of a given kind. The parameter value is the
# artifacts-relative path of a recorded artifact; Knit resolves it, refuses a
# missing artifact or a kind mismatch, and --- with --verify-checksum ---
# re-hashes the bytes and refuses a table that changed since it was produced.
@command "summarize" "Consume a csvfile artifact and count its data rows."
@with_input_artifact "table:csvfile" "Artifacts-relative path of the CSV to read." --verify-checksum
@with_output "rows:integer" "0" "Number of data rows in the consumed table." --result
@with_table
_summarize() {
    # Resolve the recorded artifacts-relative path to the on-disk file.
    local csv rows
    csv="$(knit_input_artifact_path "$(knit_get_parameter table "$@")")"
    rows=$(( $(wc -l < "${csv}") - 1 ))       # one header line; the rest are data
    knit_output "rows" "${rows}"
    printf 'summarize: %s data rows\n' "${rows}"
}
@done

# A consumer requires an artifact of a given kind. The parameter value is the
# artifacts-relative path of a recorded artifact; Knit resolves it, refuses a
# missing artifact or a kind mismatch, and --- with --verify-checksum ---
# re-hashes the bytes and refuses a table that changed since it was produced.
knit_register "summarize" _summarize "Consume a csvfile artifact and count its data rows."
knit_with_input_artifact "table:csvfile" "Artifacts-relative path of the CSV to read." --verify-checksum
knit_with_output "rows:integer" "0" "Number of data rows in the consumed table." --result
knit_with_table
_summarize() {
    # Resolve the recorded artifacts-relative path to the on-disk file.
    local csv rows
    csv="$(knit_input_artifact_path "$(knit_get_parameter table "$@")")"
    rows=$(( $(wc -l < "${csv}") - 1 ))       # one header line; the rest are data
    knit_output "rows" "${rows}"
    printf 'summarize: %s data rows\n' "${rows}"
}
knit_done

Before the body runs, Knit resolves the path to its artifacts row and refuses the run when the value is empty, when no artifact is recorded at that path, or when the recorded artifact’s kind is not the required one — so the body never sees the wrong kind of input. Add --verify-checksum (as here) to also re-hash the bytes and refuse a run when the artifact changed since it was produced; the digest recorded at production time is the reference. The kind must be registered (builtin or via Declare an artifact kind).

Consuming an artifact records a used_by provenance edge from the artifact’s row to the consuming invocation, the mirror of the produced edge the producer left. Together they complete the lineage producer --produced--> artifact --used_by--> consumer, which a single query walks end to end (see Walk an artifact’s full lineage). A plain file input that is not a recorded artifact takes a bare path parameter instead and leaves no edge — reach for knit_with_input_artifact only when the input is a tracked artifact whose lineage you want.

Discover and merge a fan-out#

Pair a *-output producer with a +/ -input consumer so one command scatters many artifacts and the next gathers them by glob.*

APIs: knit_with_output_artifact, knit_artifact, knit_with_input_artifact, knit_input_artifact_paths

Variadic artifacts compose into a discover-and-merge pattern across two commands: a producer fans out a * collection, and a consumer gathers the whole set through a + input with one glob argument — without either side hard-coding how many members there are. The producer binds one collection name per member:

# A "*" (zero or more) output is a COLLECTION: the body may bind the same name
# any number of times, and each binding is its own artifacts row with its own
# "produced" edge. (Use "+" for one-or-more, which is fatal if the body binds
# nothing.) Here one run scatters `n` CSV shards under the artifacts root.
@command "shard" "Fan out a range into several CSV shards."
@with_optional "n:integer" "3" "How many shards to write."
@with_output_artifact "shards:csvfile*" "The CSV shards (zero or more)."
@with_table
_shard() {
    local n out i
    n="$(knit_get_parameter "n" "$@")"
    out="$(knit_artifact_dir)"
    mkdir -p "${out}"
    for (( i = 1; i <= n; i++ )); do
        printf 'id,sq\n%d,%d\n' "${i}" "$(( i * i ))" > "${out}/shard-${i}.csv"
        knit_artifact "shards" "shard-${i}.csv"    # bind one member of the collection
    done
    printf 'wrote %s shard(s)\n' "${n}"
}
@done

# A "*" (zero or more) output is a COLLECTION: the body may bind the same name
# any number of times, and each binding is its own artifacts row with its own
# "produced" edge. (Use "+" for one-or-more, which is fatal if the body binds
# nothing.) Here one run scatters `n` CSV shards under the artifacts root.
knit_register "shard" _shard "Fan out a range into several CSV shards."
knit_with_optional "n:integer" "3" "How many shards to write."
knit_with_output_artifact "shards:csvfile*" "The CSV shards (zero or more)."
knit_with_table
_shard() {
    local n out i
    n="$(knit_get_parameter "n" "$@")"
    out="$(knit_artifact_dir)"
    mkdir -p "${out}"
    for (( i = 1; i <= n; i++ )); do
        printf 'id,sq\n%d,%d\n' "${i}" "$(( i * i ))" > "${out}/shard-${i}.csv"
        knit_artifact "shards" "shard-${i}.csv"    # bind one member of the collection
    done
    printf 'wrote %s shard(s)\n' "${n}"
}
knit_done

The consumer discovers them with a glob and merges what it finds:

# A "+" (one or more) input requires at least one member. Its argument is a
# comma-separated list of artifacts-relative paths, and any element holding a
# glob metacharacter (*, ?, [) is expanded against the artifacts root --- so one
# `--shards 'shard-*.csv'` gathers the whole fan-out. knit_input_artifact_paths
# fills a bash array with the resolved on-disk paths, in order, de-duplicated.
@command "merge" "Merge every shard matched by a glob into one row count."
@with_input_artifact "shards:csvfile+" "Artifacts-relative glob of the shards to merge (one or more)."
@with_output "rows:integer" "0" "Total data rows across the merged shards." --result
@with_table
_merge() {
    local -a paths=()
    knit_input_artifact_paths paths "$(knit_get_parameter "shards" "$@")"
    local total=0 p
    for p in "${paths[@]}"; do
        total=$(( total + $(wc -l < "${p}") - 1 ))     # drop each shard's header line
    done
    knit_output "rows" "${total}"
    printf 'merged %s shard(s), %s data row(s)\n' "${#paths[@]}" "${total}"
}
@done

# A "+" (one or more) input requires at least one member. Its argument is a
# comma-separated list of artifacts-relative paths, and any element holding a
# glob metacharacter (*, ?, [) is expanded against the artifacts root --- so one
# `--shards 'shard-*.csv'` gathers the whole fan-out. knit_input_artifact_paths
# fills a bash array with the resolved on-disk paths, in order, de-duplicated.
knit_register "merge" _merge "Merge every shard matched by a glob into one row count."
knit_with_input_artifact "shards:csvfile+" "Artifacts-relative glob of the shards to merge (one or more)."
knit_with_output "rows:integer" "0" "Total data rows across the merged shards." --result
knit_with_table
_merge() {
    local -a paths=()
    knit_input_artifact_paths paths "$(knit_get_parameter "shards" "$@")"
    local total=0 p
    for p in "${paths[@]}"; do
        total=$(( total + $(wc -l < "${p}") - 1 ))     # drop each shard's header line
    done
    knit_output "rows" "${total}"
    printf 'merged %s shard(s), %s data row(s)\n' "${#paths[@]}" "${total}"
}
knit_done

Run them in sequence — shard --n 3 then merge --shards 'shard-*.csv' — and the glob resolves to whatever the fan-out produced. The provenance keeps each member distinct on both sides: the producer leaves one produced edge per bound member, and the consume leaves one used_by edge per resolved member, so the lineage producer --produced--> member --used_by--> consumer holds for every file in the set. One join over the two edge kinds recovers the whole group, and what merged this shard? or which shards fed this merge? are the two ends of that same walk (see Walk an artifact’s full lineage). This scales a sweep without a fixed member count: add more shards and the same glob gathers them.