Recording

Recording#

Recording command invocations, their parameters, and their outputs in per-command tables.

Record invocations in a table#

Declare a table with knit_with_table so knit records one row per invocation, with a column per parameter and output.

APIs: knit_with_table

Add knit_with_table between knit_register and knit_done and knit records one row per invocation of the command in a SQLite table:

@command "add" "Add two integers and record the run."
@with_required "x:integer" "First value."
@with_required "y:integer" "Second value."
@with_output "total:integer" "0" "x + y."
@with_table
_add() {
    local x y
    x="$(knit_get_parameter "x" "$@")"
    y="$(knit_get_parameter "y" "$@")"
    knit_output "total" "$((x + y))"
    printf 'total=%s\n' "$((x + y))"
}
@done

knit_register "add" _add "Add two integers and record the run."
knit_with_required "x:integer" "First value."
knit_with_required "y:integer" "Second value."
knit_with_output "total:integer" "0" "x + y."
knit_with_table
_add() {
    local x y
    x="$(knit_get_parameter "x" "$@")"
    y="$(knit_get_parameter "y" "$@")"
    knit_output "total" "$((x + y))"
    printf 'total=%s\n' "$((x + y))"
}
knit_done

The table’s columns are derived from the command’s declaration: an id (a uuid) first, then its required parameters, optional parameters, flags, and outputs — here id, x, y, and total. The schema is created (and migrated if the declaration later changes) automatically when the experiment loads, so you never write CREATE TABLE yourself.

The table name defaults to the command’s name (colon-joined for a subcommand, e.g. foo:bar); pass an explicit name to override it, as in knit_with_table "my_runs". Two commands cannot claim the same table.

The row is written automatically after the body returns: knit fills the parameter columns from the invocation’s arguments and the output columns from whatever the body emitted with knit_output (see Emit outputs from a command). Recording needs a bootstrapped experiment — before bootstrap it is a no-op — and each invocation is recorded once. Read the rows back later with knit query sql; invoking add --x 2 --y 3 leaves a row whose x, y, and total are 2, 3, and 5.

Emit outputs from a command#

Declare a result column with knit_with_output and set it from the body with knit_output.

APIs: knit_with_output, knit_output

A command’s parameters are its inputs; its outputs are the results you want recorded alongside them. Declare an output at registration with knit_with_output name:type default description, then set its value from the body with knit_output name value:

@command "scale" "Multiply an integer by a factor."
@with_required "value:integer" "The value to scale."
@with_optional "factor:integer" "2" "The multiplier (defaults to 2)."
@with_output "result:integer" "0" "value * factor."
_scale() {
    local value factor
    value="$(knit_get_parameter "value" "$@")"
    factor="$(knit_get_parameter "factor" "$@")"
    knit_output "result" "$((value * factor))"
    printf 'result=%s\n' "$((value * factor))"
}
@done

knit_register "scale" _scale "Multiply an integer by a factor."
knit_with_required "value:integer" "The value to scale."
knit_with_optional "factor:integer" "2" "The multiplier (defaults to 2)."
knit_with_output "result:integer" "0" "value * factor."
_scale() {
    local value factor
    value="$(knit_get_parameter "value" "$@")"
    factor="$(knit_get_parameter "factor" "$@")"
    knit_output "result" "$((value * factor))"
    printf 'result=%s\n' "$((value * factor))"
}
knit_done

Outputs are typed exactly like parameters (see Type-annotate a parameter): a value that does not match its declared type is a fatal error. knit_output may only set an output the command declared, and only from within the command’s own function.

On its own, scale just computes result and prints it — an emitted output is not persisted until the command has somewhere to record it. Give the command a table and each declared output becomes one of its columns, filled from the last knit_output value (see Record invocations in a table; the add command there records its total output this way).

Under knit run, only rank 0 records, so a single knit_output writes one value no matter how many ranks executed the body — you do not need to guard it with a rank check (see Register an MPI app).

Checksum a file or directory parameter#

Record the path and a sha256 content checksum of a file/directory input or output, and opt out with –no-checksum.

APIs: knit_with_required, knit_with_optional, knit_with_output, knit_output, knit_with_table

A parameter or output typed file or directory (alias dir) is more than a string: knit checks it exists at runtime and, by default, fingerprints its content with a sha256. Declare them as usual — no extra call is needed:

# A file/directory parameter is checked for existence and, unless declared
# --no-checksum, fingerprinted: knit records the path AND a sha256 of the content
# in a companion <name>_checksum column. An input is hashed before the body runs
# (the digest reflects the artifact as consumed); an output after it returns.
@command "report" "Summarize a data file into a report."
@with_table
@with_required "input:file" "The data file to summarize."
@with_output "summary:file" "" "The written summary (path + sha256 recorded)."
@with_output "workdir:directory" "" "Scratch tree, recorded by path only." --no-checksum
_report() {
    local input
    input=$(knit_get_parameter "input" "$@")
    mkdir -p work
    wc -l < "${input}" > work/lines.txt
    knit_output "workdir" "work"
    printf 'lines=%s\n' "$(cat work/lines.txt)" > summary.txt
    knit_output "summary" "summary.txt"
}
@done

# A file/directory parameter is checked for existence and, unless declared
# --no-checksum, fingerprinted: knit records the path AND a sha256 of the content
# in a companion <name>_checksum column. An input is hashed before the body runs
# (the digest reflects the artifact as consumed); an output after it returns.
knit_register "report" _report "Summarize a data file into a report."
knit_with_table
knit_with_required "input:file" "The data file to summarize."
knit_with_output "summary:file" "" "The written summary (path + sha256 recorded)."
knit_with_output "workdir:directory" "" "Scratch tree, recorded by path only." --no-checksum
_report() {
    local input
    input=$(knit_get_parameter "input" "$@")
    mkdir -p work
    wc -l < "${input}" > work/lines.txt
    knit_output "workdir" "work"
    printf 'lines=%s\n' "$(cat work/lines.txt)" > summary.txt
    knit_output "summary" "summary.txt"
}
knit_done

With knit_with_table in place, each such parameter records two columns: the path (input, summary, workdir) and, next to it, a companion <name>_checksum holding sha256:<hex>. So report --input data.txt leaves a row whose input is data.txt and whose input_checksum is the digest of that file’s bytes.

The digest is computed off the timed path and reflects the artifact as used: an input is hashed before the body runs, an output after it returns. A directory is hashed recursively over its structure and contents. Existence is enforced by direction: a missing required input is fatal before the body runs; a declared output missing on a successful completion is an error.

Append --no-checksum to record the path only, skipping the hash — useful for a large or volatile artifact whose content you do not want to fingerprint (here the workdir scratch tree). Such a declaration has no <name>_checksum column, but existence is still checked. --no-checksum on a non-file/directory type is a declaration error.

How knit types map to SQL columns#

A recorded column’s SQL affinity follows its knit type — integer to INTEGER, real to REAL, everything else to TEXT.

APIs: knit_with_table, knit_with_output

When a command declares a table (see Record invocations in a table), knit derives each column’s SQL type from the knit type of the parameter, flag, or output it records. The mapping is:

  • integer (alias int) → INTEGER

  • real (aliases float, double) → REAL

  • everything else → TEXT — including string, boolean, path, file, filename, date, time, datetime, uuid, and any enum you define.

Only integer and real get a numeric affinity, so numeric comparison and aggregation (SUM, AVG, ORDER BY) work as expected on those columns. Everything else is stored verbatim as text — a boolean is the string true or false, and a date/datetime is its literal string — so compare those as text or with SQLite’s date functions.

The id column knit adds first is always TEXT (a uuid). Because the mapping is by type, annotating a parameter or output well (see Type-annotate a parameter) is what gives you a well-typed column: the add command in Record invocations in a table declares x, y, and a total output all as integer, so those three columns are INTEGER and their sums are numeric.

Mark an output as the result#

Flag the output that is what the experiment was for with –result, so knit describe highlights it.

APIs: knit_with_output, knit_output

A command may record several outputs, but usually one of them is the result — what the experiment was for. Append --result to its knit_with_output declaration to say so:

@command "measure" "Square a value and mark the result."
@with_required "x:integer" "The value to square."
@with_output   "square:integer" "0" "The squared value (the result)." --result
@with_output   "note:string"    ""  "An intermediate note (not a result)."
@with_table
_measure() {
    local x
    x="$(knit_get_parameter "x" "$@")"
    knit_output "square" "$(( x * x ))"        # the headline value result
    knit_output "note"   "squared x=${x}"      # recorded, but not a result
    printf 'square=%s\n' "$(( x * x ))"
}
@done

knit_register "measure" _measure "Square a value and mark the result."
knit_with_required "x:integer" "The value to square."
knit_with_output   "square:integer" "0" "The squared value (the result)." --result
knit_with_output   "note:string"    ""  "An intermediate note (not a result)."
knit_with_table
_measure() {
    local x
    x="$(knit_get_parameter "x" "$@")"
    knit_output "square" "$(( x * x ))"        # the headline value result
    knit_output "note"   "squared x=${x}"      # recorded, but not a result
    printf 'square=%s\n' "$(( x * x ))"
}
knit_done

--result is orthogonal to the output’s type and to recording: it is valid on any output (a scalar such as square here, or a file / directory), and it changes no value and moves no file. It is a declaration-time marker only, so knit describe can surface it: the flagged output carries a result tag in the human and Markdown views and a "result": true field in the JSON and YAML views (see Describe the command tree). An unflagged output such as note reports result false.

Set the value from the body with knit_output exactly as for any output (see Emit outputs from a command); the flag lives on the declaration, not the emission. For a file or directory you also want packaged for export, declare it with knit_with_output_artifact ... --result instead (see Declare and bind an artifact).

Find and erase failed invocations#

Record each command’s exit status in __exit_status__, query which invocations failed, and prune them all at once with remove –failed.

APIs: knit_with_table, knit_no_record_on_failure, remove

Every recorded command also records its exit status. A command that declares a table (knit_with_table) and is not marked knit_no_record_on_failure gets a reserved __exit_status__ column, written on every invocation: 0 on success, the non-zero code on failure. A failed command keeps its row — and any output or artifact it produced before failing — so a failure deep in a chain stays queryable instead of vanishing:

# A command that records a row and may fail. Because it declares a table and is
# NOT @no_record_on_failure, knit adds a reserved __exit_status__ column and
# records it on every invocation: 0 on success, the non-zero code on failure. A
# failed row stays in the database (with any output it produced) so you can query
# what failed, and prune it later with `remove --failed`.
@command "boom" "Fail when --code is non-zero, so a failed row is recorded."
@with_table
@with_optional "code:integer" "0" "Exit code to return."
_boom() {
    return "$(knit_get_parameter code "$@")"
}
@done

# A command that records a row and may fail. Because it declares a table and is
# NOT @no_record_on_failure, knit adds a reserved __exit_status__ column and
# records it on every invocation: 0 on success, the non-zero code on failure. A
# failed row stays in the database (with any output it produced) so you can query
# what failed, and prune it later with `remove --failed`.
knit_register "boom" _boom "Fail when --code is non-zero, so a failed row is recorded."
knit_with_table
knit_with_optional "code:integer" "0" "Exit code to return."
_boom() {
    return "$(knit_get_parameter code "$@")"
}
knit_done

Because the row survives, you can ask the database which invocation failed, and with which parameters — exactly what you need to diagnose it:

$ ./exp.sh query sql --format column --header --exec \
    "SELECT id, code, __exit_status__ FROM boom WHERE __exit_status__ <> 0"

__exit_status__ is an ordinary column: it shows up in knit db show, knit query, and the knit describe schema view like any other. NULL (an outcome not yet known) and a row back-filled by migration are both “unknown” and count as neither success nor failure.

To clean up, remove --failed selects every invocation with a non-zero exit status at once, across every table that carries the column:

$ ./exp.sh remove --failed --dry-run              # preview the failed set
$ ./exp.sh remove --failed --yes                  # erase the failed rows
$ ./exp.sh remove --failed --from-root --yes      # erase the jobs that contain them too

--failed uses the same closure and guards as every remove subcommand. On its own it erases the failed rows, but a failed callee whose caller is kept is refused — a failed run inside a job that stays, for instance — with a hint to pass --from-root, which widens to the whole lineage so the enclosing job is erased too (see Erase a whole lineage with –from-root). There is no per-code selector: --failed means any non-zero code. To remove by a specific code, write the SELECT yourself and pass the ids to remove <kind> --id.

Scope it to one kind#

--failed is also a filter on each row-kind subcommand, where it composes with the selector. On its own it means every failed one of that kind; with a selector it narrows to the failed rows the selector chose (an AND):

$ ./exp.sh remove command --failed                 # every failed plain command
$ ./exp.sh remove setup --failed --type buildenv   # failed buildenv setups only
$ ./exp.sh remove run --failed --from-root --yes    # failed runs and their jobs

So failures are prunable one kind at a time instead of all at once. A failed setup / resource / command is a row of that kind with a non-zero status; a failed job or run is one whose body returned non-zero (a job that tolerated a failed run inside it did itself succeed, so it is not a failed job — reach the run with remove run --failed). A selector that matches nothing is still an error, but a selection that simply contains no failures is a quiet “nothing to erase”. remove artifact has no --failed: an artifact is produced, not invoked, so it has no exit status.

A command marked knit_no_record_on_failure is the exception: it records no row on failure, so it has no __exit_status__ column and never appears in the --failed set.