Cleanup

Erasing recorded entities and their provenance with knit remove, from a single row to a whole lineage.

Remove an entity and its dependents

Erase a setup, resource, job, run, or command by name, type, or id, cascading down the provenance graph to everything that used it.

APIs: remove:setup, remove:resource, remove:job, remove:run, remove:command, remove:artifact

knit remove erases a recorded entity from the database together with the rows, provenance edges, and on-disk directories that depend on it — the safe, provenance-aware alternative to leaving stale setups and runs to accumulate forever. There is one subcommand per kind (remove setup, remove resource, remove job, remove run, remove command, remove artifact), and each takes exactly one selector:

• --id <id> — one row by its recorded id (remove run --id also accepts the id of the app row that run launched);

• --name <name> — a setup / resource / job instance by its instance name;

• --type <type> — every setup / resource / job of a type, every run of an app, or every invocation of a command, at once;

• --group <group> — every job in a submission group (remove job only);

• --path <path> — an artifact by its artifacts-relative path (remove artifact only).

$ ./exp.sh remove setup --name buildenv
$ ./exp.sh remove job --id "$id"
$ ./exp.sh remove resource --type dataset

By default remove cascades downward: it follows call and produced edges from the selection, and used_by edges only outward from a provider. So removing a provider (a setup or a resource) also erases every consumer that used it — the jobs, their runs, and their artifacts. Removing a consumer (a job) does the opposite: the setup and resource it used are left in place, and only the used_by edge into the erased job is dropped.

remove prints an itemized report of exactly what it will erase and asks for confirmation. Pass --yes to skip the prompt (the report is still printed) so remove is usable from a script; a non-interactive shell without --yes declines safely rather than deleting. See Preview a removal with –dry-run to inspect the blast radius first.

Two guards keep the graph consistent. Selecting a callee whose caller is kept (an artifact on its own, a run whose job stays) is refused with a hint to remove the caller instead or pass --from-root (see Erase a whole lineage with –from-root). And a job that has not finished (submitted / running / prepared) in the erase set is a hard refusal — stop it first with job cancel.

Preview a removal with –dry-run

Print the exact rows, edges, and files a remove would erase without deleting anything or prompting.

APIs: remove:setup, remove:job

A cascade can reach further than expected — removing one provider may erase every job that used it. --dry-run computes and prints the full erase set, then exits without prompting and without deleting, so you can inspect the blast radius before committing:

$ ./exp.sh remove setup --type env --dry-run
The following will be permanently erased:

  Data rows (4):
    setup    env (buildenv)       01a0...
    job                           01a0...  (state: completed)
    job      crunch               01a0...  (body row)
    artifact result.txt           01a0...

  Provenance edges (6):
    fetch:srcpkg 01a0... --used_by--> setup:env 01a0...
    ...

  Directories and artifacts removed (3):
    setups/buildenv
    jobs/01a0...
    artifacts/result.txt

The report has the same three sections remove always prints — the data rows (setup rows show type (name)), the provenance edges, and the directories and artifact files — plus a Left on disk section when a file is deliberately kept (see Keep the files while pruning provenance). The report is exactly what the confirmation prompt would show; --dry-run just stops before the prompt.

--dry-run changes neither the database nor the filesystem, so it is the way to check what a --type or --from-root selection actually pulls in before running it for real.

Erase a whole lineage with –from-root

Point at any row (or an artifact) and erase the entire call/produced tree it belongs to, both up to the root and back down.

APIs: remove:artifact, remove:job

By default remove only cascades downward and refuses to erase a row whose caller or producer is kept — so naming an artifact on its own fails, because its producing invocation would be left dangling:

$ ./exp.sh remove artifact --path result.txt
[knit:fatal] remove: cannot erase artifact ...; it was produced by
"submit:crunch" ..., which is not being erased. Pass --from-root to erase the
whole lineage.

--from-root widens the erase set to the whole call/produced lineage tree the selection belongs to. remove first walks up call and produced edges to the root of the tree, then removes the entire tree downward — so pointing at any row, or at an artifact, erases everything in its lineage:

$ ./exp.sh remove artifact --path result.txt --from-root

This is the tool for pruning a complete experiment branch: the producing command, its parent submission, sibling outputs, and every artifact in the tree all go together. used_by edges are never followed either way, so a setup or resource that the tree used is still left intact — --from-root widens along the lineage, not along shared providers. Combine it with --dry-run first to see the full tree before committing.

Keep the artifacts while pruning provenance

Erase the database rows and edges of a removal but leave the artifact files on disk, listed under “Left on disk”.

APIs: remove:job, remove:artifact

Sometimes you want a run out of the database but its output files kept — to archive them by hand, or hand them off before pruning the provenance. --keep-artifacts does exactly that: it erases the database side as usual (every row and provenance edge in the set, including each artifact’s artifacts row and its produced edge) but does not delete the on-disk artifact entries under artifacts/:

$ ./exp.sh remove job --id "$id" --keep-artifacts --yes
Erased:
  ...
The following files/directories were NOT removed:
  artifacts/result.txt   (artifact, --keep-artifacts)

The kept entries are listed in a Left on disk section so the record of what survived is explicit. Scope is artifacts only: job, setup, and resource directories are framework-managed containers and are still removed (keeping them would just orphan them). To keep the whole on-disk tree instead — directories included — use --keep-files (see the next recipe). Plain (non-artifact) command outputs are always left on disk and listed there too, with or without the flag — remove never deletes a file you wrote outside the artifacts/ root.

Because the artifact rows are gone, the kept files are no longer tracked: they will not be re-checksummed, exported, or found by a provenance walk. --keep-artifacts is about salvaging bytes while discarding the record, not about keeping a partial record.

Remove the records only, keep every file

Erase the database rows and edges of a removal while making no filesystem change at all — every directory, artifact, and plain output stays.

APIs: remove:job, remove:artifact

When you want the provenance out of the database but the whole on-disk tree left exactly as it is, use --keep-files. It erases the database side as usual (every row and provenance edge in the set) but makes no filesystem change: job, setup, and resource directories, artifact entries, and plain outputs all stay:

$ ./exp.sh remove job --id "$id" --keep-files --yes
Erased:
  ...
The following files/directories were NOT removed:
  jobs/018f9c3a-...      (job directory, --keep-files)
  artifacts/result.txt   (artifact, --keep-files)

Everything that stays is listed under Left on disk, tagged by kind, so the record of what survived is explicit. This is the superset of --keep-artifacts: that flag keeps only the artifact entries and still removes the directories, while --keep-files keeps the directories too. If you pass both, --keep-files wins.

Because the rows and edges are gone, none of the kept files are tracked any more: they will not be re-checksummed, exported, or found by a provenance walk. --keep-files discards the record while leaving the bytes untouched.

Find and erase failed invocations

Record each command’s exit status in __exit_status__, query which invocations failed, and prune them all at once with remove –failed.

APIs: knit_with_table, knit_no_record_on_failure, remove

Every recorded command also records its exit status. A command that declares a table (knit_with_table) and is not marked knit_no_record_on_failure gets a reserved __exit_status__ column, written on every invocation: 0 on success, the non-zero code on failure. A failed command keeps its row — and any output or artifact it produced before failing — so a failure deep in a chain stays queryable instead of vanishing:

Shorthand

# A command that records a row and may fail. Because it declares a table and is
# NOT @no_record_on_failure, knit adds a reserved __exit_status__ column and
# records it on every invocation: 0 on success, the non-zero code on failure. A
# failed row stays in the database (with any output it produced) so you can query
# what failed, and prune it later with `remove --failed`.
@command "boom" "Fail when --code is non-zero, so a failed row is recorded."
@with_table
@with_optional "code:integer" "0" "Exit code to return."
_boom() {
    return "$(knit_get_parameter code "$@")"
}
@done

Long form

# A command that records a row and may fail. Because it declares a table and is
# NOT @no_record_on_failure, knit adds a reserved __exit_status__ column and
# records it on every invocation: 0 on success, the non-zero code on failure. A
# failed row stays in the database (with any output it produced) so you can query
# what failed, and prune it later with `remove --failed`.
knit_register "boom" _boom "Fail when --code is non-zero, so a failed row is recorded."
knit_with_table
knit_with_optional "code:integer" "0" "Exit code to return."
_boom() {
    return "$(knit_get_parameter code "$@")"
}
knit_done

Because the row survives, you can ask the database which invocation failed, and with which parameters — exactly what you need to diagnose it:

$ ./exp.sh query sql --format column --header --exec \
    "SELECT id, code, __exit_status__ FROM boom WHERE __exit_status__ <> 0"

__exit_status__ is an ordinary column: it shows up in knit db show, knit query, and the knit describe schema view like any other. NULL (an outcome not yet known) and a row back-filled by migration are both “unknown” and count as neither success nor failure.

To clean up, remove --failed selects every invocation with a non-zero exit status at once, across every table that carries the column:

$ ./exp.sh remove --failed --dry-run              # preview the failed set
$ ./exp.sh remove --failed --yes                  # erase the failed rows
$ ./exp.sh remove --failed --from-root --yes      # erase the jobs that contain them too

--failed uses the same closure and guards as every remove subcommand. On its own it erases the failed rows, but a failed callee whose caller is kept is refused — a failed run inside a job that stays, for instance — with a hint to pass --from-root, which widens to the whole lineage so the enclosing job is erased too (see Erase a whole lineage with –from-root). There is no per-code selector: --failed means any non-zero code. To remove by a specific code, write the SELECT yourself and pass the ids to remove <kind> --id.

Scope it to one kind

--failed is also a filter on each row-kind subcommand, where it composes with the selector. On its own it means every failed one of that kind; with a selector it narrows to the failed rows the selector chose (an AND):

$ ./exp.sh remove command --failed                 # every failed plain command
$ ./exp.sh remove setup --failed --type buildenv   # failed buildenv setups only
$ ./exp.sh remove run --failed --from-root --yes    # failed runs and their jobs

So failures are prunable one kind at a time instead of all at once. A failed setup / resource / command is a row of that kind with a non-zero status; a failed job or run is one whose body returned non-zero (a job that tolerated a failed run inside it did itself succeed, so it is not a failed job — reach the run with remove run --failed). A selector that matches nothing is still an error, but a selection that simply contains no failures is a quiet “nothing to erase”. remove artifact has no --failed: an artifact is produced, not invoked, so it has no exit status.

A command marked knit_no_record_on_failure is the exception: it records no row on failure, so it has no __exit_status__ column and never appears in the --failed set.