Cleanup

Erasing recorded entities and their provenance with knit remove, from a single row to a whole lineage.

Remove an entity and its dependents

Erase a setup, resource, job, run, or command by name, type, or id, cascading down the provenance graph to everything that used it.

APIs: remove:setup, remove:resource, remove:job, remove:run, remove:command, remove:artifact

knit remove erases a recorded entity from the database together with the rows, provenance edges, and on-disk directories that depend on it — the safe, provenance-aware alternative to leaving stale setups and runs to accumulate forever. There is one subcommand per kind (remove setup, remove resource, remove job, remove run, remove app, remove command, remove artifact), and each takes exactly one selector:

• --id <id> — one row by its recorded id;

• --name <name> — a setup / resource / job instance by its instance name;

• --type <type> — every setup / resource / job of a type at once;

• --group <group> — every job in a submission group (remove job only);

• --path <path> — an artifact by its artifacts-relative path (remove artifact only).

$ ./exp.sh remove setup --name buildenv
$ ./exp.sh remove job --id "$id"
$ ./exp.sh remove resource --type dataset

By default remove cascades downward: it follows call and produced edges from the selection, and used_by edges only outward from a provider. So removing a provider (a setup or a resource) also erases every consumer that used it — the jobs, their runs, and their artifacts. Removing a consumer (a job) does the opposite: the setup and resource it used are left in place, and only the used_by edge into the erased job is dropped.

remove prints an itemized report of exactly what it will erase and asks for confirmation. Pass --yes to skip the prompt (the report is still printed) so remove is usable from a script; a non-interactive shell without --yes declines safely rather than deleting. See Preview a removal with –dry-run to inspect the blast radius first.

Two guards keep the graph consistent. Selecting a callee whose caller is kept (an artifact on its own, a run whose job stays) is refused with a hint to remove the caller instead or pass --from-root (see Erase a whole lineage with –from-root). And a job that has not finished (submitted / running / prepared) in the erase set is a hard refusal — stop it first with job cancel.

Preview a removal with –dry-run

Print the exact rows, edges, and files a remove would erase without deleting anything or prompting.

APIs: remove:setup, remove:job

A cascade can reach further than expected — removing one provider may erase every job that used it. --dry-run computes and prints the full erase set, then exits without prompting and without deleting, so you can inspect the blast radius before committing:

$ ./exp.sh remove setup --type env --dry-run
The following will be permanently erased:

  Data rows (4):
    setup    env (buildenv)       01a0...
    job                           01a0...  (state: completed)
    job      crunch               01a0...  (body row)
    artifact result.txt           01a0...

  Provenance edges (6):
    fetch:srcpkg 01a0... --used_by--> setup:env 01a0...
    ...

  Directories and artifacts removed (3):
    setups/buildenv
    jobs/01a0...
    artifacts/result.txt

The report has the same three sections remove always prints — the data rows (setup rows show type (name)), the provenance edges, and the directories and artifact files — plus a Left on disk section when a file is deliberately kept (see Keep the files while pruning provenance). The report is exactly what the confirmation prompt would show; --dry-run just stops before the prompt.

--dry-run changes neither the database nor the filesystem, so it is the way to check what a --type or --from-root selection actually pulls in before running it for real.

Erase a whole lineage with –from-root

Point at any row (or an artifact) and erase the entire call/produced tree it belongs to, both up to the root and back down.

APIs: remove:artifact, remove:job

By default remove only cascades downward and refuses to erase a row whose caller or producer is kept — so naming an artifact on its own fails, because its producing invocation would be left dangling:

$ ./exp.sh remove artifact --path result.txt
[knit:fatal] remove: cannot erase artifact ...; it was produced by
"submit:crunch" ..., which is not being erased. Pass --from-root to erase the
whole lineage.

--from-root widens the erase set to the whole call/produced lineage tree the selection belongs to. remove first walks up call and produced edges to the root of the tree, then removes the entire tree downward — so pointing at any row, or at an artifact, erases everything in its lineage:

$ ./exp.sh remove artifact --path result.txt --from-root

This is the tool for pruning a complete experiment branch: the producing command, its parent submission, sibling outputs, and every artifact in the tree all go together. used_by edges are never followed either way, so a setup or resource that the tree used is still left intact — --from-root widens along the lineage, not along shared providers. Combine it with --dry-run first to see the full tree before committing.

Keep the artifacts while pruning provenance

Erase the database rows and edges of a removal but leave the artifact files on disk, listed under “Left on disk”.

APIs: remove:job, remove:artifact

Sometimes you want a run out of the database but its output files kept — to archive them by hand, or hand them off before pruning the provenance. --keep-artifacts does exactly that: it erases the database side as usual (every row and provenance edge in the set, including each artifact’s artifacts row and its produced edge) but does not delete the on-disk artifact entries under artifacts/:

$ ./exp.sh remove job --id "$id" --keep-artifacts --yes
Erased:
  ...
The following files/directories were NOT removed:
  artifacts/result.txt   (artifact, --keep-artifacts)

The kept entries are listed in a Left on disk section so the record of what survived is explicit. Scope is artifacts only: job, setup, and resource directories are framework-managed containers and are still removed (keeping them would just orphan them). To keep the whole on-disk tree instead — directories included — use --keep-files (see the next recipe). Plain (non-artifact) command outputs are always left on disk and listed there too, with or without the flag — remove never deletes a file you wrote outside the artifacts/ root.

Because the artifact rows are gone, the kept files are no longer tracked: they will not be re-checksummed, exported, or found by a provenance walk. --keep-artifacts is about salvaging bytes while discarding the record, not about keeping a partial record.

Remove the records only, keep every file

Erase the database rows and edges of a removal while making no filesystem change at all — every directory, artifact, and plain output stays.

APIs: remove:job, remove:artifact

When you want the provenance out of the database but the whole on-disk tree left exactly as it is, use --keep-files. It erases the database side as usual (every row and provenance edge in the set) but makes no filesystem change: job, setup, and resource directories, artifact entries, and plain outputs all stay:

$ ./exp.sh remove job --id "$id" --keep-files --yes
Erased:
  ...
The following files/directories were NOT removed:
  jobs/018f9c3a-...      (job directory, --keep-files)
  artifacts/result.txt   (artifact, --keep-files)

Everything that stays is listed under Left on disk, tagged by kind, so the record of what survived is explicit. This is the superset of --keep-artifacts: that flag keeps only the artifact entries and still removes the directories, while --keep-files keeps the directories too. If you pass both, --keep-files wins.

Because the rows and edges are gone, none of the kept files are tracked any more: they will not be re-checksummed, exported, or found by a provenance walk. --keep-files discards the record while leaving the bytes untouched.