Bundle & export

Packing an experiment into one shippable archive with knit bundle, and exporting a standard RO-Crate manifest of its provenance.

Make a shippable bundle

Pack the whole experiment into one relocatable archive with knit bundle, ready to send to a collaborator or deposit in a repository.

APIs: bundle

knit bundle packs the current experiment into a single archive that unpacks anywhere. Every path inside is relative to the experiment script at the archive root, and symlinks that point outside the tree are dereferenced, so the tree keeps working after a move — the archive is self-contained.

$ ./exp.sh bundle
[knit:info] Wrote bundle to montecarlo-pi-bundle.tar.gz

The default output is ./<project>-bundle.tar.gz (the project name from the metadata table, falling back to the script name without .sh). Choose the path with --output, and write a zip instead of a tarball with --zip (what Zenodo and WorkflowHub commonly expect):

$ ./exp.sh bundle --output /tmp/pi.zip --zip

By default the archive carries the code (the script and knit.sh), the provenance database (a pruned .knit/knit.db), the job logs and scripts, the setup manifests, and the declared artifacts — everything needed to read what happened and re-run it. It leaves out what is bulky and regenerable (the built Spack tree, the provisioned toolchain, fetched resources), recording how to rebuild them instead. Drop a default group with --no-knit, --no-db, --no-job-logs, --no-job-scripts, or --no-artifacts, and opt bulky content in with --include-job-content, --include-resources <name,…>, or --include-all-resources.

Preview the contents before writing anything with --dry-run, which prints the planned files as a tree; add --list for a flat, root-relative list and --size to annotate each entry with its size and a total:

$ ./exp.sh bundle --dry-run --size

knit bundle needs a bootstrapped experiment (it reads knit.db) and is read-only: it records nothing. To carry files Knit does not track, see Declare extra files to bundle; to add a standard metadata manifest, see Export an RO-Crate manifest.

Declare extra files to bundle

List side files Knit does not track — a config, an input, a plotting script, or a whole glob — with knit_bundle_requires so knit bundle carries them.

APIs: knit_bundle_requires

An experiment often needs files Knit never tracks: a config, a small input, a post-processing script. knit_bundle_requires lists them so knit bundle (see Make a shippable bundle) carries them. Call it at the top of the script, like knit_set_program_description — not inside a command:

# Files Knit does not track but the experiment needs. Listed at the top of the
# script (a top-level declaration, like knit_set_program_description), they are
# added to every `knit bundle`. Paths are relative to this script: a plain path
# is a single file or directory, and a glob is expanded at bundle time so every
# match is packed.
knit_bundle_requires "config/params.yaml"
knit_bundle_requires "inputs/*.dat"

Rules:

• Relative paths only. Each path is relative to the experiment script, so it relocates cleanly when the archive is unpacked. An absolute path is rejected at bundle time.

• A file, a directory, or a glob. A directory is added with its contents. A glob (inputs/*.dat) is expanded at bundle time and every match is packed; a pattern that matches nothing draws a warning.

• Record-only at load time. The declaration never touches the filesystem — no existence check, no error. That keeps re-sourcing the script safe (a job re-enters it on the compute node). All validation — the path exists, is relative, and stays inside the tree — happens when knit bundle runs.

A file kept its path relative to the script root, so config/params.yaml lands at <bundle-root>/config/params.yaml. The @bundle_requires shorthand is an equivalent twin. A setup that declares a Spack environment from a file (knit_with_spack_env "envs/mclib.yaml") records that file automatically — no extra knit_bundle_requires needed.

Export an RO-Crate manifest

Describe the experiment’s provenance as a standard RO-Crate / Process Run Crate manifest, embedded in a bundle with –ro-crate or emitted alone with knit export ro-crate.

APIs: bundle, export:ro-crate

RO-Crate is a standard way to package research data with machine-readable metadata. Knit generates an ro-crate-metadata.json from the recorded provenance — one CreateAction per recorded run, wired to its inputs and outputs, following the RO-Crate 1.1 / Process Run Crate profile — so a bundle is also a self-describing, FAIR-friendly research object.

Add the manifest to a bundle (see Make a shippable bundle) with --ro-crate. It is written at the archive root and describes exactly the packed files, so --ro-crate --zip yields a crate ready for Zenodo or WorkflowHub:

$ ./exp.sh bundle --ro-crate --zip --output montecarlo-pi.zip

To inspect or regenerate the manifest alone, with no archive and no copied files, use knit export ro-crate. It describes the experiment’s on-disk files by their current relative paths. --output defaults to ./ro-crate-metadata.json; --output - writes to stdout for a pipe:

$ ./exp.sh export ro-crate
$ ./exp.sh export ro-crate --output - | jq '.["@graph"] | length'

Both entry points share one generator and the same prerequisites as bundle: a bootstrapped experiment (the manifest is built from knit.db), read-only, recording nothing. The --no-* and --include-* choices that decide what a bundle contains also decide which entities the embedded manifest can reference: it describes exactly what the bundle carries, no more.