Public API

The public API is the stable interface of Knit. These functions and variables have names without a leading underscore and are intended to remain backwards compatible across releases.

Functions

int knit()

This is the main function that invokes the Knit framework. Users should call it as follows at the end of their bash script to forward all arguments to it.

knit $@

int knit_artifact()

Bind a declared artifact of the currently executing command to a path inside the artifacts root. It is the file/directory counterpart of knit_output: the body first puts the file (or a symlink to it) under the artifacts root, then declares it here.

out="$(knit_artifact_dir)"
compute > "${out}/table.csv"
knit_artifact "table" "table.csv"

<linked-path> is either relative to the artifacts root or absolute and inside it. The entry's own location (not its target) must sit inside the artifacts root; a symlink entry is allowed and its target may be anywhere. The entry must exist and match the declared type (a symlink is followed to its target for this check). The content digest of the resolved target is recorded automatically in the companion "<name>-checksum" column, and the recorded value of the artifact is always the artifacts-relative path, whatever form was passed, so the record holds no absolute machine path. Binding the same artifacts-relative path twice in one invocation is fatal: artifacts are write-once.

The optional --link-from / --copy-from shortcuts create the entry from a source that lives elsewhere, so the body does not have to place it by hand. They are mutually exclusive. Both create the parent directories inside the artifacts root and refuse to overwrite an on-disk entry. --link-from resolves <real-path> to an absolute path and makes an absolute-target symlink at <linked-path> (the real bytes stay where they are, at zero copy cost); --copy-from does "cp -r" into <linked-path>. Either way <real-path> must exist, and the created entry then goes through the same existence, type, and checksum path as the direct-write form.

knit_artifact "dataset" "dataset.h5" --link-from /fast/aaa/bigfile
knit_artifact "figure"  "figure.svg" --copy-from "${out}/figure.svg"

Fails if called outside an executing command, if <name> is not a declared artifact of that command, if <linked-path> is empty or outside the artifacts root, if the entry does not exist or has the wrong type, if the path was already bound in this invocation, if both shortcuts are given, if a shortcut's <real-path> is missing or does not exist, or if a shortcut would overwrite an on-disk entry.

Parameters:

• name -- [in] Artifact name (hyphens and underscores are interchangeable).

• linked_path -- [in] Path inside the artifacts root where the entry lives.

• --link-from -- [in] Optional; symlink <linked-path> to this <real-path>.

• --copy-from -- [in] Optional; copy this <real-path> to <linked-path>.

int knit_artifact_dir()

Print the resolved artifact root (see _knit_artifact_root) to stdout. This is the "write into artifacts/ then declare" helper: a command body puts a file (or a symlink to it) under this directory, then binds it with knit_artifact:

out="$(knit_artifact_dir)"
compute > "${out}/table.csv"
knit_artifact "table" "table.csv"

The directory is not created here; creation is lazy, on first bind. Called at most a handful of times per body, so it returns via stdout rather than a nameref.

int knit_as()

Name a call so distinct invocations of the same command can be told apart in a query. Used at a call site as knit_as <alias> <cmd> …: it records <alias> on the provenance "call" edge of the delegated invocation, then runs knit <cmd> …. A later query addresses each call independently by its alias (an edge property). Without knit_as a call edge has a NULL alias.

knit_as fast run --procs 8 -- mcrank
knit_as slow run --procs 1 -- mcrank

The alias is one-shot: it lands only on the directly named call's edge, never on the nested edges that call's body records. It is validated at the call site: it must be non-empty, must not be a registered table name (which would collide with a node label in a query), and must not already have been used within the current invocation (two edges sharing an alias would be indistinguishable).

Parameters:

• alias -- [in] The name to record on the call edge.

• cmd -- [in] The command to invoke (followed by its arguments).

• ... -- [in] Arguments for the command.

int knit_bundle_requires()

Declare an extra file, directory, or glob that "knit bundle" must add to the archive. Called at the top of an experiment script (like knit_set_program_description), not inside a knit_register/knit_done block. It describes the experiment as a whole, so it appends to the global _KNIT_BUNDLE_REQUIRES array rather than to any one command.

The path is stored verbatim. This function MUST NOT touch the filesystem — no stat, no existence check, no knit_fatal. The experiment script is re-sourced on every invocation, including on a compute node that re-enters a job, so a declaration that failed on a missing or non-relocatable path would abort that re-entry and kill the job. All validation (path exists, path is relative, path stays inside the tree) and glob expansion are deferred to "knit bundle".

Parameters:

• path -- [in] A file, directory, or glob pattern, relative to the experiment script's directory.

int knit_check_arguments()

Validate that a plain (non-registered) function received only expected arguments. This is the counterpart, for ordinary helper functions, of the validation that the command registration system performs automatically. It is intended for functions that parse their own "$@" with knit_get_parameter but don't go through knit_register.

The expected parameters are described by two space-separated lists: options (which take a value, as "--name value" or "--name=value") and flags (which do not). Everything from a literal "--" onwards is treated as extra arguments and is not validated. Hyphens and underscores in names are interchangeable, as elsewhere in the framework.

On the first unexpected argument the function logs an error attributed to the calling function and returns 1. It returns 0 if every argument is recognized.

Example:

_knit_submit_local() {
    local -a args=("$@")
    knit_check_arguments "stdout stderr stdin walltime" "" "${args[@]}" \
        || return 1
    ...
}

Parameters:

• options -- [in] Space-separated names of options that take a value.

• flags -- [in] Space-separated names of flags that take no value.

• ... -- [in] The arguments to validate (typically "$@").

int knit_critical()

Logging function for critical-level messages. Works like echo but will only print if the logging level was set to "critical".

Parameters:

• ... -- [in] Arguments for printf.

int knit_debug()

Logging function for debug-level messages. Works like echo but will only print if the logging level was set to "debug".

Parameters:

• ... -- [in] Arguments for printf.

int knit_done()

Finishes to register a function or a parameter set.

int knit_empty()

Empty function to register commands with no behaviors.

int knit_enum()

Define a new enum type with the given name and possible values.

Example:

knit_enum "color" "red" "green" "blue"

Parameters:

• name -- [in] Name of the enum type to define.

• ...values -- [in] Possible values for the enum.

int knit_enum_values()

Print the possible values of an enum type. By default, values are separated by newlines. If a second argument is provided, it is used as the separator instead.

Example:

knit_enum_values "color"          # prints each value on its own line
knit_enum_values "color" ", "     # prints "red, green, blue"

Parameters:

• name -- [in] Name of the enum type.

• separator -- [in] Optional separator (default: newline).

Returns:

1 if the enum does not exist.

int knit_error()

Logging function for error-level messages. Works like echo but will only print if the logging level was set to "error".

Parameters:

• ... -- [in] Arguments for printf.

int knit_extra_index()

Print the index at which extra arguments start (i.e. arguments passed after "--" in the list of arguments). This index will be the size of the list if no extra arguments are found. The way this function can be used is as follows.

local args=("$@")
local extra_index=$(knit_extra_index "${args[@]}")
local extra=("${args[@]:extra_index}")

Parameters:

• ... -- [in] List of arguments.

int knit_fatal()

Logging function for fatal error messages. Will be printed no matter the log level, and the program will exit with an error value.

Parameters:

• ... -- [in] Arguments for printf.

int knit_framed()

Piping a command's output into knit_framed will make the output appear inside a frame with the given height and width. If stdout is not a TTY, stdin is forwarded to stdout unchanged.

Usage: my-command arg1 arg2 ... | knit_framed [<height> [<width>]] [--title <title>] [--cleanup]

Parameters:

• height -- [in] Height of the frame (default: terminal height).

• width -- [in] Width of the frame (default: terminal width).

• --title -- [in] Optional title displayed centered on the top border.

• --cleanup -- [in] If present, erase the frame from the screen after completion.

• --frame-color -- [in] Foreground color of the frame borders (key in _KNIT_COLORS).

• --frame-bg-color -- [in] Background color of the frame borders (key in _KNIT_COLORS).

• --text-color -- [in] Foreground color of the text inside the frame (key in _KNIT_COLORS).

• --text-bg-color -- [in] Background color of the text inside the frame (key in _KNIT_COLORS).

• --log-level -- [in] Minimum log level required to display the frame (trace, debug, info, warning, error, critical). If KNIT_LOG_LEVEL is above this level, stdin is drained silently and nothing is output.

int knit_get_parameter()

Search the list of arguments for a specific parameter. If found, the function will print the value associated with the parameter (flags will lead to this function printing "true" or "false"). If not found, this function will print nothing and return 1.

The parameter value may be supplied either as two tokens ("--name value") or inline with an equals sign ("--name=value"); both forms are recognized.

Parameters:

• param -- [in] Parameter to search for (without the -- prefix).

• ... -- [in] Arguments in which to search for the parameter.

int knit_get_profile_field()

Extract one field from the bootstrapped experiment's profile using a jq path expression. Reads the resolved profile JSON frozen at bootstrap in the "__profile_json__" metadata. Prints the field value (without enclosing quotes for strings) or an empty string when no profile is configured or the field is absent.

Parameters:

• jq_path -- [in] jq path expression, e.g. '.scheduler.type'.

int knit_hidden()

Mark a command as hidden, i.e. it will not appear in usage help messages. This is the static, unconditional hide flag (the "_is_hidden" boolean), also read by _knit_provenance_enabled and describe, and it stands apart from the dynamic knit_hidden_if predicates.

knit_hidden and knit_hidden_if are mutually exclusive per command (last-writer wins with a warning): calling knit_hidden after one or more knit_hidden_if statements shadows them, so it emits a warning, sets the static flag, and drops the collected dynamic predicates.

int knit_hidden_if()

Declare that the command currently being registered is hidden from its parent's "--help" whenever <predicate> returns 0. <predicate> is the name of a user-defined shell function that receives the demangled command name as its single argument and returns 0 ("hide") or non-zero ("show"). Unlike knit_hidden this is a dynamic, "--help"-only hide: the command remains fully invokable and is still visible to _knit_provenance_enabled and describe.

Repeatable: multiple calls register multiple predicates and the command is hidden if any of them returns 0 (logical OR).

knit_hidden and knit_hidden_if are mutually exclusive per command: if the command is already statically hidden (knit_hidden was called first), the dynamic predicate would be meaningless, so this call emits a warning and is ignored. The per-command storage array (_hidden_pred) is declared lazily on first use.

Parameters:

• predicate -- [in] Name of the predicate function.

int knit_hidden_if_not_usable()

Shorthand for a knit_hidden_if whose predicate hides the command from "--help" exactly when at least one of its knit_usable_if predicates is false. Takes no arguments. Backed by the internal predicate _knit_hidden_if_not_usable_pred, appended to the command's _hidden_pred array; a command with no knit_usable_if predicates is always usable, hence never hidden by this shorthand.

Subject to the same mutual exclusion with knit_hidden as knit_hidden_if.

int knit_highlight_if()

Declare that the command currently being registered has its name highlighted (bold) in its parent's "--help" whenever <predicate> returns 0 and the output is a terminal. <predicate> is the name of a user-defined shell function that receives the demangled command name as its single argument and returns 0 ("highlight") or non-zero ("plain"). Highlighting is purely cosmetic: it never affects invokability, provenance, or describe.

Repeatable: multiple calls register multiple predicates and the command is highlighted if any of them returns 0 (logical OR). The per-command storage array (_highlight_pred) is declared lazily on first use.

Parameters:

• predicate -- [in] Name of the predicate function.

int knit_info()

Logging function for info-level messages. Works like echo but will only print if the logging level was set to "info".

Parameters:

• ... -- [in] Arguments for printf.

int knit_input_artifact_path()

Resolve an input-artifact parameter value (an artifacts-relative path) into its absolute on-disk location under the experiment's artifacts root and print it to stdout. This is how the body of a command that consumes an artifact turns the parameter value into a usable path:

csv="$(knit_input_artifact_path "$(knit_get_parameter input_table "$@")")"

The value must resolve inside the artifacts root (the same containment rule knit_artifact enforces on a produced entry); an absolute path outside it is fatal. Fatal too when the value is empty or the resolved entry does not exist, since a body should never run against an artifact that is not present. Called at most a handful of times per body (one per declared input artifact), so it returns via stdout rather than a nameref. Mirrors knit_resource_path.

Parameters:

• value -- [in] The input-artifact parameter value (an artifacts-relative path).

int knit_input_artifact_paths()

Resolve a variadic input-artifact parameter value (a comma-separated list of artifacts-relative paths, each possibly a glob) into an array of absolute on-disk locations, filling a caller-named bash array. This is how the body of a command that consumes a variadic ("name:kind*" / "name:kind+") input turns the parameter value into usable paths, iterated safely — no comma splitting, so a path that holds a space stays intact:

local -a tables
knit_input_artifact_paths tables "$(knit_get_parameter tables "$@")"
for csv in "${tables[@]}"; do cat "${csv}" >> combined.csv; done

The raw value is resolved with the shared list resolver (comma split, then glob expansion relative to the artifacts root under nullglob, first-seen de-duped, in order), the same one the before-callback validated against and the after-callback recorded edges for, so the body's view cannot drift from validation and lineage. Each resolved element is turned into its absolute path under the artifacts root; an element that resolves outside that root is fatal (the same containment rule knit_input_artifact_path enforces). An empty value fills the array with no elements. The scalar knit_input_artifact_path is unchanged and still serves a scalar input (one path, printed to stdout).

This returns by nameref (not stdout) because the result is an array that may hold paths with spaces, which a command-substitution capture would re-split. Its internal locals are all "__"-prefixed: it is public API, so a caller may name the output array anything, and a plain internal sharing that name would shadow the nameref (see the project's nameref-shadow-collision note).

Parameters:

• __knit_ret -- [out] Name of the array variable to fill with absolute paths.

• __raw -- [in] The raw parameter value (comma-separated list of paths).

int knit_job_hostnames()

Print the hostnames the current job is running on, discovered from the active scheduler backend (see _knit_sched_hostfile). Intended to be called inside a job body. Outside a scheduler allocation it reports the local hostname.

By default each host is printed once, on its own line, with any trailing ":N" slot-count or extra columns removed and duplicates collapsed (first-seen order preserved). Use --raw to print the backend's hostfile entries verbatim (e.g. one line per launchable slot), which is the form an MPI launcher hostfile wants.

Usage: knit_job_hostnames [--json] [--separator <sep>] [--raw] [--select <s>:<n>]

Parameters:

• --json -- [in] Print the hostnames as a JSON array of strings.

• --separator -- [in] Separator used to join hostnames (default: a newline). Ignored when --json is given.

• --raw -- [in] Print the raw hostfile entries verbatim (no ":N" stripping, no deduplication); only the separator / JSON wrapping is applied.

• --select -- [in] Print only a slice of the resulting list: <start>:<length>, where <start> is a 0-based index and <length> is the number of hostnames to print. The slice is taken after the raw or deduplication step, so it counts the entries that would otherwise be printed. Out-of-range requests are clamped (a start past the end yields nothing).

int knit_job_nodecount()

Print the number of distinct nodes allocated to the current job: the count of deduplicated hostnames reported by knit_job_hostnames (so a host contributing several launchable slots is counted once). Intended to be called inside a job body; outside a scheduler allocation it reports 1 (the local host). A common use is to launch one rank per node: knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- <app>.

Takes no arguments.

int knit_list_profiles()

Print the union of the profiles known to knit, one per line in sorted order, laid out like a command's parameters in "--help": a name column, a "[source]" annotation, then the profile's description word-wrapped with a hanging indent under the annotation column. The source is the committed in-repo index (fetched from the default branch) or the admin store under _KNIT_PROFILE_ADMIN_DIR; an admin profile that shares a name with a repo one shadows it (§4.5) and is marked accordingly (its description wins). The repo index is fetched best-effort so an offline, admin-only machine still lists its own profiles.

Hidden profiles (JSON "_hide": true) are omitted by default. When show_hidden is "true" they are included and annotated ", hidden" after the source.

Wrapping reuses _knit_terminal_width / _knit_help_render_entry (the same helpers "--help" uses), so alignment, wrap-around, and the pipe/redirect single-line fallback all match option listings.

Parameters:

• show_hidden -- [in] "true" to also list hidden profiles (default "false").

int knit_log_set_level()

Set the log level. The level should be either trace, debug, info, warning, error, or critical.

Parameters:

• level -- [in] Log level.

Returns:

0 if the log level was set, 1 otherwise.

int knit_no_record_on_failure()

Mark the command currently being registered so that a failed invocation records no database row. By default a command with a table (knit_with_table) records a row after its body runs, whatever the body's exit status; with this directive a non-zero body exit skips recording, so a failed run leaves no data row behind. Use it for a command that removes its own partial output on failure (as knit fetch removes a partial resource instance), where a dangling row would misrepresent a run that produced nothing.

Must be called between a knit_register* call and knit_done. It is not valid on a job (knit_register_job): a job's row carries a "state" column written from its callbacks and signal traps (running / killed / completed), which suppressing a failed job's row would drop. It is not valid on a wrapper (knit_register_wrapper) either, which forwards its arguments verbatim and has no body of its own. Calling it more than once on the same command is harmless.

int knit_output()

Record a named output value from within a registered command function. Fails if called outside of an executing command, if the name was not declared with knit_with_output, or if the value does not match the declared type.

Parameters:

• name -- [in] Output name (hyphens and underscores are interchangeable).

• value -- [in] Value to record.

int knit_parameter_set()

Begins the definition of a named parameter set. A call to this function should be followed by any number of knit_with_required, knit_with_optional, and knit_with_flag calls, then a call to knit_done. The resulting set can then be imported into one or more commands with knit_with_parameter_set.

Parameters:

• name -- [in] Name of the parameter set (letters, digits, hyphens, underscores).

int knit_platform_name()

Print the platform name recorded for the bootstrapped experiment (the "__platform__" metadata key). This is the value of bootstrap's --platform option, or, when that was omitted and a --profile was given, the profile's own "name" field (e.g. "anl/polaris"). Prints an empty string when no platform was recorded.

int knit_popd()

Silent version of popd.

int knit_provides_launcher()

Declare that the setup currently being registered may supply an MPI launcher on a machine that has none. Must be called between knit_register_setup and knit_done, at most once per setup. A bare directive (the only form for now): it means "this setup may provide a launcher where the machine offers one of its

own" — it sits

below a concrete machine launcher (launcher) in the precedence, so a profile's launcher still wins (see _knit_launch_backend). A deliberate override is future work.

The launcher is detected once at setup-build time (not at run time) by the after-callback and frozen into the setup's .activate.sh as KNIT_PROVIDED_LAUNCHER; it is recorded as the mpi_launcher provenance output (a column in the setup's table). Setups are the only valid target: wrappers and non-setup commands are rejected.

Example:

knit_register_setup "juliaenv" _juliaenv "Build against MPI."
knit_with_spack_specs "cmake" "mpi"
knit_provides_launcher            # may launch where the machine has no MPI
_juliaenv() { ... }
knit_done

int knit_pushd()

Silent version of pushd.

int knit_register()

Register a function for use with a CLI. A call to this function should be followed by any number of knit_with_* calls, followed by the declaration of the function to register, then a call to knit_done.

Parameters:

• cmd -- [in] Command (demangled).

• name -- [in] Name of the function to register.

• description -- [in] Description of the command.

int knit_register_app()

Register an app, i.e. a subcommand of the "run" command that executes as an MPI launch inside a job. Mirrors knit_register_job: it registers run:<name>, backs it with a per-app table named after the app, records the name in _KNIT_APPS, and installs a before-callback asserting the run context. An app inherits the ambient setup of the surrounding job by default, but may declare knit_with_setup to explicitly depend on and re-source a setup. There is no after-callback.

A call to this function must be followed by any knit_with_* declarations, the definition of <fn>, and a call to knit_done.

Example:

knit_register_app "hello" "hello_fn" "Hello world MPI app."
knit_with_optional "n:integer" "100" "Problem size."
hello_fn() {
  ...
}
knit_done

Parameters:

• name -- [in] Short name for the app (used as the subcommand name).

• fn -- [in] Name of the Bash function implementing the app.

• description -- [in] One-line description shown in --help.

int knit_register_artifact()

Declare an artifact kind: a semantic category an artifact may carry (e.g. "csvfile", "rundir"), backed by a physical type ("file" or "directory"). A top-level declaration, like knit_enum — not called between knit_register and knit_done. Once declared, a kind may be referenced by knit_with_output_artifact and knit_with_input_artifact with the "name:kind" syntax.

The builtin kinds "file", "directory", and the "dir" alias are pre-registered (each backed by its matching type), so the common case needs no call here and re-registering one of them is fatal.

Example:

knit_register_artifact "csvfile:file"     "Tabulated result in CSV format."
knit_register_artifact "rundir:directory" "A run's output directory."

Fatal when the annotation has no ":type", when the kind name is invalid, when the kind is already registered, or when the type is not a checksummable ("file"/"directory") type.

Parameters:

• spec -- [in] Kind name followed by ":type" ("file" or "directory").

• description -- [in] Optional one-line description for knit describe / --help.

int knit_register_job()

Register a job, i.e. a subcommand of the "submit" command that executes as a job sumitted on the supercomputer.

A call to this function must be followed by any knit_with_* declarations, the definition of <fn>, and a call to knit_done.

The names "prepared", "next", and "from" are reserved: "submit prepared" and "submit next" release prepared jobs and "prepare from" reads a plan, so a job registered under one of these would mangle to the same command name and shadow the release/plan subcommand. Registering such a job is fatal.

Example:

knit_register_job "hello" "hello_fn" "Hello world job script."
knit_with_optional "name:string" "Matthieu" "Name of the person to greet."
hello_fn() {
  ...
}
knit_done

Parameters:

• name -- [in] Short name for the job (used as the subcommand name).

• fn -- [in] Name of the Bash function implementing the job.

• description -- [in] One-line description shown in --help.

int knit_register_resource()

Register a resource type: a downloadable input artifact acquired through the knit fetch dispatcher. Unlike knit_register_setup there is no user body to supply — the type is backed by the shared _knit_resource_fetch_body, which dispatches on the declared download method. The type is backed by a database table named "resource:<type>" (via knit_with_table) and recorded in the resource registry so knit_with_resource can validate a declared dependency.

A call to this function must be followed by exactly one download decorator (knit_with_git / knit_with_url / knit_with_local), an optional knit_with_checksum, and a call to knit_done. It must run before any knit_with_resource that references the type.

Example:

knit_register_resource "julia_code" "Julia fractal source."
knit_with_git "https://github.com/knit-sh/julia-fractal-example.git" "main"
knit_done

Parameters:

• type -- [in] Short name for the resource type (used as the dispatch target).

• description -- [in] One-line description shown in --help.

int knit_register_setup()

Register a setup, i.e. a subcommand of the "setup" command that builds software and records the resulting environment. The setup is automatically backed by a database table named "setup:<name>". A before-callback checks that KNIT_SETUP_PREFIX is set; an after-callback saves the environment to $KNIT_SETUP_PREFIX/.activate.sh.

A call to this function must be followed by any knit_with_* declarations, the definition of <fn>, and a call to knit_done.

Example:

knit_register_setup "hello" "hello_fn" "Builds the hello program."
knit_with_optional "version:string" "main" "Branch or tag to check out."
hello_fn() {
  ...
}
knit_done

Parameters:

• name -- [in] Short name for the setup (used as the subcommand name).

• fn -- [in] Name of the Bash function implementing the setup.

• description -- [in] One-line description shown in --help.

int knit_register_wrapper()

Register a wrapper command: a command that forwards all of its arguments verbatim to an underlying command (e.g. "knit spack ..." forwarding to the knit-installed spack). A wrapper differs from a regular command in that it:

1. cannot declare parameters or outputs (knit_with_required/optional/flag/ output/dispatch/parameter_set are fatal);

2. performs no argument validation, expansion, or --when checking;

3. forwards "$@" verbatim to <fn>, including "--help" and "--";

4. may declare a table with knit_with_table, in which case the whole forwarded command line is recorded in a single "args" column;

5. may install before/after callbacks, which still run around <fn>.

A call to this function must be followed by the definition of <fn>, an optional knit_with_table, and a call to knit_done.

Example:

knit_register_wrapper "spack" "_knit_spack" "Wrapper for the spack command"
_knit_spack() { spack "$@"; }
knit_done

Parameters:

• name -- [in] Command name (used as the wrapper's invocation name).

• fn -- [in] Name of the Bash function the wrapper forwards to.

• description -- [in] One-line description shown in "--help".

int knit_resource_path()

Resolve a resource instance name into its absolute directory under the experiment's resource root: <resource-root>/<name> (see _knit_resource_root), and print it to stdout. This is how the body of a command that depends on a resource turns a resource parameter value (the instance name) into an on-disk path:

train_dir="$(knit_resource_path "$(knit_get_parameter training_dataset "$@")")"

The name is validated as a single path component first (fatal otherwise). Fatals when the named instance does not exist, since a body should never run against a resource that was never fetched. Called at most a handful of times per body (one per declared resource), so it returns via stdout rather than a nameref.

Parameters:

• name -- [in] The resource instance name (as passed to knit fetch --name).

int knit_set_program_description()

Set the description of the program.

int knit_setup_activate_line()

Declare a verbatim activation line. Called from a setup body, it both runs the line in the current build shell (so the rest of the body sees its effect) and records it verbatim into the setup's .activate.sh, so every dependent job runs the same line. Use it for activation steps that the env helpers do not cover, e.g. module load gcc/12 or source <some tool's env script>.

Parameters:

• line -- [in] The shell line to run now and record.

int knit_setup_env_append()

Declare that a setup appends an entry to a colon-separated list variable (e.g. PATH). Called from a setup body, it both appends to VAR in the current build shell and records a composable line into the setup's .activate.sh: export VAR="${VAR:+${VAR}:}"<value>. That line extends the job's own VAR at activation time rather than replacing it, and is empty-safe (no leading colon when VAR was unset).

Parameters:

• var -- [in] The list variable name (a shell identifier).

• value -- [in] The entry to append.

int knit_setup_env_prepend()

Declare that a setup prepends an entry to a colon-separated list variable. Called from a setup body, it both prepends to VAR in the current build shell and records a composable line into the setup's .activate.sh: export VAR=<value>"${VAR:+:${VAR}}". That line prefixes the job's own VAR at activation time rather than replacing it, and is empty-safe (no trailing colon when VAR was unset).

Parameters:

• var -- [in] The list variable name (a shell identifier).

• value -- [in] The entry to prepend.

int knit_setup_env_set()

Declare that a setup sets an environment variable. Called from a setup body, it both exports VAR=value in the current build shell and records export VAR=<value> into the setup's .activate.sh, so every dependent job gets the same assignment. The value is recorded with printf q so any characters survive re-sourcing.

Parameters:

• var -- [in] The variable name (a shell identifier).

• value -- [in] The value to assign.

int knit_setup_env_unset()

Declare that a setup unsets an environment variable. Called from a setup body, it both unsets VAR in the current build shell and records unset VAR into the setup's .activate.sh.

Parameters:

• var -- [in] The variable name to unset (a shell identifier).

int knit_trace()

Logging function for trace-level messages. Works like echo but will only print if the logging level was set to "trace".

Parameters:

• ... -- [in] Arguments for printf.

int knit_type_check()

Check whether a value conforms to the specified type. For enum types, checks that the value is one of the defined enum values.

Type validation rules:

• integer: optional sign followed by digits

• real: decimal number with optional exponent (e.g. 3.14, .5, 1e10)

• boolean: "true" or "false"

• string: any value (always passes)

• path, filename, file, directory: non-empty string (shape only; existence of a file or directory is checked separately as direction-aware runtime logic, not by this pure type check)

• date: YYYY-MM-DD with valid month/day ranges

• time: hh:mm:ss with valid hour/minute/second ranges

• datetime: "YYYY-MM-DD hh:mm:ss" combining date and time rules

Example:

knit_type_check "integer" "42"          # returns 0
knit_type_check "integer" "hello"       # returns 1
knit_type_check "date" "2025-03-13"     # returns 0
knit_type_check "color" "red"           # returns 0 (if color enum defined)

Parameters:

• type -- [in] Type name (or alias) to check against.

• value -- [in] Value to validate.

Returns:

0 if the value is valid for the type, 1 otherwise.

int knit_type_exists()

Check whether a type name is valid. Returns 0 if the name is a built-in type, a type alias, or a user-defined enum.

Example:

knit_type_exists "integer"    # returns 0
knit_type_exists "int"        # returns 0 (alias for integer)
knit_type_exists "unknown"    # returns 1

Parameters:

• type_name -- [in] Type name to check.

Returns:

0 if the type exists, 1 otherwise.

int knit_usable_before_bootstrap()

Mark the command currently being registered as usable before bootstrap, i.e. it may be invoked (and appears in "--help") on a fresh checkout where no ".knit/" directory exists yet. Commands are not usable before bootstrap by default.

A command usable before bootstrap must not declare a database table (knit_with_table) nor carry any "--when" constraint on its parameters (both rely on binaries that bootstrap provisions), and a subcommand may only be usable before bootstrap if its parent is too. These rules are enforced at knit_done time by knit_usable_before_bootstrap_validate (registered here as a knit_done callback), so that they see the final command definition regardless of the order in which knit_with_table / knit_with* / this decorator are called.

Calling it more than once on the same command is harmless (idempotent): the validation callback is registered only on the first call.

int knit_usable_if()

Declare that the command currently being registered may be used only when <predicate> returns 0. <predicate> is the name of a user-defined shell function that receives the demangled command name as its single argument and returns 0 ("usable") or non-zero ("not usable"). is a human-readable string explaining why the command cannot run; it is shown as a fatal error if the user invokes the command while the predicate is false.

Repeatable: multiple calls register multiple predicates. At invocation time the predicates are evaluated in declaration order, stopping at the first that returns non-zero (whose becomes the error message); a command is usable only if all of its predicates pass.

The predicate is not called here — only its name and description are recorded. Enforcement happens at invocation time via _knit_command_check_usable. The per-command storage arrays (_usable_pred / _usable_desc) are declared lazily on first use so commands that do not use this decorator pay no registration cost.

Parameters:

• predicate -- [in] Name of the predicate function.

• description -- [in] Message shown if the command is invoked while not usable.

int knit_warning()

Logging function for warning-level messages. Works like echo but will only print if the logging level was set to "warning".

Parameters:

• ... -- [in] Arguments for printf.

int knit_with_checksum()

Declare an optional integrity pin for the resource type: a sha256 verified at fetch time, but only when the source defaults are used (an overridden source would legitimately hash differently) and not bypassed with --ignore-checksum. Valid only on a resource type, at most once. The pin is the sha256 of the downloaded archive (url), the sha256 of the local file or a recursive digest of a local directory (local), or the expected commit SHA the default ref resolves to (git). A mismatch fails the fetch and removes the partial instance.

Parameters:

• sha256 -- [in] The expected sha256 (a commit SHA for the git backend).

int knit_with_dispatch()

Mark the command currently being registered as a dispatcher: a command that takes a target after "--" and forwards the remaining arguments to it (e.g. "submit" dispatches to a job, "setup" to a setup. This changes how "--help" renders the usage line, both for the dispatcher itself (cmd [OPTIONS] -- <placeholder> [OPTIONS]) and for its subcommands, which are invoked through it rather than directly.

The description is stored as the extra-arguments description (as with knit_with_extra), so the "Extra" help section and the "extra allowed after

--" argument check keep working. If no description is given, the placeholder is used.

Parameters:

• placeholder -- [in] Name shown after "--" in the usage line (e.g. "job").

• description -- [in] Optional description of the extra arguments.

int knit_with_extra()

Adds a description for extra parameters coming after "--".

Parameters:

• description -- [in] Description of the extra parameters.

int knit_with_flag()

This function should be called right after a call to knit_register (or one of its variants) to declare flag parameters that the command may accept.

Example:

knit_register "greet" "say_hello" "Say hello to someone"
knit_with_flag "capitalize" "Make the output upper-case"
say_hello() {
   ...
}

Parameters:

• param -- [in] Parameter name.

• description -- [in] Description of the parameter.

• --when -- [in] Optional boolean constraint expression (jq syntax referring to the command's other parameters); the parameter only applies when the expression evaluates to true.

int knit_with_git()

Download decorator: acquire the resource type by cloning a git repository and checking out a ref. Declares the automatic parameters --url (default <url>) and --ref (default <ref>); both arguments are required at registration because a resource type must name the ref it pins. Also declares the "commit" output, in which the fetch records the commit SHA the ref resolved to. Valid only on a resource type, at most one download decorator per type.

Parameters:

• url -- [in] The default git repository URL.

• ref -- [in] The default git ref (branch, tag, or commit) to check out.

int knit_with_input_artifact()

Declare that the command currently being registered consumes a recorded artifact of a given kind, given as "name:kind" (e.g. "input_table:csvfile"). Must be called between a knit_register* call and knit_done; a command may declare several input artifacts. The kind must be a registered artifact kind (see knit_register_artifact); the builtin kinds "file", "directory", and the "dir" alias are always available. It is the consuming counterpart of knit_with_output_artifact.

Under the hood (mirroring knit_with_resource) this registers an ordinary required string parameter named name whose value is the artifact's artifacts-relative path, so the CLI, knit_get_parameter, and the backing table all treat it as a plain value; the command body turns that path into an on-disk location with knit_input_artifact_path. A per-parameter marker (KNIT_CMD<cmd>input_artifact=<kind>) records the required kind, and a companion input_artifact_verify marker records the --verify-checksum opt-in; both are read by the validation before-callback and, later, by describe / --help. A before-callback resolves the path to the artifacts row and validates it (existence + kind, plus an opt-in checksum re-verification) before the body runs, and an after-callback records a "used_by" provenance edge from the consumed artifact to this command.

knit_register "plot" plot "Plot a results table."
knit_with_input_artifact "input_table:csvfile" "The table to plot." --verify-checksum
plot() {
    local csv
    csv="$(knit_input_artifact_path "$(knit_get_parameter input_table "$@")")"
    gnuplot -e "..." "${csv}"
}
knit_done

Invoked as: ./exp.sh plot --input-table tables/run7.csv.

Wrappers cannot declare knit_with_input_artifact (they forward their arguments verbatim and take no parsed parameters). Every declared input artifact is required.

Parameters:

• spec -- [in] The dependency as "name:kind" (a registered artifact kind).

• description -- [in] One-line description of the input-artifact parameter.

• --verify-checksum -- [in] Optional flag; re-verify the recorded checksum before the body runs.

int knit_with_local()

Download decorator: acquire the resource type from a local path, by default as a symlink (so a large staged dataset is not duplicated) or, with --copy, as a self-contained read-only snapshot. Declares the automatic parameters --path (default <path>) and the --copy flag. Valid only on a resource type, at most one download decorator per type.

Parameters:

• path -- [in] The default local source path to link or copy.

int knit_with_optional()

This function should be called right after a call to knit_register (or one of its variants) to declare optional parameters for the command. The parameter name may include a type annotation using the "name:type" syntax (e.g. "count:integer"). If no type is given, "string" is assumed.

Example:

knit_register "greet" "say_hello" "Say hello to someone"
knit_with_optional "name:string" "world" "Name of the person to greet"
knit_with_optional "count:integer" "1" "Number of times to greet"
say_hello() {
   ...
}

Indicates that the command "greet" has an optional parameter --name (string, default "world") and --count (integer, default "1").

The default may be written as "ENV[NAME]" to mean "fall back to the value of

the NAME environment variable when the parameter is not provided". This is resolved when the parameter is filled in, so a job whose environment is set up by a

knit setup (e.g. knit_with_optional "seed:integer" "ENV[MC_SEED]" ...) picks up the value exported by that setup.

Parameters:

• param -- [in] Parameter name followed by ":type".

• default -- [in] Default value (or "ENV[NAME]" to read the NAME env variable).

• description -- [in] Description of the parameter.

• --when -- [in] Optional boolean constraint expression (jq syntax referring to the command's other parameters); the parameter only applies when the expression evaluates to true.

• --no-checksum -- [in] Optional flag; for a file/directory parameter, disable the content checksum and its companion column.

int knit_with_output()

This function should be called right after a call to knit_register (or one of its variants) to declare an output that the command produces. The output name must include a type annotation using the "name:type" syntax (e.g. "result:integer").

Example:

knit_register "compute" "compute" "Compute something."
knit_with_output "result:real" "0.0" "The computed result."
compute() {
   ...
}

Parameters:

• param -- [in] Output name followed by ":type".

• default -- [in] Default value.

• description -- [in] Description of the output.

• --no-checksum -- [in] Optional flag; for a file/directory output, disable the content checksum and its companion column.

• --result -- [in] Optional flag; mark the output as a result (what the experiment was for). Valid on an output of any type.

int knit_with_output_artifact()

This function should be called right after a call to knit_register (or one of its variants) to declare an artifact that the command produces: a file or directory, kept under the artifacts root, that the command binds at runtime with knit_artifact. It is the file/directory counterpart of knit_with_output. The artifact name must include a kind annotation using the "name:kind" syntax, and the kind must be a registered artifact kind (see knit_register_artifact). The builtin kinds "file", "directory", and the "dir" alias are always available; a user kind such as "csvfile" is declared once at the top level. The physical type behind the kind ("file" or "directory") drives the existence check and the checksum, while the kind itself is recorded on the artifacts row.

Example:

knit_register "tabulate" "tabulate" "Tabulate results."
knit_with_output_artifact "table:csvfile" "The results table (CSV)."
tabulate() {
   out="$(knit_artifact_dir)"
   compute > "${out}/table.csv"
   knit_artifact "table" "table.csv"
}

Unlike an ordinary output, an artifact is NOT a column of the command's own table. Each binding is recorded at runtime as one row in the framework-owned artifacts table (its artifacts-relative path, name, physical type, semantic kind, content checksum, and result flag) with a "produced" edge from the producing invocation, so a file can be traced back to what made it. The content digest is always recorded for an artifact, so there is no --no-checksum opt-out here. The artifact is added to the command's artifacts set, and its kind/description are kept in registration state so knit describe can report it.

Because a "produced" edge needs the producing invocation's row as its source, a command that declares an artifact records an invocation row even if it declared no table of its own: a table is ensured automatically at knit_done time.

Parameters:

• param -- [in] Artifact name followed by ":kind" (a registered artifact kind).

• description -- [in] Description of the artifact.

• --result -- [in] Optional flag; mark the artifact as a result (what the experiment was for).

int knit_with_parameter_set()

Import parameters from a previously defined parameter set into the command currently being registered. May be called multiple times with different sets. Conflicts between the set's parameters and parameters already declared for the command are reported as fatal errors.

By default every parameter the set declares is imported. The optional --exclude and --only options (mutually exclusive) narrow the import:

• --exclude "a,b" imports the set minus the named parameters, freeing those names so the command can declare them itself (e.g. with a different kind or default).

• --only "a,b" imports only the named parameters (an allow-list).

Every name in either list must exist in the set, else the call is fatal.

Parameters:

• name -- [in] Name of the parameter set to import.

• --exclude -- [in] Optional comma-separated list of parameters to skip.

• --only -- [in] Optional comma-separated allow-list of parameters to import.

int knit_with_provenance()

Mark the command being registered as participating in the provenance graph: an invocation of it records a "call" edge and acts as an in-process parent frame for the commands it invokes. This is an explicit override of the default (which is by visibility — see _knit_provenance_enabled), so it forces a hidden command into the graph. The mark also propagates to unmarked lexical descendants (e.g. "a" marked "with" makes an unmarked "a:b" participate).

int knit_with_required()

This function should be called right after a call to knit_register (or one of its variants) to declare required parameters that the command expects. The parameter name may include a type annotation using the "name:type" syntax (e.g. "width:integer"). If no type is given, "string" is assumed.

Example:

knit_register "greet" "say_hello" "Say hello to someone"
knit_with_required "name:string" "Name of the person to greet"
knit_with_required "count:integer" "Number of times to greet"
say_hello() {
   ...
}

Indicates that the command "greet" requires a parameter --name (string) and --count (integer).

Parameters:

• param -- [in] Parameter name followed by ":type".

• description -- [in] Description of the parameter.

• --when -- [in] Optional boolean constraint expression (jq syntax referring to the command's other parameters); the parameter only applies when the expression evaluates to true.

• --no-checksum -- [in] Optional flag; for a file/directory parameter, disable the content checksum and its companion column.

int knit_with_resource()

Declare that the command currently being registered consumes a fetched resource instance, given as "<param>:<type>" (e.g. "training_dataset:image_dataset"). Must be called between a knit_register* call and knit_done; a command may declare several resources. The <type> must be a resource type registered with knit_register_resource before this call.

Under the hood this registers an ordinary required string parameter named <param> (so the CLI, knit_get_parameter, and the backing table all treat it as a plain value) whose value is the resource instance name. The command body turns that name into a path with knit_resource_path. A per-parameter marker (KNIT_CMD<cmd>resource=<type>) records the declared type for validation and, later, for describe / --help. A before-callback validates the named instance (existence + recorded type) before the body runs, and an after-callback records a "used_by" provenance edge from the fetched instance to this command.

Wrappers cannot declare knit_with_resource (they forward their arguments verbatim and take no parsed parameters). Every declared resource is required for now.

Example:

knit_register "train" _train "Train a model."
knit_with_resource "training_dataset:image_dataset" "Training images."
_train() {
    local dir
    dir="$(knit_resource_path "$(knit_get_parameter training_dataset "$@")")"
    ...
}
knit_done

Parameters:

• spec -- [in] The dependency as "<param>:<type>".

• description -- [in] One-line description of the resource parameter.

int knit_with_setup()

Declare that the command currently being registered requires a setup of a given type (the name of a setup registered with knit_register_setup). Must be called between a knit_register* call and knit_done, at most once per command.

How the requirement is consumed depends on the kind of command:

• Jobs (knit_register_job): knit submit makes --setup mandatory, rejects a --setup that was not built by the declared type, and the job re-sources the setup environment on the compute node.

• Any other command: knit_with_setup adds a --setup option to the command, and a before-callback validates the given setup's type and sources its .activate.sh so the command body runs in the setup environment.

Setups and wrappers may NOT declare knit_with_setup and are rejected:

• a setup's KNIT_SETUP_PREFIX is its own output directory, so chaining setups would make that prefix ambiguous;

• a wrapper forwards its arguments verbatim and cannot take a parsed --setup.

Example:

knit_register_job "montecarlo" _montecarlo_job "Estimate pi as a job."
knit_with_setup "mcenv"   # requires a setup built by the "mcenv" setup
_montecarlo_job() { ... }
knit_done

Parameters:

• type -- [in] Name of the required setup type.

int knit_with_spack_env()

Declare that the setup currently being registered needs a Spack environment, built as the setup's first step and inherited by jobs via re-activation. Must be called between knit_register_setup and knit_done, at most once per setup (also mutually exclusive with knit_with_spack_specs, which funnels through here).

The environment is described either by a file (non-empty argument, resolved to an absolute path at registration) or, when no argument is given, by a here-doc / stdin manifest consumed at registration time. In the no-argument form the manifest must actually be redirected: if stdin is an interactive terminal (so there is nothing to read) the directive fails fast instead of blocking on input, and an empty stdin manifest is likewise rejected. The directive installs the build/activation callbacks, declares the spack_yaml / spack_lock provenance outputs (which become columns in the setup's table), advertises the requirement in --help, and sets _KNIT_SPACK_REQUIRED so bootstrap auto-provisions Spack.

Example:

knit_register_setup "libs" "libs_fn" "Build deps with Spack."
knit_with_spack_env "spack.yaml"          # path form
# --- or ---
knit_with_spack_env <<'EOF'               # here-doc form
spack:
  specs: [hdf5@1.14, fftw]
  view: true
EOF
libs_fn() { ... }
knit_done

Parameters:

• file -- [in] Optional path to a spack.yaml manifest. If omitted, the manifest is read from stdin.

int knit_with_spack_specs()

Lightweight sugar over knit_with_spack_env for the common "just install these

specs" case: it synthesizes a minimal spack.yaml (the given specs plus "view: true") and feeds it to knit_with_spack_env, so provenance capture, activation, and auto-provisioning all apply identically. Must be called between knit_register_setup and knit_done, and is mutually exclusive with knit_with_spack_env on one setup.

Example:

knit_register_setup "libs" "libs_fn" "Build deps."
knit_with_spack_specs "hdf5@1.14" "fftw" "boost"
libs_fn() { ... }
knit_done

Parameters:

• ... -- [in] One or more Spack specs.

int knit_with_subcommand_title()

Change the title of subcommands for the command being registered (default subcommand name is "Subcommands"). This is the title displayed when calling --help.

int knit_with_table()

Declare a database table for recording invocations of the command currently being registered. Must be called between knit_register and knit_done.

If no table name is given, the demangled command name is used (e.g. "foo:bar" for a subcommand "foo bar"). An error is raised if the same table name is claimed by more than one command.

At knit_done time a callback checks whether the table already exists with the correct schema. If absent it is created; if the schema has changed it is migrated. The table always has an "id" (uuid) column first, followed by all required parameters, optional parameters, flags, and outputs, each group sorted alphabetically.

Example:

knit_register "run" my_func "Run an experiment."
knit_with_required "count:integer" "Number of iterations."
knit_with_table           # uses table name "run"
knit_with_table "my_runs" # uses table name "my_runs"
my_func() { ... }
knit_done

Parameters:

• table_name -- [in] Optional name of the database table. Defaults to the colon-separated command name.

int knit_with_url()

Download decorator: acquire the resource type by downloading a URL with curl. Declares the automatic parameters --url (default <url>) and the --uncompress flag (unpack the archive after download). Valid only on a resource type, at most one download decorator per type.

Parameters:

• url -- [in] The default URL of the artifact to download.

int knit_without_provenance()

Mark the command being registered as excluded from the provenance graph: an invocation of it records no "call" edge and is transparent when a command it invokes resolves its parent (the child links to the nearest participating ancestor instead). This is an explicit override of the default, so it silences a visible command. The mark also propagates to unmarked lexical descendants (e.g. "a" marked "without" silences an unmarked "a:b"). Data-row recording (knit_with_table) is orthogonal and still happens.

int knit_without_setup()

Declare that the job currently being registered opts out of the implicit "default" setup. Must be called between a knit_register* call and knit_done.

Jobs that declare neither knit_with_setup nor knit_without_setup run in the builtin "default" setup (see _knit_submit), inheriting the platform environment with no boilerplate. knit_without_setup makes such a job run with no setup at all — no setup directory, no platform activation. It is mutually exclusive with knit_with_setup.

Setups and wrappers are rejected. For any non-job command it is a no-op: plain commands and apps are already setup-less unless they explicitly declare knit_with_setup, so there is no implicit default to opt out of.

Variables

String KNIT_IGNORE_CHECKSUM

Public environment variable exported into a resource download body: "true" when checksum verification is disabled for this fetch, "false" otherwise. A download body reads it to decide whether to skip its own integrity check.

String KNIT_JOB_PREFIX

Public environment variable exported into a running job's environment: the absolute path of the job's own working directory, <job-root>/<uuid>. It is set by the generated job script (see _knit_sched_write_jobscript), which also makes it the process working directory, so a job body may either read it explicitly (e.g. to build an absolute output path) or rely on relative paths landing in it. It is unset outside a running job (on the login/submit side). The basename of this path is the job UUID, which is how compute-side callbacks recover their own row id in the jobs table.

ExportedString KNIT_LOG_LEVEL

Log level. Valid values: trace, debug, info, warning, error, critical.

String KNIT_MPI_LOCAL_RANK

Public environment variable exported into an app body: the node-local rank of this process (0-based within its node). Normalized by the framework from the active launcher, so an app body reads the same variable everywhere. It is 0 for a single-process run.

String KNIT_MPI_RANK

Public environment variable exported into an app body: the rank of this process in MPI_COMM_WORLD (0-based). Normalized by the framework from whichever launcher is in use (OpenMPI, MPICH/PMI, Slurm, PALS, or Flux), so an app body reads the same variable everywhere. It is 0 for a single-process run.

String KNIT_MPI_SIZE

Public environment variable exported into an app body: the number of processes in MPI_COMM_WORLD. Normalized by the framework from the active launcher, so an app body reads the same variable everywhere. It is 1 for a single-process run.

String KNIT_RESOURCE_EXPECTED_CHECKSUM

Public environment variable exported into a resource download body: the expected checksum of the resource when the caller requested one (empty when none was given). A download body may verify the fetched content against it.

String KNIT_RESOURCE_PREFIX

Public environment variable exported into a resource download body: the absolute path the body must create and populate with the fetched resource. Set by the framework before it runs the resource type's download body.

String KNIT_SCRIPT_NAME

Base name (without directories) of the experiment script that sourced knit.sh. Useful for constructing user-facing messages.

String KNIT_SCRIPT_PATH

Absolute path of the experiment script that sourced knit.sh. Used when generating batch scripts so the compute node can re-enter the experiment (exp.sh submit <job-name> ...) regardless of its current directory.

String KNIT_SETUP_PREFIX

Public environment variable exported into a setup body and every command that depends on the setup: the absolute path of the setup's own directory, where it should install its artifacts. A setup body reads it to build install paths (e.g. cmake --install "${KNIT_SETUP_PREFIX}"), and a dependent job or app sees it in its environment after the setup's activation runs. Set by the framework; unset when no setup is in effect.

ExportedReadOnlyString KNIT_VERSION

Version of the Knit framework. This committed value is a development placeholder: released artifacts are stamped with the exact git tag at build time (see the knit.sh target in the Makefile), so this file never needs editing when cutting a release.