Public API

Public API#

The public API is the stable interface of Knit. These functions and variables have names without a leading underscore and are intended to remain backwards compatible across releases.

Functions#

int knit()#

This is the main function that invokes the Knit framework. Users should call it as follows at the end of their bash script to forward all arguments to it.

knit $@

int knit_add_command_note()#

Append a line to the "Requirements" block shown in the command's "--help" output. A decorator uses it to document a precondition it installs (e.g. "Needs

a conda environment."). Fatal when no command is being registered.

Parameters:

text -- [in] The note line to append.

int knit_artifact()#

Bind a declared artifact of the currently executing command to a path inside the artifacts root. It is the file/directory counterpart of knit_output: the body first puts the file (or a symlink to it) under the artifacts root, then declares it here.

out="$(knit_artifact_dir)"
compute > "${out}/table.csv"
knit_artifact "table" "table.csv"

<linked-path> is either relative to the artifacts root or absolute and inside it. The entry's own location (not its target) must sit inside the artifacts root; a symlink entry is allowed and its target may be anywhere. The entry must exist and match the declared type (a symlink is followed to its target for this check). The content digest of the resolved target is recorded automatically in the companion "<name>-checksum" column, and the recorded value of the artifact is always the artifacts-relative path, whatever form was passed, so the record holds no absolute machine path. Binding the same artifacts-relative path twice in one invocation is fatal: artifacts are write-once.

The optional --link-from / --copy-from shortcuts create the entry from a source that lives elsewhere, so the body does not have to place it by hand. They are mutually exclusive. Both create the parent directories inside the artifacts root and refuse to overwrite an on-disk entry. --link-from resolves <real-path> to an absolute path and makes an absolute-target symlink at <linked-path> (the real bytes stay where they are, at zero copy cost); --copy-from does "cp -r" into <linked-path>. Either way <real-path> must exist, and the created entry then goes through the same existence, type, and checksum path as the direct-write form.

knit_artifact "dataset" "dataset.h5" --link-from /fast/aaa/bigfile
knit_artifact "figure"  "figure.svg" --copy-from "${out}/figure.svg"

Fails if called outside an executing command, if <name> is not a declared artifact of that command, if <linked-path> is empty or outside the artifacts root, if the entry does not exist or has the wrong type, if the path was already bound in this invocation, if both shortcuts are given, if a shortcut's <real-path> is missing or does not exist, or if a shortcut would overwrite an on-disk entry.

Parameters:

name -- [in] Artifact name (hyphens and underscores are interchangeable).
linked_path -- [in] Path inside the artifacts root where the entry lives.
--link-from -- [in] Optional; symlink <linked-path> to this <real-path>.
--copy-from -- [in] Optional; copy this <real-path> to <linked-path>.

int knit_artifact_dir()#

Print the resolved artifact root (see _knit_artifact_root) to stdout. This is the "write into artifacts/ then declare" helper: a command body puts a file (or a symlink to it) under this directory, then binds it with knit_artifact:

out="$(knit_artifact_dir)"
compute > "${out}/table.csv"
knit_artifact "table" "table.csv"

The directory is not created here; creation is lazy, on first bind. Called at most a handful of times per body, so it returns via stdout rather than a nameref.

int knit_as()#

Name a call so distinct invocations of the same command can be told apart in a query. Used at a call site as knit_as <alias> <cmd> …: it records <alias> on the provenance "call" edge of the delegated invocation, then runs knit <cmd> …. A later query addresses each call independently by its alias (an edge property). Without knit_as a call edge has a NULL alias.

knit_as fast run --procs 8 -- mcrank
knit_as slow run --procs 1 -- mcrank

The alias is one-shot: it lands only on the directly named call's edge, never on the nested edges that call's body records. It is validated at the call site: it must be non-empty, must not be a registered table name (which would collide with a node label in a query), and must not already have been used within the current invocation (two edges sharing an alias would be indistinguishable).

Parameters:

alias -- [in] The name to record on the call edge.
cmd -- [in] The command to invoke (followed by its arguments).
... -- [in] Arguments for the command.

int knit_bundle_auto_require()#

Record a path that Knit adds to the bundle on the experiment's behalf (see KNIT_BUNDLE_AUTO_REQUIRES). Called by a decorator — a generated knit_with<nm>_env directive in its file form (e.g. the built-in knit_with_spack_env), or a plugin-authored one — rather than directly by the experiment (which uses knit_bundle_requires). Like knit_bundle_requires it stores the string verbatim and MUST NOT touch the filesystem, so it stays safe when the experiment script is re-sourced on a compute node during job re-entry.

Parameters:

path -- [in] A file path, meant relative to the experiment script's directory.

int knit_bundle_requires()#

Declare an extra file, directory, or glob that "knit bundle" must add to the archive. Called at the top of an experiment script (like knit_set_program_description), not inside a knit_register/knit_done block. It describes the experiment as a whole, so it appends to the global _KNIT_BUNDLE_REQUIRES array rather than to any one command.

The path is stored verbatim. This function MUST NOT touch the filesystem — no stat, no existence check, no knit_fatal. The experiment script is re-sourced on every invocation, including on a compute node that re-enters a job, so a declaration that failed on a missing or non-relocatable path would abort that re-entry and kill the job. All validation (path exists, path is relative, path stays inside the tree) and glob expansion are deferred to "knit bundle".

Parameters:

path -- [in] A file, directory, or glob pattern, relative to the experiment script's directory.

int knit_check_arguments()#

Validate that a plain (non-registered) function received only expected arguments. This is the counterpart, for ordinary helper functions, of the validation that the command registration system performs automatically. It is intended for functions that parse their own "$@" with knit_get_parameter but don't go through knit_register.

The expected parameters are described by two space-separated lists: options (which take a value, as "--name value" or "--name=value") and flags (which do not). Everything from a literal "--" onwards is treated as extra arguments and is not validated. Hyphens and underscores in names are interchangeable, as elsewhere in the framework.

On the first unexpected argument the function logs an error attributed to the calling function and returns 1. It returns 0 if every argument is recognized.

Example:

_knit_submit_local() {
    local -a args=("$@")
    knit_check_arguments "stdout stderr stdin walltime" "" "${args[@]}" \
        || return 1
    ...
}

Parameters:

options -- [in] Space-separated names of options that take a value.
flags -- [in] Space-separated names of flags that take no value.
... -- [in] The arguments to validate (typically "$@").

int knit_check_subcommand_args()#

Resolve the given command-name tokens to a command (fatal when unknown), then validate the remaining tokens as that command's arguments: the required/typed argument checks (_knit_check_command_arguments) and the "--when" constraint checks (_knit_check_constraints), with the same fatal, standard messages the invoker produces. A dispatcher uses it to reject a bad dispatched invocation early, before it acts.

A wrapper target forwards its arguments verbatim and declares none, so there is nothing to validate; likewise a "--help" request is introspection, not an invocation. Both short-circuit to success.

Parameters:

... -- [in] The command-name tokens followed by the arguments to validate.

int knit_check_subcommand_exists()#

Return 0 when the given command-name tokens resolve to a registered command, 1 otherwise. Lazy subcommand discovery runs as an idempotent side effect (as a real invocation would), but no callbacks, body, or recording run. Non-fatal, so a dispatcher can branch on the result.

Parameters:

... -- [in] The command-name tokens.

Returns:

0 if the command exists, 1 otherwise.

int knit_command_has_marker()#

Return success when the named marker is set on the command currently being registered, failure otherwise. Fatal when no command is being registered.

Parameters:

name -- [in] The marker name.

int knit_command_outputs()#

Public accessor listing a command's declared output names, in declaration order. This is the stable surface plugins use to discover which outputs a command produces instead of reaching into the KNIT_CMD<cmd>_outputs set directly.

Parameters:

__knit_ret -- [out] Name of the array variable to populate.
cmd -- [in] Command to list (the mangled/registered name).

int knit_command_parameters()#

Public accessor listing a command's parameter names (normalized), in declaration order: required first, then optional, then flags. This is the stable surface the search proposers use to discover which parameters a command has, instead of iterating the KNIT_CMD<cmd>_{required,optional,flags} sets.

Parameters:

__knit_ret -- [out] Name of the array variable to populate.
cmd -- [in] Command to list (the mangled/registered name).

int knit_command_run_after()#

Install an after-callback on an already-registered command, by name. This is the by-name twin of knit_run_after (which targets the command currently being registered); it is what a plugin uses to hook a command it did not register, e.g.

knit_command_run_after run my_plugin_collect

It is sugar over "knit_extend <cmd>; knit_run_after <fn...>; knit_done": the callback is appended to the command's after-callback list, so an existing callback is kept and the new one runs after it.

Parameters:

name -- [in] Command name (hyphens/underscores/colons as the user types it).
... -- [in] Callback function and its arguments.

int knit_command_run_before()#

Install a before-callback on an already-registered command, by name. This is the by-name twin of knit_run_before (which targets the command currently being registered); it is what a plugin uses to hook a command it did not register, e.g.

knit_command_run_before bootstrap my_plugin_prepare

It is sugar over "knit_extend <cmd>; knit_run_before <fn...>; knit_done": the callback is appended to the command's before-callback list, so an existing callback is kept and the new one runs after it.

Parameters:

name -- [in] Command name (hyphens/underscores/colons as the user types it).
... -- [in] Callback function and its arguments.

int knit_command_set_marker()#

Set a named marker on the command currently being registered. Together with knit_command_has_marker this lets a decorator enforce "at most one" of itself: test the marker, fatal if already set, then set it. Fatal when no command is being registered.

Parameters:

name -- [in] The marker name.

int knit_command_table()#

Public accessor returning the table name declared for a command via knit_with_table. Returns an empty string when the command declared no table. This is the stable surface plugins use to learn which SQLite table holds a command's recorded rows.

Parameters:

__knit_ret -- [out] Name of the variable to hold the table name.
cmd -- [in] Command to query (the mangled/registered name).

int knit_copy_function()#

Define a function <new> with the same body as an existing function <old>, leaving <old> in place. This is the canonical way for a plugin to preserve a function before overriding it — e.g. to save a built-in decorator under a new name, then redefine the original to wrap it:

knit_copy_function knit_with_optional knit_orig_with_optional
knit_with_optional() { ...; knit_orig_with_optional "$@"; ...; }

Fatal when <old> is not a defined function. An existing <new> is overwritten.

Parameters:

old -- [in] Name of the function to copy from (must be defined).
new -- [in] Name of the function to define.

int knit_critical()#

Logging function for critical-level messages. Works like echo but will only print if the logging level was set to "critical".

Parameters:

... -- [in] Arguments for printf.

int knit_current_command()#: Print the name of the command currently being registered, in its colon-separated form (hyphens normalized to underscores). A decorator uses it to key per-command state or to build a message. Fatal when no command is being registered.

int knit_db_update_row()#

Update columns of an existing row, identified by its "id". Each assignment is a "column=value" string; the column is a knit name (normalized to underscores to match the schema). Used to record later state transitions of a recorded invocation (e.g. a job moving to "completed", or a plugin driving a long-running campaign row to its final status and best result). Public: the stable surface for a cross-invocation row update.

Parameters:

table -- [in] Table to update.
id -- [in] Value of the "id" column identifying the row.
... -- [in] One or more "column=value" assignments.

int knit_debug()#

Logging function for debug-level messages. Works like echo but will only print if the logging level was set to "debug".

Parameters:

... -- [in] Arguments for printf.

int knit_done()#: Finishes to register a function or a parameter set.

int knit_empty()#: Empty function to register commands with no behaviors.

int knit_ensure_subcommand_exists()#

Fatal twin of knit_check_subcommand_exists: resolve the given command-name tokens (running lazy discovery) and stop with a fatal error when they name no command. Returns 0 when the command exists.

Parameters:

... -- [in] The command-name tokens.

int knit_enum()#

Define a new enum type with the given name and possible values.

Example:

knit_enum "color" "red" "green" "blue"

Parameters:

name -- [in] Name of the enum type to define.
...values -- [in] Possible values for the enum.

int knit_enum_value()#

Add one or more values to the enum whose definition is open for extension. Used inside a "knit_extend <enum> ... knit_done" block (the enum-edit context established by knit_extend); the values are appended to the existing enum, so a plugin can grow a core or another plugin's enum (e.g. add a scheduler value) without redefining it. Duplicate values are ignored. It is a fatal to call this outside an enum-edit context.

Example:

knit_extend "__scheduler__"
knit_enum_value "flux-nested"
knit_done

Parameters:

...values -- [in] Values to add to the current enum.

int knit_enum_values()#

Print the possible values of an enum type. By default, values are separated by newlines. If a second argument is provided, it is used as the separator instead.

Example:

knit_enum_values "color"          # prints each value on its own line
knit_enum_values "color" ", "     # prints "red, green, blue"

Parameters:

name -- [in] Name of the enum type.
separator -- [in] Optional separator (default: newline).

Returns:

1 if the enum does not exist.

int knit_env_activate()#

Declare the activate verb: <fn> <env_dir> prints the shell lines that re-enter the environment (appended to the setup's .activate.sh).

Parameters:

fn -- [in] Name of the activate function.

int knit_env_done()#: Close an environment-manager registration. Asserts every mandatory verb and scalar was declared (fatal listing any that are missing), appends the manager to the registry, and — when the manager declared --options — reopens the bootstrap command (knit_extend) to declare each option as an optional with an auto-generated description. Clears the registration context.

int knit_env_ensure_provisioned()#

Ensure a manager's tool is present under .knit/<name>, provisioning it on demand when absent. This is the public entry point a setup's before-callback and a manager's own knit <name> wrapper use, since a setup may declare the manager's environment after bootstrap (which cannot be re-run). On-demand provisioning calls the provision verb with all options empty (latest).

A no-op when the tool is already present. Fatal for an unknown manager, or when the experiment is not bootstrapped (there is no .knit to provision into).

Parameters:

name -- [in] Manager name (any spelling).

int knit_env_install()#

Declare the install verb: <fn> <env_dir> <manifest_path> builds the environment into <env_dir> from the manifest, returning non-zero on failure.

Parameters:

fn -- [in] Name of the install function.

int knit_env_lockfile()#

Declare the basename of the lockfile (e.g. "spack.lock"): the reproducibility contract and the "environment is built" marker.

Parameters:

filename -- [in] Lockfile basename.

int knit_env_manifest()#

Declare the basename of the manifest file inside the environment directory (e.g. "spack.yaml").

Parameters:

filename -- [in] Manifest basename.

int knit_env_provision()#

Declare the provision verb and, optionally, the bootstrap options it consumes. The function is called at bootstrap (and on-demand) to ensure the tool is present under .knit/<name>, pinned to the given refs (each empty means "latest"). Its arguments are the declared --options values, positional and in declared order.

--options is a comma-separated list of "name:type" pairs (Knit's usual parameter type annotations). Each pair becomes an optional bootstrap option (declared by knit_env_done via knit_extend); the bootstrap provisioning dispatcher reads each one back and forwards it to the provision function.

Parameters:

fn -- [in] Name of the provision function.
--options -- [in] Optional "name:type,..." list of bootstrap options.

int knit_env_specs()#

Declare the specs verb: <fn> <spec...> prints a minimal manifest body for a spec list (the sugar backing knit_with_<name>_specs).

Parameters:

fn -- [in] Name of the specs-to-manifest function.

int knit_error()#

Logging function for error-level messages. Works like echo but will only print if the logging level was set to "error".

Parameters:

... -- [in] Arguments for printf.

int knit_extend()#

Reopen the registration context of an already-registered entity so the ordinary decorators apply to it again, then close the block with knit_done. This is the general primitive for growing an entity a plugin (or the core) already defined:

A command, setup, job, app, or wrapper (all registered through knit_register and sharing the same KNIT_CMD<name>* state): the command context (_KNIT_CURRENT_COMMAND / _FUNCTION / _DEMANGLED) is re-established and _KNIT_DONE_CBS is reset, so subsequent knit_with* / knit_run_* calls append to the existing parameter sets and callback arrays. Any done callbacks pushed in the extend block run at the closing knit_done; they must be idempotent (knit_with_table's _knit_db_setup_table is — it is create-or-migrate).
An enum: an enum-edit context is established (see _KNIT_CURRENT_ENUM) so knit_enum_value appends values to the existing enum.

Example:

knit_extend "bootstrap"                 # add an option + body hook to a command
knit_with_optional "spack:string" "" "Spack ref to provision."
knit_run_after _spack_bootstrap_after
knit_done

knit_extend "__scheduler__"             # add a value to an enum
knit_enum_value "flux-nested"
knit_done

A name that resolves to neither a registered enum nor a registered command is a fatal. The enum namespace is consulted first, by the raw name, so an enum name that would be altered by command-name mangling is matched as written.

Parameters:

name -- [in] Demangled command name or enum name to reopen.

int knit_extra_index()#

Print the index at which extra arguments start (i.e. arguments passed after "--" in the list of arguments). This index will be the size of the list if no extra arguments are found. The way this function can be used is as follows.

local args=("$@")
local extra_index=$(knit_extra_index "${args[@]}")
local extra=("${args[@]:extra_index}")

Parameters:

... -- [in] List of arguments.

int knit_fatal()#

Logging function for fatal error messages. Will be printed no matter the log level, and the program will exit with an error value. Any callbacks registered with knit_on_fatal are run first (see the drain below).

Parameters:

... -- [in] Arguments for printf.

int knit_framed()#

Piping a command's output into knit_framed will make the output appear inside a frame with the given height and width. If stdout is not a TTY, stdin is forwarded to stdout unchanged.

Usage: my-command arg1 arg2 ... | knit_framed [<height> [<width>]] [--title <title>] [--cleanup]

Parameters:

height -- [in] Height of the frame (default: terminal height).
width -- [in] Width of the frame (default: terminal width).
--title -- [in] Optional title displayed centered on the top border.
--cleanup -- [in] If present, erase the frame from the screen after completion.
--frame-color -- [in] Foreground color of the frame borders (key in _KNIT_COLORS).
--frame-bg-color -- [in] Background color of the frame borders (key in _KNIT_COLORS).
--text-color -- [in] Foreground color of the text inside the frame (key in _KNIT_COLORS).
--text-bg-color -- [in] Background color of the text inside the frame (key in _KNIT_COLORS).
--log-level -- [in] Minimum log level required to display the frame (trace, debug, info, warning, error, critical). If KNIT_LOG_LEVEL is above this level, stdin is drained silently and nothing is output.

int knit_framed_run()#

Run a command with its combined stdout/stderr written to _KNIT_TRACE_FILE and, when KNIT_LOG_LEVEL is trace, also displayed live in a 10-line frame. Returns the exit status of the command. Used by every subsystem that provisions or fetches over the network (Spack provisioning, plugin import, knit_github_download); public so a plugin can frame its own long steps the same way.

Parameters:

title -- [in] Title shown on the frame's top border.
... -- [in] Command and arguments to execute.

Returns:

The exit status of the command.

int knit_get_invocation_args()#

Copy the current command's pending (expanded) argument list into the caller's array. Meant for a before-callback: it returns exactly the invocation args, with no offset to track (unlike the callback's own "$@", which is prefixed by the arguments the callback was registered with), and it reflects edits a previous before-callback made. Returns an empty list outside a before-callback.

Parameters:

args_out -- [out] Name of the array to fill with the invocation args.

int knit_get_parameter()#

Search the list of arguments for a specific parameter. If found, the function will print the value associated with the parameter (flags will lead to this function printing "true" or "false"). If not found, this function will print nothing and return 1.

The parameter value may be supplied either as two tokens ("--name value") or inline with an equals sign ("--name=value"); both forms are recognized.

Parameters:

param -- [in] Parameter to search for (without the -- prefix).
... -- [in] Arguments in which to search for the parameter.

int knit_get_profile_field()#

Extract one field from the bootstrapped experiment's profile using a jq path expression. Reads the resolved profile JSON frozen at bootstrap in the "__profile_json__" metadata. Prints the field value (without enclosing quotes for strings) or an empty string when no profile is configured or the field is absent.

Parameters:

jq_path -- [in] jq path expression, e.g. '.scheduler.type'.

int knit_github_api()#

Fetch a GitHub REST API URL and print the response body. When GITHUB_TOKEN or GH_TOKEN is set, the request is authenticated: the anonymous api.github.com limit is 60 requests per hour per address, shared by every process behind that address, so continuous-integration runners exhaust it and knit sees an error body instead of JSON. An authenticated request uses the far higher per-token limit. curl also retries a few times so a transient network error does not abort the caller, and "-f" makes an HTTP error status a clean failure (empty output) instead of a non-JSON body that would confuse the caller's parser. Public so a plugin can query the API (for example to resolve the latest release of a tool it provisions).

Parameters:

url -- [in] The api.github.com URL to fetch.

Returns:

Prints the response body; non-zero on a hard failure or HTTP error.

int knit_github_download()#

Download a GitHub repository at a specific commit as a tarball, compute the tarball's sha256, and extract it into the destination directory. Uses curl and tar (as the sqlite/jq provisioning does) so knit has no git dependency. GitHub serves any commit at archive/<sha>.tar.gz, whose single top-level "<repo>-<sha>/" directory is stripped so files land directly in the destination.

The download and extraction are framed with knit_framed_run (shown live when the log level is trace). The commit should be an exact SHA (see knit_github_resolve_commit) so the download is reproducible. This is public so a plugin that provisions a tool from GitHub reuses the same no-git path.

Parameters:

sha256_out -- [out] Nameref that receives the tarball's sha256 hex digest.
org -- [in] Repository owner (organization or user).
repo -- [in] Repository name under that owner.
dest -- [in] Destination directory.
sha -- [in] Commit SHA to download.

int knit_github_download_asset()#

Download one published asset of a GitHub release to a destination file, compute its sha256, and return the digest by nameref. The stable release-asset URL (releases/download/<tag>/<asset>) is built directly, so no API call is needed once the tag and asset name are known. Uses curl (as the other GitHub helpers do) so knit needs no git, and frames the transfer with knit_framed_run (shown live at trace level). Release assets are public, so no token is sent.

The transfer retries on a transient failure (connection reset, 5xx, timeout), like knit_github_api: an asset download is a large transfer over a possibly flaky network (CI runners especially), and a single blip must not abort a whole bootstrap. --retry-all-errors covers connection-level failures that --retry alone would not; the connect/low-speed timeouts turn a stalled transfer into a retryable failure rather than an indefinite hang.

This complements knit_github_download, which fetches a repository source archive by commit. A tool shipped as a prebuilt per-platform release binary (for example uv) is fetched with this function instead. The destination is a file (the asset verbatim); the caller unpacks it as the asset format requires, since a release asset may be a tarball, a zip, or a raw binary. The tag should name an immutable published release so the download is reproducible; pair the returned digest with a recorded checksum to pin the asset.

Parameters:

sha256_out -- [out] Nameref that receives the asset's sha256 hex digest.
org -- [in] Repository owner (organization or user).
repo -- [in] Repository name under that owner.
tag -- [in] Release tag the asset belongs to (e.g. "0.5.0").
asset -- [in] Asset filename (e.g. "uv-x86_64-unknown-linux-gnu.tar.gz").
dest -- [in] Destination file path for the downloaded asset.

int knit_github_resolve_commit()#

Resolve a ref (tag, branch, or commit SHA) to its exact commit SHA via the GitHub API. Resolving upstream gives a single download path (the archive URL takes a SHA) and the exact commit for provenance, without needing git. Public so a plugin pins a moving ref it provisions to an exact commit.

Parameters:

org -- [in] Repository owner (organization or user).
repo -- [in] Repository name under that owner.
ref -- [in] Tag, branch, or commit SHA to resolve.

Returns:

Prints the commit SHA; fatal if it cannot be resolved.

int knit_hidden()#

Mark a command as hidden, i.e. it will not appear in usage help messages. This is the static, unconditional hide flag (the "_is_hidden" boolean), also read by _knit_provenance_enabled and describe, and it stands apart from the dynamic knit_hidden_if predicates.

knit_hidden and knit_hidden_if are mutually exclusive per command (last-writer wins with a warning): calling knit_hidden after one or more knit_hidden_if statements shadows them, so it emits a warning, sets the static flag, and drops the collected dynamic predicates.

int knit_hidden_if()#

Declare that the command currently being registered is hidden from its parent's "--help" whenever <predicate> returns 0. <predicate> is the name of a user-defined shell function that receives the demangled command name as its single argument and returns 0 ("hide") or non-zero ("show"). Unlike knit_hidden this is a dynamic, "--help"-only hide: the command remains fully invokable and is still visible to _knit_provenance_enabled and describe.

Repeatable: multiple calls register multiple predicates and the command is hidden if any of them returns 0 (logical OR).

knit_hidden and knit_hidden_if are mutually exclusive per command: if the command is already statically hidden (knit_hidden was called first), the dynamic predicate would be meaningless, so this call emits a warning and is ignored. The per-command storage array (_hidden_pred) is declared lazily on first use.

Parameters:

predicate -- [in] Name of the predicate function.

int knit_hidden_if_not_usable()#

Shorthand for a knit_hidden_if whose predicate hides the command from "--help" exactly when at least one of its knit_usable_if predicates is false. Takes no arguments. Backed by the internal predicate _knit_hidden_if_not_usable_pred, appended to the command's _hidden_pred array; a command with no knit_usable_if predicates is always usable, hence never hidden by this shorthand.

Subject to the same mutual exclusion with knit_hidden as knit_hidden_if.

int knit_highlight_if()#

Declare that the command currently being registered has its name highlighted (bold) in its parent's "--help" whenever <predicate> returns 0 and the output is a terminal. <predicate> is the name of a user-defined shell function that receives the demangled command name as its single argument and returns 0 ("highlight") or non-zero ("plain"). Highlighting is purely cosmetic: it never affects invokability, provenance, or describe.

Repeatable: multiple calls register multiple predicates and the command is highlighted if any of them returns 0 (logical OR). The per-command storage array (_highlight_pred) is declared lazily on first use.

Parameters:

predicate -- [in] Name of the predicate function.

int knit_import()#

Import a plugin: resolve its spec to an exact commit, ensure that commit's code is available, and source its plugin.sh so its commands and decorators become available. Call it at the top of the experiment, after "source knit.sh" and before any use of what the plugin provides.

Once the experiment is bootstrapped the code is sourced from the pinned tree under ".knit/plugins/<id>/<commit>/" — so a re-run, and a submitted job that re-enters the experiment on a compute node sharing .knit, need no network and no personal cache. Only when that tree is absent is the commit fetched into the shared "$HOME/.cache" staging area and (when bootstrapped) copied under .knit; before bootstrap, with no .knit yet, the import is sourced from the cache.

A plugin is sourced once per canonical id ("<host>/<org>/<repo>"); a repeated import of the same id dedups. Importing the same id at two different explicit refs is a fatal version conflict. A plugin's own plugin.sh may call knit_import for its dependencies; because sourcing is synchronous, a dependency is fully sourced before the dependent continues.

Resolution prefers a lockfile pin, so a fresh checkout resolves deterministically even before bootstrap; a mutable ref resolved for the first time is written back to the lockfile. When the plugin's code cannot be obtained (offline, with no cached commit and no pin), the import is deferred rather than fatal: it is recorded for the describe/--help footer and a command_not_found safety net keeps the experiment source loading.

See also

KNIT_OFFLINE=1 forces cache-only resolution (never touch the network).

Parameters:

spec -- [in] The import spec (see _knit_plugin_parse_spec for the forms).
--sha256 -- [in] <hash> Optional hard pin: fail unless the fetched tarball matches this sha256.

int knit_info()#

Logging function for info-level messages. Works like echo but will only print if the logging level was set to "info".

Parameters:

... -- [in] Arguments for printf.

int knit_input_artifact_path()#

Resolve an input-artifact parameter value (an artifacts-relative path) into its absolute on-disk location under the experiment's artifacts root and print it to stdout. This is how the body of a command that consumes an artifact turns the parameter value into a usable path:

csv="$(knit_input_artifact_path "$(knit_get_parameter input_table "$@")")"

The value must resolve inside the artifacts root (the same containment rule knit_artifact enforces on a produced entry); an absolute path outside it is fatal. Fatal too when the value is empty or the resolved entry does not exist, since a body should never run against an artifact that is not present. Called at most a handful of times per body (one per declared input artifact), so it returns via stdout rather than a nameref. Mirrors knit_resource_path.

Parameters:

value -- [in] The input-artifact parameter value (an artifacts-relative path).

int knit_input_artifact_paths()#

Resolve a variadic input-artifact parameter value (a comma-separated list of artifacts-relative paths, each possibly a glob) into an array of absolute on-disk locations, filling a caller-named bash array. This is how the body of a command that consumes a variadic ("name:kind*" / "name:kind+") input turns the parameter value into usable paths, iterated safely — no comma splitting, so a path that holds a space stays intact:

local -a tables
knit_input_artifact_paths tables "$(knit_get_parameter tables "$@")"
for csv in "${tables[@]}"; do cat "${csv}" >> combined.csv; done

The raw value is resolved with the shared list resolver (comma split, then glob expansion relative to the artifacts root under nullglob, first-seen de-duped, in order), the same one the before-callback validated against and the after-callback recorded edges for, so the body's view cannot drift from validation and lineage. Each resolved element is turned into its absolute path under the artifacts root; an element that resolves outside that root is fatal (the same containment rule knit_input_artifact_path enforces). An empty value fills the array with no elements. The scalar knit_input_artifact_path is unchanged and still serves a scalar input (one path, printed to stdout).

This returns by nameref (not stdout) because the result is an array that may hold paths with spaces, which a command-substitution capture would re-split. Its internal locals are all "__"-prefixed: it is public API, so a caller may name the output array anything, and a plain internal sharing that name would shadow the nameref (see the project's nameref-shadow-collision note).

Parameters:

__knit_ret -- [out] Name of the array variable to fill with absolute paths.
__raw -- [in] The raw parameter value (comma-separated list of paths).

int knit_job_hostnames()#

Print the hostnames the current job is running on, discovered from the active scheduler backend (see _knit_sched_hostfile). Intended to be called inside a job body. Outside a scheduler allocation it reports the local hostname.

By default each host is printed once, on its own line, with any trailing ":N" slot-count or extra columns removed and duplicates collapsed (first-seen order preserved). Use --raw to print the backend's hostfile entries verbatim (e.g. one line per launchable slot), which is the form an MPI launcher hostfile wants.

Usage: knit_job_hostnames [--json] [--separator <sep>] [--raw] [--select <s>:<n>]

Parameters:

--json -- [in] Print the hostnames as a JSON array of strings.
--separator -- [in] Separator used to join hostnames (default: a newline). Ignored when --json is given.
--raw -- [in] Print the raw hostfile entries verbatim (no ":N" stripping, no deduplication); only the separator / JSON wrapping is applied.
--select -- [in] Print only a slice of the resulting list: <start>:<length>, where <start> is a 0-based index and <length> is the number of hostnames to print. The slice is taken after the raw or deduplication step, so it counts the entries that would otherwise be printed. Out-of-range requests are clamped (a start past the end yields nothing).

int knit_job_nodecount()#

Print the number of distinct nodes allocated to the current job: the count of deduplicated hostnames reported by knit_job_hostnames (so a host contributing several launchable slots is counted once). Intended to be called inside a job body; outside a scheduler allocation it reports 1 (the local host). A common use is to launch one rank per node: knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- <app>.

Takes no arguments.

int knit_list_profiles()#

Print the union of the profiles known to knit, one per line in sorted order, laid out like a command's parameters in "--help": a name column, a "[source]" annotation, then the profile's description word-wrapped with a hanging indent under the annotation column. The source is the committed in-repo index (fetched from the default branch) or the admin store under _KNIT_PROFILE_ADMIN_DIR; an admin profile that shares a name with a repo one shadows it (§4.5) and is marked accordingly (its description wins). The repo index is fetched best-effort so an offline, admin-only machine still lists its own profiles.

Hidden profiles (JSON "_hide": true) are omitted by default. When show_hidden is "true" they are included and annotated ", hidden" after the source.

Wrapping reuses _knit_terminal_width / _knit_help_render_entry (the same helpers "--help" uses), so alignment, wrap-around, and the pipe/redirect single-line fallback all match option listings.

Parameters:

show_hidden -- [in] "true" to also list hidden profiles (default "false").

int knit_log_set_level()#

Set the log level. The level should be either trace, debug, info, warning, error, or critical.

Parameters:

level -- [in] Log level.

Returns:

0 if the log level was set, 1 otherwise.

int knit_no_record_on_failure()#

Mark the command currently being registered so that a failed invocation records no database row. By default a command with a table (knit_with_table) records a row after its body runs, whatever the body's exit status; with this directive a non-zero body exit skips recording, so a failed run leaves no data row behind. Use it for a command that removes its own partial output on failure (as knit fetch removes a partial resource instance), where a dangling row would misrepresent a run that produced nothing.

Must be called between a knit_register* call and knit_done. It is not valid on a job (knit_register_job): a job's row carries a "state" column written from its callbacks and signal traps (running / killed / completed), which suppressing a failed job's row would drop. It is not valid on a wrapper (knit_register_wrapper) either, which forwards its arguments verbatim and has no body of its own. Calling it more than once on the same command is harmless.

int knit_on_fatal()#

Register a callback to run when knit_fatal is about to abort the process. The arguments form one command (and its arguments), stored pre-escaped with printf q (as knit_run_before / knit_run_after store their callbacks) in the global _KNIT_ON_FATAL list, and re-run by knit_fatal's drain just before it exits.

This exists so a compute-side job can record a terminal state before a fatal abort tears the process down: the normal after-callback (which records the state) never runs when the body or a before-callback exits via knit_fatal.

Parameters:

... -- [in] The callback command and its arguments.

int knit_output()#

Record a named output value from within a registered command function. Fails if called outside of an executing command, if the name was not declared with knit_with_output, or if the value does not match the declared type.

Parameters:

name -- [in] Output name (hyphens and underscores are interchangeable).
value -- [in] Value to record.

int knit_output_type()#

Public accessor for the declared type of a command's output. The output name is normalized (hyphens and underscores are interchangeable). Returns the empty string and status 1 when the command has no such output. This is the stable surface plugins use instead of reaching into the KNIT_CMD<cmd>3<out>_type internals.

Parameters:

__knit_ret -- [out] Name of the variable to hold the type.
cmd -- [in] Command the output belongs to (the mangled/registered name).
output -- [in] Output name (hyphen or underscore spelling).

Returns:

0 if the output exists, 1 otherwise.

int knit_parameter_set()#

Begins the definition of a named parameter set. A call to this function should be followed by any number of knit_with_required, knit_with_optional, and knit_with_flag calls, then a call to knit_done. The resulting set can then be imported into one or more commands with knit_with_parameter_set.

Parameters:

name -- [in] Name of the parameter set (letters, digits, hyphens, underscores).

int knit_parameter_space()#

Public accessor for a command parameter's declared search domain, returned in structured form. This is the lazy parse+validate point: it reads the raw descriptor stored at declaration (the <ns>2_domain token and its _domain_log flag), parses it with the shared grammar, and validates it against the parameter's declared type (so a type x kind error surfaces here, on first read, not at declaration). The parameter name is normalized (hyphen or underscore spelling). Returns status 1 with all outputs cleared when the parameter has no declared domain.

The output arguments come first, in the order kind, type, lo, hi, values, step, log, glob; then the command and parameter.

Parameters:

__knit_ret1 -- [out] Domain kind ("list", "range", or "glob").
__knit_ret2 -- [out] The parameter's declared type.
__knit_ret3 -- [out] Range lower bound (empty unless kind=range).
__knit_ret4 -- [out] Range upper bound (empty unless kind=range).
__knit_ret5 -- [out] Array of list values (empty unless kind=list).
__knit_ret6 -- [out] Range step (empty when unset or kind!=range).
__knit_ret7 -- [out] Log-scale flag ("true"/"false").
__knit_ret8 -- [out] Glob pattern (empty unless kind=glob).
cmd -- [in] Command the parameter belongs to (the mangled/registered name).
param -- [in] Parameter name (hyphen or underscore spelling).

Returns:

0 if the parameter has a declared domain, 1 otherwise.

int knit_parameter_space_json()#

Print a command parameter's declared search domain as a compact JSON object (one fork), or the literal "null" when the parameter has no declared domain. The object has the keys kind, type, lo, hi, step, glob (strings), log (boolean), and values (array of strings). A convenient read for "describe

--space" and the Bayesian-optimization plugin. Like knit_parameter_space, it parses and validates on read, so an invalid declared domain is fatal here.

Parameters:

cmd -- [in] Command the parameter belongs to (the mangled/registered name).
param -- [in] Parameter name (hyphen or underscore spelling).

int knit_parameter_type()#

Public accessor for the declared type of a command's parameter. The parameter name is normalized (hyphens and underscores are interchangeable). Returns the empty string and status 1 when the command has no such parameter. This is the stable surface the search proposers (and any plugin) use instead of reaching into the KNIT_CMD<cmd>2_type internals.

Parameters:

__knit_ret -- [out] Name of the variable to hold the type.
cmd -- [in] Command the parameter belongs to (the mangled/registered name).
param -- [in] Parameter name (hyphen or underscore spelling).

Returns:

0 if the parameter exists, 1 otherwise.

int knit_platform_name()#: Print the platform name recorded for the bootstrapped experiment (the "__platform__" metadata key). This is the value of bootstrap's --platform option, or, when that was omitted and a --profile was given, the profile's own "name" field (e.g. "anl/polaris"). Prints an empty string when no platform was recorded.

int knit_popd()#: Silent version of popd.

int knit_prefix()#

Print Knit's runtime directory (the experiment's ".knit" directory) to stdout. This is the public accessor over the private _KNIT_PREFIX; a plugin uses it to locate where to write its own logs, caches, or materialized trees, e.g.

local mydir
mydir="$(knit_prefix)/my-plugin"
mkdir -p "${mydir}"

The path is always set (it defaults to "<cwd>/.knit" before bootstrap), and is returned whether or not the directory yet exists; a caller that needs it present should create it.

int knit_progress_bar()#

Draw one full-width framed progress bar live to stderr, in place. A no-op unless stderr is a TTY and KNIT_LOG_LEVEL <= info (see _knit_progress_gate). Call once per observable state change; call knit_progress_bar_finish once at the end.

Parameters:

title -- [in] Title shown in the frame's top rule.
completed -- [in] Number of completed items.
running -- [in] Number of running items.
failed -- [in] Number of failed/killed items.
remaining -- [in] Number of not-yet-started items.

int knit_progress_bar2()#

Draw two stacked framed progress bars sharing one frame, live to stderr, in place. Each bar is a labelled interior row; the legend and two labelled count lines sit below the frame. A no-op under the same gate as knit_progress_bar. Used for the two-budget case (e.g. a "current" bar and a "campaign" bar).

Parameters:

title -- [in] Title shown in the frame's top rule.
c1 -- [in] r1 f1 rem1 Counts for the first bar.
label1 -- [in] Label for the first bar.
c2 -- [in] r2 f2 rem2 Counts for the second bar.
label2 -- [in] Label for the second bar.

int knit_progress_bar_finish()#: Finish a progress-bar sequence: reset the redraw state and emit a trailing newline so a following knit_info prints on a fresh line below the final frame. A no-op under the same gate as knit_progress_bar.

int knit_provides_launcher()#

Declare that the setup currently being registered may supply an MPI launcher on a machine that has none. Must be called between knit_register_setup and knit_done, at most once per setup. A bare directive (the only form for now): it means "this setup may provide a launcher where the machine offers one of its

own" — it sits

below a concrete machine launcher (launcher) in the precedence, so a profile's launcher still wins (see _knit_launch_backend). A deliberate override is future work.

The launcher is detected once at setup-build time (not at run time) by the after-callback and frozen into the setup's .activate.sh as KNIT_PROVIDED_LAUNCHER; it is recorded as the mpi_launcher provenance output (a column in the setup's table). Setups are the only valid target: wrappers and non-setup commands are rejected.

Example:

knit_register_setup "juliaenv" _juliaenv "Build against MPI."
knit_with_spack_specs "cmake" "mpi"
knit_provides_launcher            # may launch where the machine has no MPI
_juliaenv() { ... }
knit_done

int knit_push_done_cb()#

In the context of a knit_register, push a callback to be called at the next call to knit_done. Multiple callbacks may be pushed; they are all called in reverse order of installation. The callback list is cleared after knit_done.

Parameters:

... -- [in] Callback function and its arguments.

int knit_pushd()#: Silent version of pushd.

int knit_register()#

Register a function for use with a CLI. A call to this function should be followed by any number of knit_with_* calls, followed by the declaration of the function to register, then a call to knit_done.

Parameters:

cmd -- [in] Command (demangled).
name -- [in] Name of the function to register.
description -- [in] Description of the command.

int knit_register_app()#

Register an app, i.e. a subcommand of the "run" command that executes as an MPI launch inside a job. Mirrors knit_register_job: it registers run:<name>, backs it with a per-app table named after the app, records the name in _KNIT_APPS, and installs a before-callback asserting the run context. An app inherits the ambient setup of the surrounding job by default, but may declare knit_with_setup to explicitly depend on and re-source a setup. There is no after-callback.

A call to this function must be followed by any knit_with_* declarations, the definition of <fn>, and a call to knit_done.

Example:

knit_register_app "hello" "hello_fn" "Hello world MPI app."
knit_with_optional "n:integer" "100" "Problem size."
hello_fn() {
  ...
}
knit_done

Parameters:

name -- [in] Short name for the app (used as the subcommand name).
fn -- [in] Name of the Bash function implementing the app.
description -- [in] One-line description shown in --help.

int knit_register_artifact()#

Declare an artifact kind: a semantic category an artifact may carry (e.g. "csvfile", "rundir"), backed by a physical type ("file" or "directory"). A top-level declaration, like knit_enum — not called between knit_register and knit_done. Once declared, a kind may be referenced by knit_with_output_artifact and knit_with_input_artifact with the "name:kind" syntax.

The builtin kinds "file", "directory", and the "dir" alias are pre-registered (each backed by its matching type), so the common case needs no call here and re-registering one of them is fatal.

Example:

knit_register_artifact "csvfile:file"     "Tabulated result in CSV format."
knit_register_artifact "rundir:directory" "A run's output directory."

Fatal when the annotation has no ":type", when the kind name is invalid, when the kind is already registered, or when the type is not a checksummable ("file"/"directory") type.

Parameters:

spec -- [in] Kind name followed by ":type" ("file" or "directory").
description -- [in] Optional one-line description for knit describe / --help.

int knit_register_env_manager()#

Open an environment-manager registration. Must be followed by the mandatory knit_env_* declarations (provision, install, activate, specs, manifest, lockfile) and closed with knit_env_done. A manager's per-registration state is kept in KNIT_ENV<nm>* globals, where <nm> is the normalized (underscore) name; the as-typed name is preserved in _KNIT_ENV<nm>_display and is used for the on-disk directory (.knit/<name>) and for messages.

Fatal when called inside an already-open registration, when the name is not a valid Knit name, or when a manager with the same normalized name is already registered.

Parameters:

name -- [in] Short name for the manager (e.g. "spack", "conda").
description -- [in] One-line description.

int knit_register_job()#

Register a job, i.e. a subcommand of the "submit" command that executes as a job sumitted on the supercomputer.

A call to this function must be followed by any knit_with_* declarations, the definition of <fn>, and a call to knit_done.

The names "prepared", "next", and "from" are reserved: "submit prepared" and "submit next" release prepared jobs and "prepare from" reads a plan, so a job registered under one of these would mangle to the same command name and shadow the release/plan subcommand. Registering such a job is fatal.

Example:

knit_register_job "hello" "hello_fn" "Hello world job script."
knit_with_optional "name:string" "Matthieu" "Name of the person to greet."
hello_fn() {
  ...
}
knit_done

Parameters:

name -- [in] Short name for the job (used as the subcommand name).
fn -- [in] Name of the Bash function implementing the job.
description -- [in] One-line description shown in --help.

int knit_register_resource()#

Register a resource type: a downloadable input artifact acquired through the knit fetch dispatcher. Unlike knit_register_setup there is no user body to supply — the type is backed by the shared _knit_resource_fetch_body, which dispatches on the declared download method. The type is backed by a database table named "resource:<type>" (via knit_with_table) and recorded in the resource registry so knit_with_resource can validate a declared dependency.

A call to this function must be followed by exactly one download decorator (knit_with_git / knit_with_url / knit_with_local), an optional knit_with_checksum, and a call to knit_done. It must run before any knit_with_resource that references the type.

Example:

knit_register_resource "julia_code" "Julia fractal source."
knit_with_git "https://github.com/knit-sh/julia-fractal-example.git" "main"
knit_done

Parameters:

type -- [in] Short name for the resource type (used as the dispatch target).
description -- [in] One-line description shown in --help.

int knit_register_setup()#

Register a setup, i.e. a subcommand of the "setup" command that builds software and records the resulting environment. The setup is automatically backed by a database table named "setup:<name>". A before-callback checks that KNIT_SETUP_PREFIX is set; an after-callback saves the environment to $KNIT_SETUP_PREFIX/.activate.sh.

A call to this function must be followed by any knit_with_* declarations, the definition of <fn>, and a call to knit_done.

Example:

knit_register_setup "hello" "hello_fn" "Builds the hello program."
knit_with_optional "version:string" "main" "Branch or tag to check out."
hello_fn() {
  ...
}
knit_done

Parameters:

name -- [in] Short name for the setup (used as the subcommand name).
fn -- [in] Name of the Bash function implementing the setup.
description -- [in] One-line description shown in --help.

int knit_register_shorthand()#

Define a new "@" shorthand that forwards verbatim to a function. A plugin uses it to give its own decorator an "@" twin, e.g. "knit_register_shorthand with_conda_env knit_with_conda_env" defines "@with_conda_env". Because the source-time generator (_knit_shorthand_generate) runs before any plugin is sourced, a plugin cannot extend the curated map and must declare its shorthand through this seam instead.

The shorthand is a pass-through: "@<token>()" forwards "$@" to <fn>. It honours KNIT_WITHOUT_SHORTHAND (the token being opted out, or "all", makes this a no-op) exactly like the built-in shorthands.

Fatal when: the token is not a valid identifier; the token names a built-in shorthand (those cannot be redefined); an "@<token>" is already defined (a second registration of the same token); the target name is not a valid identifier; or the target function is not defined.

Parameters:

token -- [in] The shorthand token (the part after "@").
fn -- [in] The function the shorthand forwards to.

int knit_register_wrapper()#

Register a wrapper command: a command that forwards all of its arguments verbatim to an underlying command (e.g. "knit spack ..." forwarding to the knit-installed spack). A wrapper differs from a regular command in that it:

cannot declare parameters or outputs (knit_with_required/optional/flag/ output/dispatch/parameter_set are fatal);
performs no argument validation, expansion, or --when checking;
forwards "$@" verbatim to <fn>, including "--help" and "--";
may declare a table with knit_with_table, in which case the whole forwarded command line is recorded in a single "args" column;
may install before/after callbacks, which still run around <fn>.

A call to this function must be followed by the definition of <fn>, an optional knit_with_table, and a call to knit_done.

Example:

knit_register_wrapper "spack" "_knit_spack" "Wrapper for the spack command"
_knit_spack() { spack "$@"; }
knit_done

Parameters:

name -- [in] Command name (used as the wrapper's invocation name).
fn -- [in] Name of the Bash function the wrapper forwards to.
description -- [in] One-line description shown in "--help".

int knit_reject_if_wrapper()#

Fatal if the command currently being registered is a wrapper. Called by the declaration functions that a wrapper is not allowed to use (parameters, outputs, dispatch, parameter sets). A parameter-set definition is never a wrapper, so the check is skipped when no command is being registered.

Parameters:

directive -- [in] Name of the calling directive (for the error message).

int knit_rename_function()#

Copy an existing function <old> to <new>, then remove <old>. Like knit_copy_function but moves the function out of the way entirely rather than leaving a second copy behind. Fatal when <old> is not a defined function.

Parameters:

old -- [in] Name of the function to rename (must be defined).
new -- [in] New name for the function.

int knit_resolve_subcommand()#

Resolve a sequence of command-name tokens to the mangled command name and the count of tokens consumed, running lazy subcommand discovery as a side effect. Shares the invoker's resolution semantics exactly (greedy non-"--" tokens, stop at the first option or wrapper). Fatal when the tokens name no command; use knit_check_subcommand_exists first for a non-fatal test.

A plugin dispatcher uses this to learn the real target of a "parent -- <leaf>

..." style invocation before it acts on it.

Parameters:

cmd_out -- [out] Nameref receiving the resolved mangled command name.
consumed_out -- [out] Nameref receiving the consumed token count.
... -- [in] The command-name tokens (optionally followed by args).

int knit_resource_path()#

Resolve a resource instance name into its absolute directory under the experiment's resource root: <resource-root>/<name> (see _knit_resource_root), and print it to stdout. This is how the body of a command that depends on a resource turns a resource parameter value (the instance name) into an on-disk path:

train_dir="$(knit_resource_path "$(knit_get_parameter training_dataset "$@")")"

The name is validated as a single path component first (fatal otherwise). Fatals when the named instance does not exist, since a body should never run against a resource that was never fetched. Called at most a handful of times per body (one per declared resource), so it returns via stdout rather than a nameref.

Parameters:

name -- [in] The resource instance name (as passed to knit fetch --name).

int knit_run_after()#

In the context of a knit_register, install a callback to run after the command currently being registered.

Example:

knit_register ...
knit_run_after echo "Running after command"

int knit_run_before()#

In the context of a knit_register, install a callback to run before the command currently being registered.

Example:

knit_register ...
knit_run_before echo "Running before command"

int knit_search_axis_qualify()#

Compose a command-qualified axis identity from a segment and a bare option name: "<segment>.<option>". This is the uniform spelling a search proposer uses to label an axis (e.g. in a recorded dimensions map, an Ax parameter name, or an error message) so the same option name in two segments ("submit.nodes" vs "job.nodes") stays distinct. The "." separator is used because the identity is passed verbatim to Ax as a parameter name and ":" is reserved by knit's command-name mangling.

Parameters:

__knit_ret -- [out] The qualified identity "<segment>.<option>".
segment -- [in] The command segment the axis belongs to.
option -- [in] The bare option name.

int knit_search_group_forward()#

Forward a campaign --group label into one trial's downstream command line, in place, so a job trial lands in the existing jobs.group column (which makes "submit next --group" and "remove --group" operate on the campaign with no new code). The group is a submission field, so it is inserted among the submit options -- right after the leading "submit" or "prepare" token, before its "--". Both are job dispatchers that share the submit option surface (and so accept --group). The line is left unchanged when the group is empty or the line is neither a submit nor a prepare line (a "run ... -- app" or a bare command has no scheduler, so its group lives only on the parent row and the call edges).

This is an optional search-core helper for a proposer that wants its trials to carry the campaign group (the ax optimizer uses it so a campaign's trial jobs share a jobs.group). A proposer is not required to call it: "knit sweep" deliberately does not, keeping the sweep's own --group (which labels the parent sweeps row) independent of any --group the user writes on the downstream submit/prepare line (which labels the produced jobs). A sweep still finds its children through the automatic provenance "call" edges, not a shared group.

Parameters:

__knit_argv -- [inout] Name of the array holding the downstream argv.
group -- [in] The campaign group label (empty to leave the line unchanged).

int knit_search_launch()#

Launch one tracked trial: invoke a fully-substituted downstream knit command line in-process, through the normal entry path, and return the child's recorded row id by nameref. The downstream line is one of:

submit ... -- <job> ... a job submission run ... -- <app> ... a run <command> ... a plain command

The engine stays opaque to the line: it forwards every argument verbatim to _knit_invoke_command (the same path knit() uses), so a dispatcher such as submit or run parses its own options and the target its parameters. The invocation runs in-process, inside the calling proposer's frame, so knit's existing recording writes the provenance "call" edge parent -> child by itself; the search core adds no edge type and no provenance hook.

The child's row id is read from _KNIT_LAST_ROW_ID, which the recorder sets to each recorded invocation's id. It is empty when the child records no row (the experiment is not bootstrapped, or recording is disabled). The function returns the downstream line's own exit status, so a caller (the search loop) can see a plain command's or an app run's success or failure.

Parameters:

__knit_ret -- [out] The child invocation's recorded row id (empty if none).
... -- [in] The downstream knit command line (command and its arguments).

Returns:

The exit status of the downstream invocation.

int knit_search_loop()#

The shared search loop: drive a proposer over a budget of trials, keeping up to --parallel in flight, launching each through knit_search_launch and detecting completion by the chosen mode. It owns budget accounting, in-flight tracking, completion detection, and routing a finished trial's outcome to the proposer; the proposer (the ask/observe callbacks) owns which assignment runs next and the failure policy, and an optional collector owns the objective read. Both proposers (the cartesian sweep and the Ax plugin) reuse this one loop.

The proposer is two callbacks. ask yields the next trial, returning the proposer's opaque tag and the fully-substituted downstream command line (the argv knit_search_launch runs); it returns non-zero when it has no more trials. observe is told each finished trial's tag, child row id, terminal status, and objective, and returns non-zero to abort the campaign (how a failure maps to an abort is the proposer's policy). The optional collector reads one finished child's objective; it runs only when given.

Completion detection has two modes: "job" polls the jobs table state column (ordinary submissions, possibly many queued at once); "app" reaps backgrounded "knit run" children with "wait -n".

Parameters:

... -- [in] Options: --mode <job|app> (required), --ask <fn> (required), --observe <fn> (required), --collect <fn> (optional), --parallel <K> (default 1), --budget <N> (default unbounded), --poll <seconds> (job mode, default _KNIT_SCHED_POLL_INTERVAL).

Returns:

0 on a completed campaign, 1 when the proposer aborted it.

int knit_search_parse_token()#

Public accessor that parses one domain descriptor token into its structured form. The same grammar covers "{...}" list, "[lo..hi]" / "[lo..hi:step]" range, "@glob:<pattern>" glob, and "@file:<path>" file tokens. Callers MUST declare the first output as an associative array (declare -A) and the second as an ordinary array. A malformed or unrecognised token is fatal.

Parameters:

__knit_ret1 -- [out] Associative array to receive the descriptor scalars: keys kind (list|range|glob), lo, hi, step, glob.
__knit_ret2 -- [out] Indexed array to receive the list values (empty unless kind=list).
token -- [in] The descriptor token to parse.

int knit_search_read_objective()#

Read one finished trial's objective value from the database, given the trial's recorded child row id (as returned by knit_search_launch) and the table/column that holds the objective. A trial is a chain of recorded rows linked by "call" edges in the provenance table: the launch id is the outer dispatcher's row (the jobs row for "submit", the runs row for "run"), while the objective is usually written by a descendant (the job body's row, the app's row). This function resolves the objective row by walking "call" edges breadth-first from the child id until it reaches a row that exists in the named table, then reads the named column from that row.

The child id itself is tried first, so an objective recorded directly in the dispatcher's own table (no descendant) is read without a walk. The caller owns the policy of which table and column hold the objective (e.g. the Bayesian optimizer's leaf-first chain walk over declared outputs); this function owns only the generic row resolution and point read.

Trust contract: only the id is SQL-escaped; table and column are interpolated into the query verbatim. They are expected to be registration-derived names (e.g. from knit_command_table and knit_output_type), the same trust knit places in table names throughout db.sh — do NOT pass an end-user string here.

A missing objective row (no reachable row in the table) and a NULL/empty objective column (a crashed trial that recorded no value) are both reported as failure: the output is cleared and the return status is non-zero, so the caller can apply its failure policy.

Parameters:

__knit_ret -- [out] The objective value (empty on failure).
child_id -- [in] The trial's recorded child row id (from knit_search_launch).
table -- [in] The table that holds the objective column.
column -- [in] The objective column name.

Returns:

0 when a non-empty objective value was read, 1 otherwise.

int knit_search_resolve_fill()#

Resolve, for one parameter, whether it varies in the campaign and from where. This is the shared "Which parameters vary" rule that both proposers reuse (each sets only its own default --fill); it does NOT read or enumerate a declared domain (how a varying parameter's domain is used -- enumerate or sample -- stays with the proposer, as does the enumerability check). The result is one of:

explicit the call site carries a concrete value or a set literal for this parameter, so the proposer uses what the line already says; domain vary this parameter from its declared domain; default leave this parameter at its default value.

The precedence (highest first): an explicit value/token always wins; then a per-parameter @domain / @default token; then the --fill policy for an otherwise unspecified parameter. Under --fill domain an unspecified parameter varies only when it has a declared domain -- a parameter with no declared domain stays at its default under either policy (it is never "the space"). The per-parameter token classification (explicit / domain / default / unspecified) comes from the proposer's own call-site scan; this helper only applies the precedence.

Parameters:

__knit_ret -- [out] The resolution ("explicit", "domain", or "default").
fill -- [in] The --fill policy for an unspecified parameter ("default" or "domain").
token -- [in] How the call site referenced this parameter: "explicit" (a concrete value or set literal), "domain" (@domain), "default" (@default), or "unspecified" (nothing).
has_domain -- [in] "true" if the parameter has a declared domain, else "false".

int knit_search_resolve_target()#

Resolve the downstream target from the downstream token list: the command whose parameters a search proposer introspects (to resolve declared domains or @domain markers), plus the dispatcher and target display names. The mapping mirrors how the downstream is actually run:

submit|prepare ... -- <job> ... -> submit:<job> run ... -- <app> ... -> run:<app> <command> ... -> the command itself

The mangled command is empty when a submit/prepare/run line carries no job or app after its inner "--" (the downstream dispatcher reports the error when the line is actually run), so the caller can still record the dispatcher name.

Parameters:

__knit_ret1 -- [out] Mangled command name to introspect (empty if none).
__knit_ret2 -- [out] Dispatcher display name (submit/prepare/run, or the command name for a bare command).
__knit_ret3 -- [out] Target display name (the job/app name, or the command name for a bare command).
... -- [in] The downstream tokens.

int knit_search_scan()#

Classify a downstream token list left to right into an ordered list of axes and a reconstruction template, without introspecting the downstream command and without enumerating literal axes. A token is handled only when it starts with "{", "[", or "@" and its preceding "--option" is not named in --passthrough; every other token is literal and is placed in the template verbatim. Each handled token is classified:

"{...}" / "[...]" / "@glob:..." / "@file:..." -> kind "literal";
"@domain" -> a marker to vary the declared domain (kind "domain");
"@default" -> a marker to pin at the default (kind "default");
any other "@..." -> a fatal (unknown directive).

An axis is named by its preceding "--option" (hyphens folded to underscores); a swept positional token is named "#<index>" by its position in the downstream. Each axis also records the command segment it belongs to: the dispatcher keyword (e.g. "submit") before the inner "--", or the target command name (e.g. "train") after it; a bare command is its own segment. The segment lets a consumer build a command-qualified axis identity ("<segment>.<option>") that stays unique when the same option name appears in two segments (e.g. "submit --nodes ... -- job --nodes ..."). The "seen" map records, for every "--option" that carried a value, how the call site referenced it ("explicit", "domain", or "default"); a literal axis folds to "explicit". The inline "--opt=value" form is self-contained and recorded as "explicit". All four outputs are reset on entry, so the "seen" map need not be empty, but it MUST be declared as an associative array (declare -A).

Unlike _knit_sweep_scan, this function stores the verbatim token text (not an enumerated value list) in rawtokens-out for literal axes. Callers that need the enumerated list enumerate each raw token themselves.

Parameters:

__knit_ret1 -- [out] Name of the array to receive the reconstruction template (a copy of the downstream tokens).
__knit_ret2 -- [out] Name of the array to receive the axis descriptors, each the string "<kind> <slot> <name> <segment>".
__knit_ret3 -- [out] Name of the array to receive per-axis raw tokens: the verbatim token text for a "literal" axis, empty for "domain"/"default".
__knit_ret4 -- [out] Name of the associative array to receive the seen map (option -> "explicit"|"domain"|"default").
passthrough_csv -- [in] Comma-separated option names whose values are never treated as sweep-syntax (may be empty).
... -- [in] The downstream tokens.

int knit_search_segment_command()#

Resolve the mangled command to introspect for a given downstream segment. A search axis records the segment it belongs to (knit_search_scan); this maps that segment to the command whose parameters and declared domains govern the axis, so a "@domain"/"@default" axis is resolved against its OWN segment's command instead of a single global target. This is the one place both search proposers (knit sweep and the knit-ax plugin) route a per-segment domain lookup, so the same option name in two segments (e.g. "submit --nodes @domain -- job --nodes @domain") resolves against two different commands.

Segments and their commands:

the dispatcher segment ("submit"/"prepare"/"run") -> the dispatcher command whose option surface it shares (submit and prepare both -> submit; run -> run);
the target segment (the <job>/<app> after the inner "--") -> the mangled "submit:<job>" / "run:<app>";
a bare command -> the command itself.

Returns the empty string when the segment matches neither the dispatcher nor the target of the downstream (an unknown segment), leaving the caller to report it.

Parameters:

__knit_ret -- [out] Mangled command name for the segment (empty if none).
segment -- [in] The segment name, as recorded in an axis descriptor.
... -- [in] The downstream tokens.

int knit_set_invocation_args()#

Replace the pending (expanded) argument list of the command whose before-callbacks are running; the body and the recorded row then see the new list. The power-user escape hatch behind knit_set_parameter — the caller owns the whole list and it is NOT re-validated (edits land after the argument and --when checks). Fatal when there is no invocation in progress (i.e. not called from a non-wrapper command's before-callback).

Parameters:

... -- [in] The new argument tokens.

int knit_set_parameter()#

Set (or override) one declared parameter of the command whose before-callbacks are running, editing the pending argument list in place: the body and the recorded row then see the new value. This is the ergonomic way to inject a value — e.g. a plugin filling a parameter from its own source. The name must be a declared required/optional parameter or flag of the command (fatal otherwise) and the value must match its type (checked with the standard type validator). An existing occurrence is replaced; an absent one is appended. Fatal outside a command's before-callback.

Note: like knit_set_invocation_args, the injection happens AFTER the required- argument and --when checks, so an injected value is not re-checked against those (only its type is).

Parameters:

name -- [in] Parameter name (hyphens and underscores interchangeable).
value -- [in] Value to set.

int knit_set_program_description()#: Set the description of the program.

int knit_set_row_id()#

Set the "id" value of the row that will be recorded for the currently executing command, instead of letting recording mint a fresh uuid. Use this when a command already owns a canonical identifier — e.g. knit submit records the job UUID it generated, and a plugin that re-attaches to an existing row (its campaign/parent row created by an earlier invocation) sets that row's id so its provenance "call" edges to children and its later updates land on that row rather than on an ephemeral one. Also updates the executing-row-id stack so a nested child resolves this id as its parent. Must be called from within an executing command function.

Parameters:

id -- [in] The id to record as the row's id.

int knit_setup_activate_line()#

Declare a verbatim activation line. Called from a setup body, it both runs the line in the current build shell (so the rest of the body sees its effect) and records it verbatim into the setup's .activate.sh, so every dependent job runs the same line. Use it for activation steps that the env helpers do not cover, e.g. module load gcc/12 or source <some tool's env script>.

Parameters:

line -- [in] The shell line to run now and record.

int knit_setup_env_append()#

Declare that a setup appends an entry to a colon-separated list variable (e.g. PATH). Called from a setup body, it both appends to VAR in the current build shell and records a composable line into the setup's .activate.sh: export VAR="${VAR:+${VAR}:}"<value>. That line extends the job's own VAR at activation time rather than replacing it, and is empty-safe (no leading colon when VAR was unset).

Parameters:

var -- [in] The list variable name (a shell identifier).
value -- [in] The entry to append.

int knit_setup_env_prepend()#

Declare that a setup prepends an entry to a colon-separated list variable. Called from a setup body, it both prepends to VAR in the current build shell and records a composable line into the setup's .activate.sh: export VAR=<value>"${VAR:+:${VAR}}". That line prefixes the job's own VAR at activation time rather than replacing it, and is empty-safe (no trailing colon when VAR was unset).

Parameters:

var -- [in] The list variable name (a shell identifier).
value -- [in] The entry to prepend.

int knit_setup_env_set()#

Declare that a setup sets an environment variable. Called from a setup body, it both exports VAR=value in the current build shell and records export VAR=<value> into the setup's .activate.sh, so every dependent job gets the same assignment. The value is recorded with printf q so any characters survive re-sourcing.

Parameters:

var -- [in] The variable name (a shell identifier).
value -- [in] The value to assign.

int knit_setup_env_unset()#

Declare that a setup unsets an environment variable. Called from a setup body, it both unsets VAR in the current build shell and records unset VAR into the setup's .activate.sh.

Parameters:

var -- [in] The variable name to unset (a shell identifier).

int knit_trace()#

Logging function for trace-level messages. Works like echo but will only print if the logging level was set to "trace".

Parameters:

... -- [in] Arguments for printf.

int knit_type_check()#

Check whether a value conforms to the specified type. For enum types, checks that the value is one of the defined enum values.

Type validation rules:

integer: optional sign followed by digits
real: decimal number with optional exponent (e.g. 3.14, .5, 1e10)
boolean: "true" or "false"
string: any value (always passes)
path, filename, file, directory: non-empty string (shape only; existence of a file or directory is checked separately as direction-aware runtime logic, not by this pure type check)
date: YYYY-MM-DD with valid month/day ranges
time: hh:mm:ss with valid hour/minute/second ranges
datetime: "YYYY-MM-DD hh:mm:ss" combining date and time rules

Example:

knit_type_check "integer" "42"          # returns 0
knit_type_check "integer" "hello"       # returns 1
knit_type_check "date" "2025-03-13"     # returns 0
knit_type_check "color" "red"           # returns 0 (if color enum defined)

Parameters:

type -- [in] Type name (or alias) to check against.
value -- [in] Value to validate.

Returns:

0 if the value is valid for the type, 1 otherwise.

int knit_type_exists()#

Check whether a type name is valid. Returns 0 if the name is a built-in type, a type alias, or a user-defined enum.

Example:

knit_type_exists "integer"    # returns 0
knit_type_exists "int"        # returns 0 (alias for integer)
knit_type_exists "unknown"    # returns 1

Parameters:

type_name -- [in] Type name to check.

Returns:

0 if the type exists, 1 otherwise.

int knit_usable_before_bootstrap()#

Mark the command currently being registered as usable before bootstrap, i.e. it may be invoked (and appears in "--help") on a fresh checkout where no ".knit/" directory exists yet. Commands are not usable before bootstrap by default.

A command usable before bootstrap must not declare a database table (knit_with_table) nor carry any "--when" constraint on its parameters (both rely on binaries that bootstrap provisions), and a subcommand may only be usable before bootstrap if its parent is too. These rules are enforced at knit_done time by knit_usable_before_bootstrap_validate (registered here as a knit_done callback), so that they see the final command definition regardless of the order in which knit_with_table / knit_with* / this decorator are called.

Calling it more than once on the same command is harmless (idempotent): the validation callback is registered only on the first call.

int knit_usable_if()#

Declare that the command currently being registered may be used only when <predicate> returns 0. <predicate> is the name of a user-defined shell function that receives the demangled command name as its single argument and returns 0 ("usable") or non-zero ("not usable"). is a human-readable string explaining why the command cannot run; it is shown as a fatal error if the user invokes the command while the predicate is false.

Repeatable: multiple calls register multiple predicates. At invocation time the predicates are evaluated in declaration order, stopping at the first that returns non-zero (whose becomes the error message); a command is usable only if all of its predicates pass.

The predicate is not called here — only its name and description are recorded. Enforcement happens at invocation time via _knit_command_check_usable. The per-command storage arrays (_usable_pred / _usable_desc) are declared lazily on first use so commands that do not use this decorator pay no registration cost.

Parameters:

predicate -- [in] Name of the predicate function.
description -- [in] Message shown if the command is invoked while not usable.

int knit_warning()#

Logging function for warning-level messages. Works like echo but will only print if the logging level was set to "warning".

Parameters:

... -- [in] Arguments for printf.

int knit_with_checksum()#

Declare an optional integrity pin for the resource type: a sha256 verified at fetch time, but only when the source defaults are used (an overridden source would legitimately hash differently) and not bypassed with --ignore-checksum. Valid only on a resource type, at most once. The pin is the sha256 of the downloaded archive (url), the sha256 of the local file or a recursive digest of a local directory (local), or the expected commit SHA the default ref resolves to (git). A mismatch fails the fetch and removes the partial instance.

Parameters:

sha256 -- [in] The expected sha256 (a commit SHA for the git backend).

int knit_with_dispatch()#

Mark the command currently being registered as a dispatcher: a command that takes a target after "--" and forwards the remaining arguments to it (e.g. "submit" dispatches to a job, "setup" to a setup. This changes how "--help" renders the usage line, both for the dispatcher itself (cmd [OPTIONS] -- <placeholder> [OPTIONS]) and for its subcommands, which are invoked through it rather than directly.

The description is stored as the extra-arguments description (as with knit_with_extra), so the "Extra" help section and the "extra allowed after

--" argument check keep working. If no description is given, the placeholder is used.

Parameters:

placeholder -- [in] Name shown after "--" in the usage line (e.g. "job").
description -- [in] Optional description of the extra arguments.

int knit_with_extra()#

Adds a description for extra parameters coming after "--".

Parameters:

description -- [in] Description of the extra parameters.

int knit_with_flag()#

This function should be called right after a call to knit_register (or one of its variants) to declare flag parameters that the command may accept.

Example:

knit_register "greet" "say_hello" "Say hello to someone"
knit_with_flag "capitalize" "Make the output upper-case"
say_hello() {
   ...
}

Parameters:

param -- [in] Parameter name.
description -- [in] Description of the parameter.
--when -- [in] Optional boolean constraint expression (jq syntax referring to the command's other parameters); the parameter only applies when the expression evaluates to true.
--no-record -- [in] Optional flag; parse, validate, and show the flag in "--help" as usual, but keep it out of the command's table (no column) and out of every recorded row. For a call-site control flag that must not widen the table schema.

int knit_with_git()#

Download decorator: acquire the resource type by cloning a git repository and checking out a ref. Declares the automatic parameters --url (default <url>) and --ref (default <ref>); both arguments are required at registration because a resource type must name the ref it pins. Also declares the "commit" output, in which the fetch records the commit SHA the ref resolved to. Valid only on a resource type, at most one download decorator per type.

Parameters:

url -- [in] The default git repository URL.
ref -- [in] The default git ref (branch, tag, or commit) to check out.

int knit_with_input_artifact()#

Declare that the command currently being registered consumes a recorded artifact of a given kind, given as "name:kind" (e.g. "input_table:csvfile"). Must be called between a knit_register* call and knit_done; a command may declare several input artifacts. The kind must be a registered artifact kind (see knit_register_artifact); the builtin kinds "file", "directory", and the "dir" alias are always available. It is the consuming counterpart of knit_with_output_artifact.

Under the hood (mirroring knit_with_resource) this registers an ordinary required string parameter named name whose value is the artifact's artifacts-relative path, so the CLI, knit_get_parameter, and the backing table all treat it as a plain value; the command body turns that path into an on-disk location with knit_input_artifact_path. A per-parameter marker (KNIT_CMD<cmd>input_artifact=<kind>) records the required kind, and a companion input_artifact_verify marker records the --verify-checksum opt-in; both are read by the validation before-callback and, later, by describe / --help. A before-callback resolves the path to the artifacts row and validates it (existence + kind, plus an opt-in checksum re-verification) before the body runs, and an after-callback records a "used_by" provenance edge from the consumed artifact to this command.

knit_register "plot" plot "Plot a results table."
knit_with_input_artifact "input_table:csvfile" "The table to plot." --verify-checksum
plot() {
    local csv
    csv="$(knit_input_artifact_path "$(knit_get_parameter input_table "$@")")"
    gnuplot -e "..." "${csv}"
}
knit_done

Invoked as: ./exp.sh plot --input-table tables/run7.csv.

Wrappers cannot declare knit_with_input_artifact (they forward their arguments verbatim and take no parsed parameters). Every declared input artifact is required.

Parameters:

spec -- [in] The dependency as "name:kind" (a registered artifact kind).
description -- [in] One-line description of the input-artifact parameter.
--verify-checksum -- [in] Optional flag; re-verify the recorded checksum before the body runs.

int knit_with_local()#

Download decorator: acquire the resource type from a local path, by default as a symlink (so a large staged dataset is not duplicated) or, with --copy, as a self-contained read-only snapshot. Declares the automatic parameters --path (default <path>) and the --copy flag. Valid only on a resource type, at most one download decorator per type.

Parameters:

path -- [in] The default local source path to link or copy.

int knit_with_optional()#

This function should be called right after a call to knit_register (or one of its variants) to declare optional parameters for the command. The parameter name may include a type annotation using the "name:type" syntax (e.g. "count:integer"). If no type is given, "string" is assumed.

Example:

knit_register "greet" "say_hello" "Say hello to someone"
knit_with_optional "name:string" "world" "Name of the person to greet"
knit_with_optional "count:integer" "1" "Number of times to greet"
say_hello() {
   ...
}

Indicates that the command "greet" has an optional parameter --name (string, default "world") and --count (integer, default "1").

The default may be written as "ENV[NAME]" to mean "fall back to the value of

the NAME environment variable when the parameter is not provided". This is resolved when the parameter is filled in, so a job whose environment is set up by a

knit setup (e.g. knit_with_optional "seed:integer" "ENV[MC_SEED]" ...) picks up the value exported by that setup.

Parameters:

param -- [in] Parameter name followed by ":type".
default -- [in] Default value (or "ENV[NAME]" to read the NAME env variable).
description -- [in] Description of the parameter.
--when -- [in] Optional boolean constraint expression (jq syntax referring to the command's other parameters); the parameter only applies when the expression evaluates to true.
--no-checksum -- [in] Optional flag; for a file/directory parameter, disable the content checksum and its companion column.
--domain -- [in] Optional search domain descriptor ("{a,b,c}", "[lo..hi]", "[lo..hi:step]", "@glob:<pat>", or "@file:<path>"; quote it). Its shape is checked here and the token is stored verbatim; it is parsed and validated against the parameter type on demand, when the parameter-sweep or Bayesian-optimization search proposer reads it.
--log -- [in] Optional flag; for a real "--domain [lo..hi]", mark the range as log-scale (used by the optimization proposer, ignored by the sweep).
--no-record -- [in] Optional flag; parse, validate, and show the parameter in "--help" as usual, but keep it out of the command's table (no column) and out of every recorded row. For a call-site control option that must not widen the table schema.

int knit_with_output()#

This function should be called right after a call to knit_register (or one of its variants) to declare an output that the command produces. The output name must include a type annotation using the "name:type" syntax (e.g. "result:integer").

Example:

knit_register "compute" "compute" "Compute something."
knit_with_output "result:real" "0.0" "The computed result."
compute() {
   ...
}

Parameters:

param -- [in] Output name followed by ":type".
default -- [in] Default value.
description -- [in] Description of the output.
--no-checksum -- [in] Optional flag; for a file/directory output, disable the content checksum and its companion column.
--result -- [in] Optional flag; mark the output as a result (what the experiment was for). Valid on an output of any type.

int knit_with_output_artifact()#

This function should be called right after a call to knit_register (or one of its variants) to declare an artifact that the command produces: a file or directory, kept under the artifacts root, that the command binds at runtime with knit_artifact. It is the file/directory counterpart of knit_with_output. The artifact name must include a kind annotation using the "name:kind" syntax, and the kind must be a registered artifact kind (see knit_register_artifact). The builtin kinds "file", "directory", and the "dir" alias are always available; a user kind such as "csvfile" is declared once at the top level. The physical type behind the kind ("file" or "directory") drives the existence check and the checksum, while the kind itself is recorded on the artifacts row.

Example:

knit_register "tabulate" "tabulate" "Tabulate results."
knit_with_output_artifact "table:csvfile" "The results table (CSV)."
tabulate() {
   out="$(knit_artifact_dir)"
   compute > "${out}/table.csv"
   knit_artifact "table" "table.csv"
}

Unlike an ordinary output, an artifact is NOT a column of the command's own table. Each binding is recorded at runtime as one row in the framework-owned artifacts table (its artifacts-relative path, name, physical type, semantic kind, content checksum, and result flag) with a "produced" edge from the producing invocation, so a file can be traced back to what made it. The content digest is always recorded for an artifact, so there is no --no-checksum opt-out here. The artifact is added to the command's artifacts set, and its kind/description are kept in registration state so knit describe can report it.

Because a "produced" edge needs the producing invocation's row as its source, a command that declares an artifact records an invocation row even if it declared no table of its own: a table is ensured automatically at knit_done time.

Parameters:

param -- [in] Artifact name followed by ":kind" (a registered artifact kind).
description -- [in] Description of the artifact.
--result -- [in] Optional flag; mark the artifact as a result (what the experiment was for).

int knit_with_parameter_set()#

Import parameters from a previously defined parameter set into the command currently being registered. May be called multiple times with different sets. Conflicts between the set's parameters and parameters already declared for the command are reported as fatal errors.

By default every parameter the set declares is imported. The optional --exclude and --only options (mutually exclusive) narrow the import:

--exclude "a,b" imports the set minus the named parameters, freeing those names so the command can declare them itself (e.g. with a different kind or default).
--only "a,b" imports only the named parameters (an allow-list).

Every name in either list must exist in the set, else the call is fatal.

Parameters:

name -- [in] Name of the parameter set to import.
--exclude -- [in] Optional comma-separated list of parameters to skip.
--only -- [in] Optional comma-separated allow-list of parameters to import.

int knit_with_pilot()#

Declare that the job currently being registered is a pilot job: one that may call knit submit in its body to schedule sub-jobs onto its own allocation, through an in-allocation knit-pilotd daemon instead of the machine scheduler. Must be called between knit_register_job and knit_done, at most once per job.

Valid only for jobs. Setups, wrappers, apps, and plain commands are rejected with a clear fatal (only a job has an allocation to schedule onto).

Options (flags to the decorator, stored on the command's pilot markers): --exclude-launch-host Drop the daemon's own host from the node pool when it is part of the allocation (for systems where the job script shares the first compute node). A no-op on separate launch/service-node systems. --kill-on-exit When the body returns with sub-jobs still outstanding, kill them instead of draining (the default).

It installs a before-callback that starts the daemon and exports KNIT_PILOT / KNIT_PILOT_SOCK, and an after-callback that stops it.

Example:

knit_register_job "campaign" _campaign "Run many sub-jobs in one allocation."
knit_with_pilot
_campaign() {
    local ids=""
    local seed
    for seed in 1 2 3; do
        ids+="${ids:+,}$(knit submit --nodes 2 -- simulate --seed "${seed}")"
    done
    knit job wait --id "${ids}"
}
knit_done

int knit_with_provenance()#: Mark the command being registered as participating in the provenance graph: an invocation of it records a "call" edge and acts as an in-process parent frame for the commands it invokes. This is an explicit override of the default (which is by visibility — see _knit_provenance_enabled), so it forces a hidden command into the graph. The mark also propagates to unmarked lexical descendants (e.g. "a" marked "with" makes an unmarked "a:b" participate).

int knit_with_required()#

This function should be called right after a call to knit_register (or one of its variants) to declare required parameters that the command expects. The parameter name may include a type annotation using the "name:type" syntax (e.g. "width:integer"). If no type is given, "string" is assumed.

Example:

knit_register "greet" "say_hello" "Say hello to someone"
knit_with_required "name:string" "Name of the person to greet"
knit_with_required "count:integer" "Number of times to greet"
say_hello() {
   ...
}

Indicates that the command "greet" requires a parameter --name (string) and --count (integer).

Parameters:

param -- [in] Parameter name followed by ":type".
description -- [in] Description of the parameter.
--when -- [in] Optional boolean constraint expression (jq syntax referring to the command's other parameters); the parameter only applies when the expression evaluates to true.
--no-checksum -- [in] Optional flag; for a file/directory parameter, disable the content checksum and its companion column.
--domain -- [in] Optional search domain descriptor ("{a,b,c}", "[lo..hi]", "[lo..hi:step]", "@glob:<pat>", or "@file:<path>"; quote it). Its shape is checked here and the token is stored verbatim; it is parsed and validated against the parameter type on demand, when the parameter-sweep or Bayesian-optimization search proposer reads it.
--log -- [in] Optional flag; for a real "--domain [lo..hi]", mark the range as log-scale (used by the optimization proposer, ignored by the sweep).
--no-record -- [in] Optional flag; parse, validate, and show the parameter in "--help" as usual, but keep it out of the command's table (no column) and out of every recorded row. For a call-site control option that must not widen the table schema.

int knit_with_resource()#

Declare that the command currently being registered consumes a fetched resource instance, given as "<param>:<type>" (e.g. "training_dataset:image_dataset"). Must be called between a knit_register* call and knit_done; a command may declare several resources. The <type> must be a resource type registered with knit_register_resource before this call.

Under the hood this registers an ordinary required string parameter named <param> (so the CLI, knit_get_parameter, and the backing table all treat it as a plain value) whose value is the resource instance name. The command body turns that name into a path with knit_resource_path. A per-parameter marker (KNIT_CMD<cmd>resource=<type>) records the declared type for validation and, later, for describe / --help. A before-callback validates the named instance (existence + recorded type) before the body runs, and an after-callback records a "used_by" provenance edge from the fetched instance to this command.

Wrappers cannot declare knit_with_resource (they forward their arguments verbatim and take no parsed parameters). Every declared resource is required for now.

Example:

knit_register "train" _train "Train a model."
knit_with_resource "training_dataset:image_dataset" "Training images."
_train() {
    local dir
    dir="$(knit_resource_path "$(knit_get_parameter training_dataset "$@")")"
    ...
}
knit_done

Parameters:

spec -- [in] The dependency as "<param>:<type>".
description -- [in] One-line description of the resource parameter.

int knit_with_setup()#

Declare that the command currently being registered requires a setup of a given type (the name of a setup registered with knit_register_setup). Must be called between a knit_register* call and knit_done, at most once per command.

How the requirement is consumed depends on the kind of command:

Jobs (knit_register_job): knit submit makes --setup mandatory, rejects a --setup that was not built by the declared type, and the job re-sources the setup environment on the compute node.
Any other command: knit_with_setup adds a --setup option to the command, and a before-callback validates the given setup's type and sources its .activate.sh so the command body runs in the setup environment.

Setups and wrappers may NOT declare knit_with_setup and are rejected:

a setup's KNIT_SETUP_PREFIX is its own output directory, so chaining setups would make that prefix ambiguous;
a wrapper forwards its arguments verbatim and cannot take a parsed --setup.

Example:

knit_register_job "montecarlo" _montecarlo_job "Estimate pi as a job."
knit_with_setup "mcenv"   # requires a setup built by the "mcenv" setup
_montecarlo_job() { ... }
knit_done

Parameters:

type -- [in] Name of the required setup type.

int knit_with_subcommand_discovery()#

Name a discovery function for the command currently being registered. The command's subcommands are then registered lazily: instead of running their "knit_register ... knit_done" blocks at load time, the framework calls the named function at most once, and only when it must know the command's subcommands (the resolver reaches for a subcommand, "--help" lists them, or "describe" walks into the command). This keeps "source knit.sh" and every command start cheaper by not building command subtrees a run never touches.

The named function registers the command's immediate subcommands, each as a full "knit_register ... knit_done" block. It runs in the main shell (never a subshell) and receives the parent command's display name as its first argument. It must be fast and side-effect free (registration only): it runs on the resolution and help paths, which must stay cheap.

State stored on the command:

KNIT_CMD<cmd>subcommand_discovery : the function name.
_KNIT_CMD<cmd>_discovered : "true" after the function has run.

Must be called between a knit_register* call and knit_done. It is not valid on a wrapper (knit_register_wrapper), which forwards its arguments verbatim and has no subcommands. A command may name at most one discovery function; a second call is fatal. The function name must be a non-empty token; its existence is checked when it is called (it may be defined after this decorator).

int knit_with_subcommand_title()#: Change the title of subcommands for the command being registered (default subcommand name is "Subcommands"). This is the title displayed when calling --help.

int knit_with_table()#

Declare a database table for recording invocations of the command currently being registered. Must be called between knit_register and knit_done.

If no table name is given, the demangled command name is used (e.g. "foo:bar" for a subcommand "foo bar"). An error is raised if the same table name is claimed by more than one command.

At knit_done time a callback checks whether the table already exists with the correct schema. If absent it is created; if the schema has changed it is migrated. The table always has an "id" (uuid) column first, followed by all required parameters, optional parameters, flags, and outputs, each group sorted alphabetically.

Example:

knit_register "run" my_func "Run an experiment."
knit_with_required "count:integer" "Number of iterations."
knit_with_table           # uses table name "run"
knit_with_table "my_runs" # uses table name "my_runs"
my_func() { ... }
knit_done

Parameters:

table_name -- [in] Optional name of the database table. Defaults to the colon-separated command name.

int knit_with_url()#

Download decorator: acquire the resource type by downloading a URL with curl. Declares the automatic parameters --url (default <url>) and the --uncompress flag (unpack the archive after download). Valid only on a resource type, at most one download decorator per type.

Parameters:

url -- [in] The default URL of the artifact to download.

int knit_without_provenance()#: Mark the command being registered as excluded from the provenance graph: an invocation of it records no "call" edge and is transparent when a command it invokes resolves its parent (the child links to the nearest participating ancestor instead). This is an explicit override of the default, so it silences a visible command. The mark also propagates to unmarked lexical descendants (e.g. "a" marked "without" silences an unmarked "a:b"). Data-row recording (knit_with_table) is orthogonal and still happens.

int knit_without_setup()#

Declare that the job currently being registered opts out of the implicit "default" setup. Must be called between a knit_register* call and knit_done.

Jobs that declare neither knit_with_setup nor knit_without_setup run in the builtin "default" setup (see _knit_submit), inheriting the platform environment with no boilerplate. knit_without_setup makes such a job run with no setup at all — no setup directory, no platform activation. It is mutually exclusive with knit_with_setup.

Setups and wrappers are rejected. For any non-job command it is a no-op: plain commands and apps are already setup-less unless they explicitly declare knit_with_setup, so there is no implicit default to opt out of.

Variables#

String KNIT_IGNORE_CHECKSUM#: Public environment variable exported into a resource download body: "true" when checksum verification is disabled for this fetch, "false" otherwise. A download body reads it to decide whether to skip its own integrity check.

String KNIT_JOB_PREFIX#: Public environment variable exported into a running job's environment: the absolute path of the job's own working directory, <job-root>/<uuid>. It is set by the generated job script (see _knit_sched_write_jobscript), which also makes it the process working directory, so a job body may either read it explicitly (e.g. to build an absolute output path) or rely on relative paths landing in it. It is unset outside a running job (on the login/submit side). The basename of this path is the job UUID, which is how compute-side callbacks recover their own row id in the jobs table.

ExportedString KNIT_LOG_LEVEL#: Log level. Valid values: trace, debug, info, warning, error, critical.

String KNIT_MPI_LOCAL_RANK#: Public environment variable exported into an app body: the node-local rank of this process (0-based within its node). Normalized by the framework from the active launcher, so an app body reads the same variable everywhere. It is 0 for a single-process run.

String KNIT_MPI_RANK#: Public environment variable exported into an app body: the rank of this process in MPI_COMM_WORLD (0-based). Normalized by the framework from whichever launcher is in use (OpenMPI, MPICH/PMI, Slurm, PALS, or Flux), so an app body reads the same variable everywhere. It is 0 for a single-process run.

String KNIT_MPI_SIZE#: Public environment variable exported into an app body: the number of processes in MPI_COMM_WORLD. Normalized by the framework from the active launcher, so an app body reads the same variable everywhere. It is 1 for a single-process run.

ExportedString KNIT_PILOT_SOCK#: Public environment variable exported alongside KNIT_PILOT: the path of the knit-pilotd control socket the pilot job's body (and its knit submit calls) use to talk to the daemon.

String KNIT_RESOURCE_EXPECTED_CHECKSUM#: Public environment variable exported into a resource download body: the expected checksum of the resource when the caller requested one (empty when none was given). A download body may verify the fetched content against it.

String KNIT_RESOURCE_PREFIX#: Public environment variable exported into a resource download body: the absolute path the body must create and populate with the fetched resource. Set by the framework before it runs the resource type's download body.

String KNIT_SCRIPT_NAME#: Base name (without directories) of the experiment script that sourced knit.sh. Useful for constructing user-facing messages.

String KNIT_SCRIPT_PATH#: Absolute path of the experiment script that sourced knit.sh. Used when generating batch scripts so the compute node can re-enter the experiment (exp.sh submit <job-name> ...) regardless of its current directory.

String KNIT_SETUP_PREFIX#: Public environment variable exported into a setup body and every command that depends on the setup: the absolute path of the setup's own directory, where it should install its artifacts. A setup body reads it to build install paths (e.g. cmake --install "${KNIT_SETUP_PREFIX}"), and a dependent job or app sees it in its environment after the setup's activation runs. Set by the framework; unset when no setup is in effect.

ExportedReadOnlyString KNIT_VERSION#: Version of the Knit framework. This committed value is a development placeholder: released artifacts are stamped with the exact git tag at build time (see the knit.sh target in the Makefile), so this file never needs editing when cutting a release.

Public API

Contents

Public API#

Functions#

Variables#