Private API

The private API consists of names with one or two leading underscores, grouped below by the source file that defines them. It is internal to Knit and may change at any time, without notice and without a compatibility guarantee.

ai.sh

Functions

int _knit_ai_ask()

int _knit_ai_chat_request()

Build an OpenAI chat-completions request from the given model, messages array, and optional tools array, POST it to <base_url>/chat/completions, and print the raw response JSON on stdout for the caller to parse.

The API key is passed to curl through a mode-600 config file (never a -H flag), so it never appears in the process argv (ps) or in any trace. Only a redacted form of the request is traced. A provider error (.error in the response) is surfaced through the logging system and turned into a fatal; the request body is never dumped.

Parameters:

• base_url -- [in] Endpoint base URL (without a trailing /chat/completions).

• api_key -- [in] Resolved API key (kept local; never logged).

• model -- [in] Model id.

• messages_json -- [in] JSON array of chat messages.

• tools_json -- [in] Optional JSON array of tool definitions (omitted when empty).

int _knit_ai_default_system_prompt()

Print the default system prompt for ai ask: a short explanation of knit, the instruction to answer about this experiment using only the read-only tools, and a compact seeded summary of the available commands. Overridden wholesale by ai ask --system.

int _knit_ai_describe_summary()

Print a compact one-line-per-command summary of the whole command tree ("- <full command path>: <description>"), used to seed the model's first turn so it need not call knit_describe just to learn what commands exist. Built from the machine-readable describe --format json --compact output. Best-effort: a failure prints nothing (the model can still call the tools for detail).

int _knit_ai_discover()

Register the ai subcommands (ask / query). Called lazily by the framework the first time an ai subcommand is resolved, listed in "--help", or walked by "describe". The query_format and ai_query_lang enums stay registered eagerly above, because src/query.sh also resolves query_format at its own registration time.

Parameters:

• parent -- [in] The parent command's display name (unused).

int _knit_ai_dispatch_tool()

Route a model-requested tool call to its handler and print the (truncated) result. Tool arguments arrive as a JSON object string and are parsed with knit_jq -r. Only allowlisted tools run; an unknown name yields an error string. Handler output (including any fatal from the underlying knit surface, e.g. an unknown job id) is captured so it becomes the tool result rather than aborting the agentic loop.

Before running the handler, the recording-suppression state is cleared (unset KNIT_DISABLE_RECORDING, reset _KNIT_RECORDING_SUPPRESSED) so a recordable tool-command would record exactly as if the user had run it directly. The ai command's own non-recording state never leaks into the commands it drives.

Parameters:

• name -- [in] The tool name (e.g. "knit_describe").

• args_json -- [in] JSON object string of the tool's arguments.

int _knit_ai_extract_query()

Extract the bare query and its language from a model reply, tolerating a reply wrapped in a Markdown code fence (sql … / cypher … ) and surrounding whitespace even though the system prompt asks for none. When a fence is present, the content between the first and next fence is taken; the fence info string (first line) is read as the language, case-insensitively (sql/cypher), and dropped. When the info string is missing or unknown, the language is inferred from the leading keyword: MATCH/OPTIONAL/UNWIND/CALL ⇒ cypher, SELECT/WITH/EXPLAIN/PRAGMA ⇒ sql, anything else ⇒ sql. Leading and trailing whitespace is trimmed from the query.

Parameters:

• lang_out -- [out] (nameref) receives the detected language ("sql" or "cypher").

• query_out -- [out] (nameref) receives the trimmed query text.

• text -- [in] The raw model reply.

int _knit_ai_loop()

Run the agentic tool-calling loop for ai ask. Seeds the conversation with the system prompt and the user question, then repeatedly POSTs the accumulating message array (with the read-only tool schema) to the provider. On each turn: the assistant message is appended; if it carries no tool calls it is the final answer (printed and the loop returns); otherwise every requested tool is dispatched and its result appended as a tool-role message before the next turn. The message array is grown with knit_jq --argjson so JSON types stay intact.

Stops after max_iterations rounds with a warning if no final answer is reached.

Parameters:

• base_url -- [in] Resolved endpoint base URL.

• api_key -- [in] Resolved API key (passed straight to the request helper).

• model -- [in] Resolved model id.

• question -- [in] The user's natural-language question.

• system_prompt -- [in] The system prompt to seed the conversation with.

• max_iterations -- [in] Hard cap on tool-call rounds.

• raw -- [in] "true" to print the raw final message JSON instead of its text.

• verbose -- [in] "true" to stream each tool call and result to stderr.

Returns:

0 when a final answer is produced, 1 on hitting the iteration cap.

int _knit_ai_query()

int _knit_ai_query_loop()

Run the bounded self-correcting loop behind ai query. Seeds the conversation with the query system prompt and the user question, then per round: asks the model for a single statement, extracts it with its language, and (unless --query-only) routes it to the matching read-only backend. SQL is guarded with the shared read-only check and run on the read path (_knit_sqlite3, never _knit_sqlite3_write); Cypher is transpiled to SQL by knit-cypher-to-sql (_knit_cypher_to_sql) reading the lens's flat schema on stdin, and that SQL is then run like any other read query. On success the formatted result is printed and the loop returns 0. A guard rejection or a backend error is fed back to the model as a follow-up message so the next round can correct it (including switching language). After max_iterations failed rounds the loop fatals, showing the last query and error.

A pinned language (lang != "auto") overrides the per-statement detection so the query is always routed to that backend.

With --query-only the first generated statement and its detected language are printed (the query on stdout, the language on stderr) and the loop returns without touching either backend.

Parameters:

• base_url -- [in] Resolved endpoint base URL.

• api_key -- [in] Resolved API key (passed straight to the request helper).

• model -- [in] Resolved model id.

• question -- [in] The user's natural-language question.

• system_prompt -- [in] The system prompt to seed the conversation with.

• max_iterations -- [in] Cap on generate→run→fix rounds.

• verbose -- [in] "true" to stream the chosen language, each generated query, and any backend error to stderr.

• query_only -- [in] "true" to print the generated query and its language and return without running either backend.

• format -- [in] The query_format enum value for the output mode.

• no_header -- [in] "true" to omit column headers.

• separator -- [in] Optional column separator for csv/list modes.

• lang -- [in] Pinned language ("auto"/"sql"/"cypher"); "auto" defers to the per-statement detection, sql/cypher override it.

• lens_array -- [in] Optional name of the resolved lens database array (current first, extras after). When given, generated SQL runs over that read-only lens; when empty, SQL runs directly on the read path (used by standalone unit calls that need no lens).

Returns:

0 on a successful (or --query-only) run; fatals on hitting the cap.

int _knit_ai_query_mode_args()

Translate an ai query --format value (a query_format enum value) plus the --no-header/--separator options into the sequence of sqlite3 -cmd ".mode …" arguments that select the requested output mode, filled into the caller's array by nameref. The enum values map 1:1 onto sqlite3 .mode names, so no lookup table is needed. Headers are on by default (.headers on) and turned off by --no-header; a non-empty separator overrides the mode default.

Parameters:

• __knit_ret -- [out] Name of the array variable to fill with the sqlite3 args.

• format -- [in] The query_format enum value (e.g. "box", "csv").

• no_header -- [in] "true" to omit column headers.

• separator -- [in] Optional column separator for csv/list modes.

int _knit_ai_query_system_prompt()

Print the system prompt for ai query. It instructs the model to translate the question into exactly ONE read-only query wrapped in a single language-tagged fenced block, and seeds it from live state: the SQLite schema (_knit_sqlite3 ".schema"), the node-label map (_knit_query_build_names), the provenance edge model, and a compact describe summary for column and edge semantics.

The SQL and Cypher halves are gated by the pinned language: with sql or cypher only that half (and the matching reply contract) is emitted so the model isn't tempted to use the other backend; with auto (the default) both halves and the when-to-prefer-each guidance are emitted and the model chooses.

When has_extra is "true" (the caller passed --extra), a cross-platform note is appended to the SQL half: the query runs over a read-only union of several platforms' databases, and a synthesized platforms view lists them.

Parameters:

• lang -- [in] Pinned language ("auto"/"sql"/"cypher"); defaults to "auto".

• has_extra -- [in] "true" when the query spans a cross-platform lens.

int _knit_ai_resolve_config()

Resolve the provider access configuration for an ai call. Reads the ai.* metadata keys, indirect-expands the stored env-var names to their values (never eval), and applies precedence/fallbacks:

• api key: value of the env var named by ai.api_key_env (required).

• base url: value of the env var named by ai.base_url_env if set, else the ai.base_url literal, else _KNIT_AI_DEFAULT_BASE_URL.

• model: the model_override argument if non-empty, else the value of the env var named by ai.model_env, else the ai.model literal.

Fatals (with a hint pointing at bootstrap --ai-*) when the provider is unconfigured, the API key env var is empty/unset, or no model can be resolved. The resolved API key is returned only through the caller-named output variable; it is never logged, traced, or echoed.

Parameters:

• __knit_ret1 -- [out] Name of the variable to hold the resolved API key.

• __knit_ret2 -- [out] Name of the variable to hold the resolved base URL.

• __knit_ret3 -- [out] Name of the variable to hold the resolved model id.

• model_override -- [in] Optional model id that takes precedence over metadata.

int _knit_ai_sql_is_readonly()

Shared read-only SQL guard for the knit_db_query tool and (in a later milestone) ai query. A statement is considered read-only when its leading keyword is one of SELECT / WITH / EXPLAIN / PRAGMA and it contains no write keyword (INSERT / UPDATE / DELETE / DROP / ALTER / CREATE / REPLACE / ATTACH) as a whole word anywhere. The second check catches piggy-backed writes such as "SELECT 1; DROP TABLE runs" that pass the leading-keyword test.

Word boundaries are handled by uppercasing the statement and replacing every non-word character with a space, then matching the space-padded keyword; this avoids relying on the non-POSIX \b regex escape and never misfires on a column named e.g. created_at (the underscore keeps it one token).

Parameters:

• sql -- [in] The SQL statement to check.

Returns:

0 if the statement is read-only, 1 otherwise.

int _knit_ai_store_config()

Write the provider-access configuration to the metadata table as ai.* key/value pairs, through the same metadata store path used elsewhere. This is the writer used by bootstrap's --ai-* options to store a full config in one shot (update mode writes each key on its own).

Only env-var names and non-secret defaults are stored; no API key ever reaches the database.

Parameters:

• api_key_env -- [in] Name of the env var holding the API key.

• base_url_env -- [in] Name of the env var holding the endpoint base URL.

• model_env -- [in] Name of the env var holding the model id.

• base_url -- [in] Literal fallback base URL.

• model -- [in] Literal fallback model id.

• overwrite -- [in] "true" to overwrite existing keys, anything else to fail on a duplicate key.

int _knit_ai_tool_db_query()

Tool handler: run a read-only SQL query against the experiment database and print the result (aligned columns with a header). The statement must pass _knit_ai_sql_is_readonly; a rejected statement returns an error string (fed back to the model) rather than being run. The query always runs on the read path (_knit_sqlite3, never _knit_sqlite3_write), so it can never mutate the DB.

Parameters:

• sql -- [in] The SQL statement to run.

int _knit_ai_tool_describe()

Tool handler: structured introspection of the command tree, as YAML. Optional only (comma-separated command list) and recursive narrow/expand the scope.

Parameters:

• only -- [in] Optional comma-separated command list (colon form, e.g. "a,b:c").

• recursive -- [in] "true" to also include the selected commands' subcommands.

int _knit_ai_tool_help()

Tool handler: the --help text for one specific command. The command name may contain spaces for a nested command (e.g. "job show stdout"); it is split into words before being passed to knit.

Parameters:

• command -- [in] The command whose help to show (space-separated for nesting).

int _knit_ai_tool_job_output()

Tool handler: a recorded job's captured stdout/stderr or its generated batch script, via the job show <stream> subcommands (there is no top-level knit stdout command).

Parameters:

• id -- [in] The job UUID.

• stream -- [in] One of "stdout" (default), "stderr", or "script".

int _knit_ai_tool_metadata_show()

Tool handler: all experiment metadata. Everything in the metadata table is safe to send (only env-var names and non-secret config, never secrets).

int _knit_ai_tools_schema()

Print the OpenAI tools[] JSON array describing the read-only tool set exposed to the model. There is deliberately no command-execution tool: the model can inspect the experiment but cannot run experiment commands or mutate anything.

int _knit_ai_truncate()

Print the given text, cut to _KNIT_AI_TOOL_OUTPUT_MAX_BYTES with an explicit "…(truncated)" marker appended when it is longer. Used to bound every tool result handed back to the model.

Parameters:

• text -- [in] The text to bound.

Variables

String _KNIT_AI_DEFAULT_BASE_URL

Literal fallback base URL used when no base-url env var resolves to a value. Kept in sync with the same default hard-coded on bootstrap's --ai-base-url option (src/boostrap.sh loads before this file, so it cannot reference this variable at registration time).

String _KNIT_AI_TOOL_OUTPUT_MAX_BYTES

Approximate ceiling on the size of a single tool result handed back to the model. A tool whose output exceeds this is cut to the budget and marked with an explicit "…(truncated)" line, so a huge describe or query result can't blow the context window silently. Measured in characters (== bytes for ASCII, which covers knit's introspection output).

app.sh

Functions

int _knit_app_before_cb()

Before-callback installed on every app subcommand by knit_register_app. Verifies that KNIT_JOB_PREFIX is set: an app only runs inside a job (launched by the run dispatcher and forwarded to each rank by the launcher), so its absence means the app command was invoked directly. Mirrors _knit_setup_before_cb / _knit_job_before_cb.

int _knit_run()

Entry point for the run CLI command: the user-facing dispatcher that launches an app across a subset of the surrounding job's allocation. A run has no meaning outside a job (the job supplies the node list the launcher places ranks on), so this fails fast when invoked outside one (KNIT_JOB_PREFIX unset), analogous to how job and setup commands reject direct invocation.

It validates the app name and its arguments, resolves the placement options into a concrete (procs, procs-per-node, hosts) triple, resolves the launcher backend, and execs the per-rank worker (_run) under that launcher. The app runs in the context of the job's directory (there is no separate run directory); the run's UUID serves only as its id in the runs table. Its exit status is the launcher's.

Usage:

./exp.sh run [placement-opts...] -- app-name [args...]   (inside a job)

int _knit_run_checksum_inputs()

Verify existence of, and hash, every checksummed file/directory input of an app once, on the login side, before it is launched. A missing input then fails fast — before an allocation is used or ranks are spawned — and each input is hashed exactly once rather than once per rank. The bare digests are returned by name (param -> 64-hex); the dispatcher forwards them to rank 0 through the launcher environment (KNIT_CHECKSUM_<param>) so the app's row records the same digest without any rank touching the filesystem. Existence is checked for every file/directory input, even one that opted out of the digest with --no-checksum; only a checksummed input is hashed.

Parameters:

• out_name -- [out] Name of the associative array to fill (param -> bare 64-hex).

• subcmd -- [in] Mangled app command name (run:<app>).

• ... -- [in] The app's expanded invocation arguments.

int _knit_run_checksum_outputs()

Verify existence of, and hash, every checksummed file/directory output of an app once, on the login side, after the launcher returns — never on a rank. An app's body runs on every rank, but only rank 0 records the per-app row, and it records only the output paths (the checksum columns are left empty by the M4 output hook, which no-ops in an app worker). Hashing here, after launch, keeps it off every rank and out of every measured duration: hashing on rank 0 would keep the launcher blocked while the other ranks idle at finalize, inflating the run's measured wall-clock.

Rank 0's per-app row is found through the provenance graph: the "run -> run:app" call edge (source_id is this run's UUID) points at the row's id. For each checksummed output, the recorded path is read back from that row, its existence verified, the digest computed, and the row's companion "<param>_checksum" column updated. An output left with no value, or one that opted out with --no-checksum, is skipped. When nothing was recorded (recording disabled, or no such row) this is a no-op.

When tolerant is "false" (a successful run) a missing output is fatal (a broken postcondition). When tolerant is "true" (a FAILED run whose row survives) a missing output is skipped instead — a failed app may have produced only some outputs — while one that IS present is still hashed and recorded.

Parameters:

• subcmd -- [in] Mangled app command name (run:<app>).

• app_name -- [in] The app name (its table is named after it).

• run_uuid -- [in] This run's UUID (source of the provenance edge to the row).

• tolerant -- [in] "true" to skip (not fatal on) a missing output; default "false" enforces existence.

int _knit_run_delete_row()

Remove a runs-table row (and any provenance edge referencing it) that was eagerly recorded before launch but must not survive. The run dispatcher records the runs row up front so even a failed launch leaves a trace; when the launched app opted out of recording a failed invocation (knit_no_record_on_failure) and the run then fails, that trace would dangle (the app's per-app row and the "run -> run:<app>" edge were never written), so it is removed here. Best-effort and bootstrap-gated: with no database there is nothing to remove. Mirrors _knit_submit_cleanup_rejected (minus the job directory and alias a run does not have).

Parameters:

• uuid -- [in] The run's UUID (its runs-row id and the edge's target_id).

int _knit_run_normalize_mpi_env()

Normalize the launcher-native MPI environment into the launcher-agnostic KNIT_MPI_* variables the app body reads, and export them so every subprocess of the rank inherits them:

KNIT_MPI_RANK rank in MPI_COMM_WORLD KNIT_MPI_SIZE size of MPI_COMM_WORLD KNIT_MPI_LOCAL_RANK node-local rank

Each is taken from the first launcher that set it, by precedence (OpenMPI -> MPICH/PMI -> Slurm srun -> PALS -> Flux), falling back to a single rank-0 / size-1 process when none are present (the none backend). Called once per rank by the worker, before forwarding to the app body.

int _knit_run_record_exit_status()

Fill in the runs row's reserved "__exit_status__" column after the launcher returns. The row is recorded eagerly, before the launch, so a failed launch still leaves a trace; the exit status is left NULL there because the outcome is not yet known. This writes the launcher's exit status once it is. Best-effort and bootstrap-gated (with no database there is nothing to update). Only called when the runs row survives — never after _knit_run_delete_row removed it.

Parameters:

• uuid -- [in] The run's UUID (its runs-row id).

• exit_status -- [in] The launcher's exit status.

int _knit_run_resolve_placement()

Resolve the three placement options (--procs, --procs-per-node, --hostnames) into a concrete (procs, procs-per-node, hosts) triple, stored by name into the caller's associative array (keys: procs, procs-per-node, hostnames). procs is always set; procs-per-node may be left empty (the launcher's default distribution applies); hostnames is always set to a comma-separated list.

Let A be the job's allocated unique hosts (knit_job_hostnames), a = |A|; c the per-node core count (node_ncpus metadata, may be unknown); n = --procs, p = --procs-per-node, H = --hostnames (k = |H|). Resolution:

1. Hosts. If --hostnames is given, each entry must be one of A (fatal otherwise); else H defaults to A. 2/3. procs / procs-per-node. If both n and p are given they are used as-is (see the consistency checks). If only n is given, H is respected; if H was explicit, p is derived as n/k. If only p is given, n is derived as p*k. If neither is given, n = c*k with p = c when c is known, else n = k (one rank per node) with a warning.

2. Consistency (fatal on conflict): n and p given => require np==0 (kreq=n/p); with explicit H require k==kreq, else require kreq<=a and take the first kreq hosts of A. n given without p but with explicit H => require nk==0 (p=n/k).

Parameters:

• out_name -- [out] Name of the associative array to fill.

• ... -- [in] The dispatcher's invocation arguments (read via knit_get_parameter).

int _knit_run_worker()

Hidden per-rank worker executed once per rank under the launcher. The run dispatcher translates placement once and execs <launcher> [flags] ./exp.sh _run -- <app> [app opts]; this worker runs on every rank, then forwards to the app body by re-entering the command machinery as run:<app> (both tokens are non-"--", so this routes to the app, never back to the run dispatcher — there is no recursion).

Each rank normalizes the launcher-native MPI environment into KNIT_MPI_* (via _knit_run_normalize_mpi_env) before forwarding to the app, and every rank but rank 0 sets _KNIT_RECORDING_SUPPRESSED so a run's outputs and per-app row are recorded exactly once.

By default an app has no setup of its own: it inherits the surrounding job's setup environment (forwarded by the launcher). An app may still declare knit_with_setup to explicitly depend on and re-source a setup (see knit_register_app).

Variables

AssociativeArray _KNIT_APPS

Associative array mapping registered app names to 1. Used to validate that an app name passed to knit run is known. Mirrors _KNIT_JOBS for jobs.

String _KNIT_RUNS_TABLE

Name of the table recording every run: the app launched, the requested placement, and the launcher. The row id is the run's own UUID. The parent job and the rank-0 per-app row are linked through the provenance graph — the "submit:<job> -> run" and "run -> run:<app>" call edges — not through a stored column or a shared id (each mints its own distinct UUID).

artifact.sh

Functions

int _knit_artifact_cardinality_phrase()

Store the human-readable cardinality phrase for a quantifier in the caller-named variable: "zero or more" for "*", "one or more" for "+", and the empty string for a scalar (empty quantifier). Shared by every describe format and --help so a collection reads the same way everywhere.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the phrase (empty if scalar).

• quant -- [in] Cardinality quantifier ("*", "+", or empty).

int _knit_artifact_check_lower_bound()

Enforce the lower bound of every variadic "+" output the command declared: a "name:kind+" collection must bind at least one member. Called at record time, after the body (see _knit_artifacts_record_sql), so a "+" output the body left unbound is fatal and names the offending output. A "*" output that bound nothing is allowed (a fan-out that produced nothing is not an error), and a scalar output has no lower bound here (its at-most-once rule lives in knit_artifact). Iterates the command's declared artifacts set and consults the per-invocation binding stash.

Parameters:

• cmd -- [in] Mangled command name of the producer.

• demangled -- [in] Demangled producer name, used in the fatal message.

int _knit_artifact_ensure_table_cb()

knit_done callback (installed by _knit_artifact_require_table) that gives an artifact-producing command a table when it declared none, so its invocation is recorded and can be the source of a "produced" edge. A command that declared its own table with knit_with_table is left untouched. The auto-created table takes the command's own (demangled) name, mirroring knit_with_table's default, is registered in the names map, and is set up immediately (deferred to first use when the experiment is not yet bootstrapped, exactly like knit_with_table).

Parameters:

• cmd -- [in] Mangled command name.

int _knit_artifact_is_bound()

Return 0 when the command bound the named artifact at least once in this invocation, 1 otherwise. Consults the per-invocation binding stash filled by knit_artifact (KNIT_CMD<cmd>_artifact_bound, keyed by normalized name).

Parameters:

• cmd -- [in] Mangled command name.

• name -- [in] Normalized artifact name.

Returns:

0 if bound, 1 otherwise.

int _knit_artifact_kind_type()

Look up the physical type ("file" or "directory") backing an artifact kind and store it in the caller-named variable. Returns non-zero when the kind is not registered, leaving the variable untouched. Consulted by the output and input artifact directives to resolve a declared "name:kind" annotation to its physical type.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the physical type.

• kind -- [in] The artifact kind name.

Returns:

0 if the kind is registered, 1 otherwise.

int _knit_artifact_parse_kind()

Split an optional trailing cardinality quantifier off the kind portion of an artifact spec (the text after "name:"). A single trailing "*" (zero or more) or "+" (one or more) marks the artifact as a variadic collection; a kind with no quantifier is scalar. The bare kind (quantifier removed) and the quantifier character ("" for scalar, "*", or "+") are stored in the caller's variables. Shared by knit_with_output_artifact and knit_with_input_artifact so the syntax and its validation cannot drift between the two directions.

A kind carrying more than one trailing quantifier (e.g. "file**" or "file*+") is fatal, so a typo is caught at registration rather than silently dropped. The bare kind is not validated here (each caller validates it against the kind registry as before).

Parameters:

• __knit_ret1 -- [out] Name of the variable to hold the bare kind.

• __knit_ret2 -- [out] Name of the variable to hold the quantifier ("", "*", "+").

• raw_kind -- [in] The kind portion of the spec (may carry a quantifier).

• context -- [in] Caller name, used in the fatal message.

int _knit_artifact_require_table()

Guarantee that the command currently being registered will record an invocation row, so a "produced" edge from it has a source. Called by knit_with_output_artifact. The row comes from the command's table; if the command declares its own table (knit_with_table) nothing more is needed, so this only pushes a knit_done callback (_knit_artifact_ensure_table_cb) that creates a default table when none was declared. The callback is pushed at most once per command, however many artifacts it declares.

int _knit_artifact_resolve_path()

Resolve a user-supplied <linked-path> into the artifact entry's absolute location and its artifacts-relative form, enforcing containment on the entry's OWN location, not its target. A relative <linked-path> is taken against the artifacts root; an absolute one is used as given. The parent directory is resolved to its real path (so a symlink in the parent chain is followed), while the final component is kept verbatim (a symlink there is not followed, so a symlink artifact stays inside the artifacts root even when its target is elsewhere). Containment holds when that real parent is the artifacts root or a directory below it.

Parameters:

• __knit_ret1 -- [out] Name of the variable to hold the absolute entry path.

• __knit_ret2 -- [out] Name of the variable to hold the artifacts-relative path.

• root -- [in] The resolved (absolute) artifacts root.

• linked_path -- [in] The user-supplied <linked-path>.

Returns:

0 on success (contained); 1 when <linked-path> is outside the root.

int _knit_artifact_root()

Store the resolved artifact root — the directory under which artifacts live — in the caller-named variable. Reads the verbatim artifact_path from the metadata table (falling back to "artifacts" when unset, for robustness) and resolves it against the experiment root via _knit_resolve_experiment_path. Mirrors _knit_setup_root / _knit_job_root / _knit_resource_root.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the resolved artifact root.

int _knit_artifacts_create_table()

Create the artifacts table if it does not already exist. Each row is one produced artifact: a stable identity ("id", a uuidv7, the target of the "produced" edge), the artifacts-relative "path" (UNIQUE, since an on-disk entry is write-once), the declared "name" the producer used, the physical "type" ("file" or "directory"), the semantic "kind" the producer declared ("csvfile", "rundir", ...; "file"/"directory" for a bare builtin), the content "checksum" ("sha256:<hex>"), and a "result" flag (1 when declared --result, else 0). The schema is fixed here rather than derived from a command's declared outputs (unlike a per-command table), so its column order and the UNIQUE constraint on "path" are explicit. Called at bootstrap alongside the metadata and provenance tables.

int _knit_artifacts_ensure_table()

Ensure the artifacts table exists before an artifact row is written, creating it lazily on first use. A freshly bootstrapped experiment already has the table (created at bootstrap); a database bootstrapped before this feature shipped does not, so ensuring it here lets a new invocation record artifacts rather than failing. The create is idempotent and runs at most once per process, guarded by _KNIT_ARTIFACTS_TABLE_ENSURED. Mirrors _knit_prov_ensure_table.

int _knit_artifacts_path_recorded()

Return 0 when the given artifacts-relative path already has a row in the artifacts table (recorded by an earlier invocation, possibly in a previous run), 1 otherwise. knit_artifact consults this to turn the write-once "path" UNIQUE constraint into a clear fatal before the INSERT is built, rather than leaking sqlite3's raw "UNIQUE constraint failed: artifacts.path" at record time. It is a read, so no lock is taken. It reports "not recorded" (returns 1) when the experiment is not bootstrapped or the artifacts table does not yet exist, since no row can be recorded then.

Parameters:

• path -- [in] Artifacts-relative path to look up.

int _knit_artifacts_record_sql()

Build (into the caller's variable, without executing) the SQL that records every artifact the given command bound during this invocation: one artifacts row plus one "producer --produced--> artifact" edge per stashed binding. The result is spliced into the producing row's own transaction (see _knit_db_record_invocation) so the row, its "call" edge, and its produced artifacts are written atomically.

Each binding was stashed by knit_artifact keyed on the artifacts-relative path; the name and content digest come from that stash, while the physical type ("file" or "directory"), the semantic kind, and the "result" flag are recovered from registration state (the fileparam marker holds the physical type, the 3<name>_type field holds the kind, and the results set holds the flag). A fresh uuid identifies each artifacts row (the target of its "produced" edge). The caller's variable is left empty when the command bound no artifact.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the built SQL.

• cmd -- [in] Mangled command name of the producer.

• producer_id -- [in] Row id of the producing invocation (the edge source).

• producer_name -- [in] Demangled producer name (the edge source_name).

int _knit_artifacts_row_sql()

Build (print, without executing) the INSERT for one artifacts row: one produced artifact, with its uuid "id" (the target of the "produced" edge), its artifacts-relative "path" (UNIQUE), the declared "name", the physical "type" ("file" or "directory"), the semantic "kind" ("csvfile"/"rundir"/...; "file"/"directory" for a bare builtin), the content "checksum", and the "result" flag. The text columns are single-quoted and escaped; "result" is emitted as a bare 0/1 integer (any value other than "1" becomes 0). Meant to be composed with a "produced" edge (see _knit_produced_edge_sql) into one transaction at record time, mirroring how _knit_prov_edge_sql composes into _knit_db_record_invocation.

Parameters:

• id -- [in] UUID of the artifact row.

• path -- [in] Artifacts-relative path (UNIQUE).

• name -- [in] Declared artifact name.

• type -- [in] Physical type: "file" or "directory".

• kind -- [in] Semantic kind the producer declared.

• checksum -- [in] Content digest of the resolved target ("sha256:<hex>").

• result -- [in] "1" for a declared result, else 0.

int _knit_input_artifact_after_cb()

After-callback installed by knit_with_input_artifact on the consuming command, one per declared input-artifact parameter. It records a "used_by" edge from each consumed artifact to the command itself. It runs as an after-callback (not the before-callback that validates the artifact) because the consumer's frame is on the executing stacks only from push time onward, so the consumer's resolved row id — the edge target — is available here but not in the before-callback (this mirrors _knit_resource_dep_after_cb).

The declared parameter name, required kind, verify flag, and cardinality quantifier are bound at registration time and carried as the first four arguments (the parameter name and the quantifier are used here); the trailing arguments are the command's own runtime arguments, scanned for the parameter value. A missing value is silently skipped: the before-callback already fataled on it, so this is only reached with a valid value.

A scalar input records the single edge for its one value. A variadic input resolves its comma-separated value with the shared list resolver (the same one the before-callback validated against) and records one "used_by" edge per resolved element, so every consumed member joins the lineage; an empty "*" list resolves to nothing and records no edge.

Parameters:

• param -- [in] Normalized name of the input-artifact parameter to read.

• kind -- [in] Required artifact kind (unused; kept for argument symmetry).

• verify -- [in] Checksum opt-in flag (unused; kept for argument symmetry).

• quant -- [in] Cardinality quantifier ("*"/"+"), empty for a scalar input.

int _knit_input_artifact_before_cb()

Before-callback installed by knit_with_input_artifact on the consuming command, one per declared input-artifact parameter. Before the command body runs it resolves the parameter value and validates it. A scalar input carries a single artifacts-relative path, validated as one element. A variadic input ("*"/"+") carries a comma-separated list, resolved (split then glob, see _knit_input_artifact_resolve_list) into elements that are validated in turn; the first bad element is fatal and names which element. A missing value for a scalar input is fatal, an empty "+" list is fatal, and an empty "*" list validates trivially (the body simply receives no element). Each element is checked by _knit_input_artifact_validate_one (existence + kind, plus an opt-in checksum). Returning normally lets the command proceed; a fatal aborts it before the body runs.

The declared parameter name, required kind, verify flag, and cardinality quantifier are bound at registration time and carried as the first four arguments; the trailing arguments are the command's own runtime arguments, scanned for the parameter value. Mirrors _knit_resource_dep_before_cb.

Parameters:

• param -- [in] Normalized name of the input-artifact parameter to read.

• kind -- [in] Artifact kind the resolved artifact(s) must carry.

• verify -- [in] "1" to re-verify the recorded checksum, empty to skip.

• quant -- [in] Cardinality quantifier: "" scalar, "*" zero or more, "+" one or more.

int _knit_input_artifact_param_kind()

Store the artifact kind a parameter was declared with in the caller-named variable, or the empty string when the parameter is an ordinary parameter (not declared through knit_with_input_artifact). Reads the per-parameter marker (KNIT_CMD<cmd>input_artifact) that knit_with_input_artifact records, so describe and --help can annotate an input-artifact parameter with its required kind from the registration tables alone (no database read). The parameter name must be normalized, as it is stored in the parameter sets. Mirrors _knit_resource_param_type.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the artifact kind (empty if none).

• cmd -- [in] Mangled command name.

• param -- [in] Normalized parameter name.

int _knit_input_artifact_quantifier()

Store the cardinality quantifier a consumed artifact parameter was declared with in the caller-named variable: "*" (zero or more), "+" (one or more), or the empty string for a scalar parameter. Reads the per-parameter marker (KNIT_CMD<cmd>input_artifact_variadic) that knit_with_input_artifact records, so describe and --help can mark a collection distinctly from a scalar from the registration tables alone (no database read). The parameter name must be normalized, as it is stored in the parameter sets.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the quantifier (empty if scalar).

• cmd -- [in] Mangled command name.

• param -- [in] Normalized parameter name.

int _knit_input_artifact_record_used_by_edge()

Record a "used_by" provenance edge from a consumed artifact to a consuming invocation. The edge's source is the artifacts row (its id resolved by path, its node name the "artifacts" label — the same node the "produced" edge points at, so lineage joins as A --produced--> artifact --used_by--> B); its target is the consumer. Delegates the gated write to _knit_record_used_by_edge, so it records nothing when recording is disabled, on a suppressed rank, before bootstrap, when the target does not participate in the graph, or when no row is found at the path (empty id -> no edge). Mirrors _knit_resource_record_used_by_edge.

Parameters:

• value -- [in] Artifacts-relative path of the consumed artifact.

• target_cmd -- [in] Mangled command name of the consumer (the edge target).

• target_id -- [in] Resolved row id of the consumer (the edge target).

int _knit_input_artifact_resolve_list()

Resolve a variadic input-artifact value into an array of artifacts-relative paths. The raw value is a comma-separated list; each element is resolved in two steps: split on the comma, then expand. An element that holds a glob metacharacter ("*", "?", "[") is expanded with bash pathname expansion relative to the artifacts root, under nullglob, so a pattern matching nothing yields no element (never the literal pattern) and recursive "**" stays disabled; the artifacts-root prefix is then stripped so each match is an artifacts-relative path (the form the artifacts row stores). An element with no metacharacter is taken verbatim. An empty element (from an empty value or a stray comma) is skipped. The result keeps element order, and within one globbed element bash's sorted order, and is de-duplicated first-seen, so overlapping patterns do not record a path twice. Word splitting is disabled during expansion, so a root or element that holds a space stays intact. Shared by the input-artifact before-callback, after-callback, and knit_input_artifact_paths, so validation, edges, and the body's view cannot drift.

Parameters:

• __knit_ret -- [out] Name of the array variable to fill with relative paths.

• raw -- [in] The raw parameter value (comma-separated list of paths).

• root -- [in] The resolved (absolute) artifacts root.

int _knit_input_artifact_validate_one()

Validate one resolved input-artifact element against the recorded artifacts table: the path must name a recorded artifacts row (an unbootstrapped experiment has recorded none, so a missing row is treated as "no artifact"), the recorded kind must equal the declared kind (the physical type is implied by the kind, so it needs no separate check), and — when --verify-checksum was declared — the recomputed digest of the resolved entry must equal the recorded checksum. Any failure is fatal and names the offending element, so the first bad element of a variadic list aborts the command before its body runs. Shared by the scalar and variadic paths of _knit_input_artifact_before_cb.

Parameters:

• kind -- [in] Artifact kind the element must carry.

• verify -- [in] "1" to re-verify the recorded checksum, empty to skip.

• value -- [in] The element's artifacts-relative path.

int _knit_output_artifact_quantifier()

Store the cardinality quantifier a produced artifact was declared with in the caller-named variable: "*" (zero or more), "+" (one or more), or the empty string for a scalar artifact. Reads the per-name marker (KNIT_CMD<cmd>artifact_variadic<name>) that knit_with_output_artifact records, so describe and --help can mark a collection distinctly from a scalar from the registration tables alone (no database read). The name must be normalized, as it is stored in the artifacts set.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the quantifier (empty if scalar).

• cmd -- [in] Mangled command name.

• name -- [in] Normalized artifact name.

int _knit_register_artifact()

Add an artifact of the command being registered to its artifacts set: a name that refers to a file or directory that lives under the artifacts root and is bound at runtime with knit_artifact. Membership in this set is what marks a name as an artifact rather than an ordinary value output; it drives how the command is described and, later, how knit_artifact validates the name. An artifact is not an output column, so it is kept out of the outputs set.

The per-command KNIT_CMD<cmd>_artifacts set is created as associative on first use (knit_register does not create it, since not every command has an artifact).

Only meaningful in a command context; a call with no command being registered is a no-op.

Parameters:

• name -- [in] The declared (un-normalized) artifact name.

Variables

String _KNIT_ARTIFACTS_TABLE

Name of the framework-owned table that records one row per produced artifact (a first-class provenance node, like "jobs" and "runs"). A "produced" edge in the provenance table links a producing invocation to the artifact row whose id it names. A direct names-map entry (see below) resolves "artifacts" as a graph node label; the table has no owning command.

String _KNIT_ARTIFACTS_TABLE_ENSURED

Set to "1" once the artifacts table has been ensured in this process (see _knit_artifacts_ensure_table), so the idempotent CREATE runs at most once per run. Mirrors _KNIT_PROV_TABLE_ENSURED.

AssociativeArray _KNIT_ARTIFACT_KINDS

Registry mapping each artifact kind name to its backing physical type ("file" or "directory"). A kind is an artifact's semantic category (e.g. "csvfile", "rundir"); the physical type drives existence checks and checksumming. Seeded with the builtin kinds "file" and "directory" (each backed by its own type) and the "dir" alias (backed by "directory"), so a bare "name:file" / "name:directory" declaration needs no knit_register_artifact call and re-registering one of them is fatal. Extended by knit_register_artifact and consulted by _knit_artifact_kind_type. Declared global (-g) so it survives being sourced from within a function (as the bats tests do). Mirrors _KNIT_RESOURCES / _KNIT_ENUMS.

AssociativeArray _KNIT_ARTIFACT_KIND_DESCRIPTIONS

Companion registry mapping each artifact kind name to its one-line description, shown by knit describe / --help. Populated alongside _KNIT_ARTIFACT_KINDS; the builtin kinds carry a short default description.

boostrap.sh

Functions

int _knit_bootstrap()

int _knit_bootstrap_check_prerequisites()

Verify the external tools that Knit needs at run time but does not itself provision are present on PATH, and fail bootstrap early (with a clear message) when one is missing. Currently checks sha256sum, which content checksums for file/directory parameters and outputs depend on. Uses _knit_command_path so it is stubbable in tests.

int _knit_bootstrap_dir_has_subdir()

Return 0 when the given directory holds at least one subdirectory, non-zero otherwise (including when the directory itself is absent). Used by update-mode --resource-path relocation to refuse moving a resource root that already holds fetched resource instances (each is a subdirectory).

Parameters:

• dir -- [in] Directory to scan.

int _knit_bootstrap_jobs_recorded()

Return 0 when the "jobs" table exists and holds at least one row, non-zero otherwise. Used by update-mode --job-path relocation to refuse moving a job root that already holds submitted jobs. The table is created lazily on the first submission, so an experiment that never submitted a job has no "jobs" table at all; a missing table reports "no jobs".

int _knit_bootstrap_on_exit()

Clean up on exit if bootstrap did not complete successfully.

int _knit_bootstrap_relocate_path()

Update-mode handler for a path-root option (--setup-path, --job-path, --resource-path). When the option is typed and its value differs from the stored one, relocate the root — but only when the current root holds no artifacts of its kind, since setups, jobs, and resources are not movable:

• setup: no user setup exists (the builtin "default" does not count).

• job: no job row is recorded in the "jobs" table.

• resource: no resource instance directory exists under the root.

A non-empty root is a hard stop (fatal). On an allowed relocation it stores the new value, recreates the builtin "default" setup under the new root (setup kind only), and removes the old root directory (which, for the setup kind, still holds the old "default", so a plain rmdir would not suffice). An untyped option or a value equal to the stored one is a no-op.

Parameters:

• kind -- [in] One of "setup", "job", "resource".

• new_value -- [in] Resolved value typed for the option.

• ... -- [in] Raw argument tokens of this invocation (see _KNIT_INVOCATION_RAW_ARGS), used to tell a typed option from a defaulted one.

Returns:

0 when the root was relocated, 1 when nothing changed.

int _knit_bootstrap_update()

Run bootstrap in update mode on an experiment that is already bootstrapped. It changes only the options the user typed on this call and leaves the database and every other setting untouched; it installs no destructive exit trap. A .knit/ directory without a database is malformed and fatals with a hint to remove it.

It handles the free-to-update options (project, platform, account, default walltime, default cpus-per-node, default nodefile, scheduler, launcher), the AI provider options (--ai-*), each written per key so one AI field can change without clearing the rest, and the path-root options (--setup-path, --job-path, --resource-path), which relocate only when the current root is empty of its kind. It also handles the environment-manager options (any imported manager's, e.g. the spack plugin's --spack/--spack-packages, via _knit_env_update_all: add a manager when absent, or re-provision on a changed ref only while no environment is built) and --profile (a changed profile is out of scope and fatals), and the tooling options: --ignore-system-sqlite/--ignore-system-jq (rebuild a system-symlinked tool from source) and --knit-cypher-to-sql-version/--knit-cypher-to-sql-url (re-provision knit-cypher-to-sql on a changed version or URL). When no handled option was typed, it reports that there is nothing to update and succeeds.

Parameters:

• raw_args -- [in] Name of an array holding the raw, pre-expansion argument tokens (the caller's copy of _KNIT_INVOCATION_RAW_ARGS).

• ... -- [in] The expanded argument list (defaults injected), as read with knit_get_parameter.

int _knit_bootstrap_update_meta()

Update-mode handler for a "free to update" bootstrap option: one stored as metadata with no constraint (project, platform, account, ...). When the option was typed on this bootstrap call, overwrite its stored value with "metadata store --force" and report that a change was made; when the option was not typed, do nothing and keep the stored value.

Parameters:

• opt -- [in] Option name as typed, without the leading "--" (e.g. "project").

• key -- [in] Metadata key to write (e.g. "__project__").

• value -- [in] Resolved value to store for that option.

• ... -- [in] Raw argument tokens of this invocation (see _KNIT_INVOCATION_RAW_ARGS), used to tell a typed option from a defaulted one.

Returns:

0 when the option was typed and its value written, 1 otherwise.

int _knit_bootstrap_update_profile()

Update-mode handler for --profile. Changing the machine profile is out of scope for the re-runnable bootstrap: a typed profile that differs from the stored one is a hard stop (fatal), while a typed profile equal to the stored one (or an untyped profile) is a no-op. The stored profile is the resolved profile label recorded at first bootstrap (profile).

Parameters:

• profile -- [in] Value typed for --profile.

• ... -- [in] Raw argument tokens of this invocation (see _KNIT_INVOCATION_RAW_ARGS), used to tell a typed option from a defaulted one.

Returns:

1 when there is nothing to do (a differing profile fatals instead).

int _knit_bootstrap_warn_absolute_root()

Warn (non-fatal) when a bootstrap path-root option was given an absolute value. Absolute setup/job roots pin the experiment to this machine's filesystem and hurt reproducibility on another machine or by another user; a relative value resolves against the experiment root and stays portable. A no-op for relative values. Factored out of _knit_bootstrap so it can be unit-tested directly.

Parameters:

• label -- [in] The option name to name in the warning (e.g. "--setup-path").

• value -- [in] The value the user supplied for that option.

int _knit_highlight_if_not_bootstrapped()

Highlight predicate (see knit_highlight_if) for the builtin "bootstrap" command: return 0 ("highlight") while the experiment has not been bootstrapped, non-zero once it has. This bolds "bootstrap" in the root "--help" on a fresh checkout — the one command to run first — and leaves it plain afterwards.

Parameters:

• cmd -- [in] The demangled command name (unused; the predicate is state-only).

int _knit_is_bootstrapped()

Return 0 if the experiment has been bootstrapped (i.e. _KNIT_PREFIX exists), 1 otherwise.

The positive result is cached in _KNIT_IS_BOOTSTRAPPED so that repeated calls within the same session avoid redundant filesystem accesses. The negative result is never cached: the directory may be created at any moment by a bootstrap invocation in the same session.

Variables

String _KNIT_IS_BOOTSTRAPPED

Cache for _knit_is_bootstrapped(). Empty means "not yet checked"; "1" means the positive result has been confirmed and the filesystem need not be re-checked.

String _KNIT_PREFIX

Prefix directory for Knit's local installation.

bundle.sh

Functions

int _knit_bundle()

Body of "knit bundle": pack the experiment into one shippable archive. It reads the --output and --zip options, resolves the experiment root, computes the default output path when none is given, collects the minimal default contents, and drops any path the writer cannot pack. With --dry-run it prints the planned contents (a tree, or a flat list with --list, and sizes with --size) and writes nothing; otherwise it writes the archive. The command is read-only: it declares no table and takes knit_without_provenance, so it records no row and writes no provenance edge.

Parameters:

• ... -- [in] The command invocation arguments.

int _knit_bundle_collect()

Fill a caller-named array with the archive contents, each as a path relative to the experiment root. The default set carries what makes the experiment readable and re-runnable: the experiment script, the knit.sh framework beside it, the pruned .knit directory (the database only), the user-declared required files, each setup's small manifest files, each job's logs and scripts, and the declared artifacts. It leaves out the bulky, regenerable parts: the provisioned tools under .knit, each setup's built environment, and the fetched resources.

The options nameref selects what to include. Each key holds "true"/"false" except include_resources, which holds a comma-separated resource-name list:

• no_knit drop the knit.sh framework file;

• no_db drop .knit/knit.db;

• no_job_logs drop each job's .stdout / .stderr;

• no_job_scripts drop each job's .job.sh / .job.id;

• include_job_content also pack the user content of each job directory;

• no_artifacts drop the artifacts tree;

• include_all_resources pack every fetched resource;

• include_resources pack the named fetched resources.

The experiment script, knit.sh, .knit/knit.db, and the user-declared required files are added unconditionally, so a later prune step can warn about a missing one. Every framework-enumerated path (setup, job, artifact, resource) is added only when it exists on disk (a regular path or a symlink, even a broken one), so an absent optional manifest — a setup that declares no environment has no manifest, say — is skipped silently rather than warned about. Glob expansion of the required entries and the portability validations come in a later milestone.

Parameters:

• __knit_ret -- [out] Name of the array to fill with relative paths.

• opts -- [in] Name of an associative array of include/exclude options.

• root -- [in] The absolute experiment root.

int _knit_bundle_default_output()

Compute the default archive path when the user gives no --output, and store it in the caller-named variable. The name is "./<project>-bundle" plus the format extension. The project name comes from the metadata table (key project); it falls back to the experiment script name without its ".sh" extension when the metadata holds no project. The extension is ".zip" for the zip format and ".tar.gz" otherwise.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the default path.

• fmt -- [in] The archive format ("zip" or "tar").

int _knit_bundle_escapes_root()

Test whether a relative path escapes its root, i.e. a ".." component takes it above the starting directory. Pure string logic (no filesystem): it walks the components, treating "." and empty as no-ops, and reports an escape as soon as the depth would go negative.

Parameters:

• path -- [in] The relative path to test.

Returns:

0 if the path escapes the root, 1 otherwise.

int _knit_bundle_expand_requires()

Fill a caller-named array with the archive-relative paths from a list of required entries (see knit_bundle_requires), validating and expanding each one. Every entry is meant relative to the experiment root. A glob pattern is expanded with pathname expansion relative to the root; every match is added and a pattern that matches nothing draws a warning. A literal entry is added as-is.

The strict flag chooses how a portability problem is handled. In strict mode (the user's own knit_bundle_requires list) an absolute path, a ".." escape, and a missing literal are each a fatal that names the offending path, since the user typed it and can fix it. In lenient mode (paths Knit auto-required on the experiment's behalf) the same problems draw a warning instead: an absolute path is normalized under its basename (its real source recorded in the extern map, so its bytes still travel), and an escape or a missing literal is skipped.

Parameters:

• __knit_ret -- [out] Name of the array to fill with relative paths.

• root -- [in] The absolute experiment root.

• strict -- [in] "true" to fatal on a bad path, "false" to warn and continue.

• ... -- [in] The required entries.

int _knit_bundle_is_glob()

Test whether a path string is a glob pattern, i.e. it holds an unescaped pathname-expansion metacharacter ("*", "?", or "[").

Parameters:

• path -- [in] The path string to test.

Returns:

0 if the string is a glob pattern, 1 otherwise.

int _knit_bundle_print_list()

Print the planned bundle contents as a flat list, one path per line, each relative to the experiment root. This is the --dry-run --list form. When size mode is on, each line is prefixed with the path's size in bytes and a final TOTAL line gives the sum; the size of a symlink is its target's real size (du dereferences with -L), and the size of a directory is its recursive content.

Parameters:

• size_mode -- [in] "true" to annotate each path with its size in bytes.

• root -- [in] The absolute experiment root.

• ... -- [in] The relative paths to print.

int _knit_bundle_print_tree()

Print the planned bundle contents as a tree, like the layout in the design document. This is the default --dry-run form. It builds a parent-to-children map from the flat path list (synthesizing the intermediate directories that no entry names on its own), prints a root header from the archive label, and hands the rendering to _knit_bundle_render_tree. When size mode is on, each real entry is annotated with its size and a final TOTAL line gives the sum.

Parameters:

• size_mode -- [in] "true" to annotate each entry with its size in bytes.

• root -- [in] The absolute experiment root.

• label -- [in] The archive root name shown as the tree header.

• ... -- [in] The relative paths to render.

int _knit_bundle_prune_paths()

Fill a caller-named array with the candidate paths that exist on disk, dropping the ones the archive writer cannot pack. Each candidate is a path relative to the experiment root. A symlink whose target cannot be reached — a dangling link or a symlink loop — is skipped with a warning, so the writer neither packs a link that dangles on the reproducer's machine nor spins on a loop. A path that is simply absent is skipped with a warning too. Every surviving path is kept in the given order.

Parameters:

• __knit_ret -- [out] Name of the array to fill with the surviving paths.

• root -- [in] The absolute experiment root.

• ... -- [in] The candidate relative paths.

int _knit_bundle_relpath()

Store, in the caller-named variable, an absolute path expressed relative to the experiment root. A path under the root has its "<root>/" prefix stripped, so it lands at the same place when the archive is unpacked.

A path outside the root is a portability hazard: a stored root (setup, job, artifact, or resource) that was bootstrapped as an absolute path does not relocate with the archive. When a canonical fallback name is given, the path is packed under that normalized relative name instead: a warning is emitted, the real absolute source is recorded in _KNIT_BUNDLE_EXTERN (so the prune, dry-run, and write steps can find its bytes), and the fallback name is returned. Without a fallback the outside path is returned unchanged.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the relative path.

• root -- [in] The absolute experiment root.

• abs -- [in] The absolute path to make relative.

• canonical -- [in] Optional normalized relative name for an outside path.

int _knit_bundle_render_tree()

Render one directory level of the bundle tree, then recurse into each child directory. It draws the box connectors ("├──", "└──", "│") from the child's position among its siblings. A node is shown with a trailing "/" when it has packed children or is a directory on disk. When size mode is on, a node that is a real bundle entry (named in the collected path list, so a leaf here) is annotated with its size in bytes and that size is added to the running total.

Parameters:

• kidsname -- [in] Name of the parent-to-children map (a value per line).

• realname -- [in] Name of the set of real bundle entries (leaf paths).

• prefix -- [in] The indentation drawn before this level's connectors.

• parentpath -- [in] The path whose children this call renders ("" is root).

• size_mode -- [in] "true" to annotate real entries with their size.

• root -- [in] The absolute experiment root.

• totalname -- [inout] Name of the running byte-total variable.

int _knit_bundle_source()

Store, in the caller-named variable, the real on-disk source of a collected relative path. For an ordinary in-tree path this is "<root>/<rel>". For a path that stands for out-of-tree content (its prefix is a key in _KNIT_BUNDLE_EXTERN) it is the recorded absolute source with the matching prefix swapped back in, so the bytes are found where they really live. Used by prune, the dry-run size report, and the archive writer.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the absolute source path.

• root -- [in] The absolute experiment root.

• rel -- [in] The collected relative path.

int _knit_bundle_warn_unselected_local()

Warn, for each fetched resource instance that is local-only (its type declared knit_with_local) and is not selected for the archive, that it will be left out. A local resource has no remote source, so — unlike a git or url resource — it cannot be re-fetched later; leaving it out should be a deliberate choice, not a silent omission. A resource is selected when --include-all-resources is set or its name is in the --include-resources list. The instance's type is read from its ".<name>.resource.type" sidecar marker, then mapped to the resource type's download method through the per-command _fetch_method marker.

Parameters:

• include_all -- [in] "true" when every resource is selected.

• include_list -- [in] Comma-separated selected resource names (may be empty).

int _knit_bundle_write_archive()

Write the archive from a list of paths, each relative to the experiment root, so the archive unpacks to the same relative tree anywhere. Symlinks are dereferenced: the target content travels in the archive, not the link, so the result is self-contained and relocatable. The tar format uses "-h" for this; the zip format dereferences by default (it stores the referenced file unless "-y" is given). The tar path reads the file list from a temporary file so a long list does not hit the command-line length limit; the zip path changes into the root first, because zip has no "-C" option.

A path may stand for content outside the experiment tree (an absolute stored root, see _KNIT_BUNDLE_EXTERN). Such paths cannot be packed relative to the root, so they are staged first: a symlink at the path's normalized relative name points to the real source, and the archive tool dereferences it into place. The in-tree paths pack straight from the root; only the outside paths need staging.

Parameters:

• fmt -- [in] The archive format ("zip" or "tar").

• output -- [in] The archive path to write.

• root -- [in] The absolute experiment root.

• ... -- [in] The relative paths to pack.

Returns:

Fatal when the archive tool is missing, no path is given, or the write fails; otherwise 0.

Variables

Array _KNIT_BUNDLE_AUTO_REQUIRES

Extra files Knit itself knows the experiment needs and adds without the user asking — the manifest a setup names through a knit_with_<nm>_env directive (for example knit_with_spack_env) in its file form. Each entry is a path string recorded verbatim, in declaration order. Unlike _KNIT_BUNDLE_REQUIRES (the user's own list, validated strictly), these are handled leniently at bundle time: a path outside the tree draws a warning and is normalized rather than a fatal, since the user did not type it.

AssociativeArray _KNIT_BUNDLE_EXTERN

Maps a normalized in-archive relative path to the absolute on-disk source it stands for, for content that lives outside the experiment tree: an absolute stored root (a setup/job/artifact/resource root bootstrapped as an absolute path) or an auto-required file outside the tree. The key is where the content lands in the archive; the value is where its bytes really are. Collection fills it; the prune, dry-run, and write steps consult it (by prefix) to find the real source of a collected path. Reset at the start of every "knit bundle".

Array _KNIT_BUNDLE_REQUIRES

Extra files the experiment needs but Knit does not otherwise track (a config file, a small input, a post-processing script). Each entry is a path string recorded verbatim by knit_bundle_requires, in declaration order. The paths are meant to be relative to the experiment script's directory; validation and glob expansion happen later, when "knit bundle" runs, not at declaration time.

cli.sh

Functions

int _knit_arg_name()

Extract the normalized parameter name from a command-line token. The token may be given as "--name" or "--name=value"; the leading "--" and any "=value" suffix are stripped and hyphens are converted to underscores. This is the single place where the "=value" form is recognized, so registered commands and plain helper functions (via knit_get_parameter / knit_check_arguments) accept it consistently.

The caller is responsible for only passing tokens that start with "--"; bare values must not be passed, otherwise they may be mistaken for parameter names.

Parameters:

• token -- [in] A raw argument token starting with "--".

int _knit_arg_was_provided()

Report whether a named option appears among a command invocation's raw arguments. Used to tell an option the user typed apart from one that took its default: the CLI engine splices optional defaults and flag values into the argument list before a command body runs, so the body cannot see the raw command line except through the tokens _knit_invoke_command publishes in _KNIT_INVOCATION_RAW_ARGS.

The bare-flag form ("--name"), the separate-value form ("--name value"), and the joined form ("--name=value") are all recognized: only tokens starting with "--" are compared, by normalized name, so a following value token is never mistaken for the option. Hyphens and underscores in the option name are interchangeable. Scanning stops at a "--" token, so an option name that appears only after "--" (as an extra) does not count as provided.

Parameters:

• name -- [in] Option name, with or without the leading "--".

• ... -- [in] Raw argument tokens to scan (typically "${_KNIT_INVOCATION_RAW_ARGS[@]}").

int _knit_build_constraint_json()

Build a jq JSON object from an expanded argument list, using type metadata to emit integers/reals/booleans as JSON native types and all other values as strings. Each parameter may be given as "--name value" or "--name=value" (flags already converted to "true"/"false" by _knit_expand_command_arguments).

Parameters:

• cmd -- [in] Mangled command name (used for type lookups).

• ... -- [in] Expanded argument list.

int _knit_check_argument_type()

Validate that the value provided for a parameter conforms to the parameter's declared type. On mismatch the script stops with a fatal error; for enum types the message lists the accepted values. Parameters with no recorded type (e.g. framework-internal ones) are left unchecked.

Parameters:

• cmd -- [in] Command the parameter belongs to (mangled).

• demangled_cmd -- [in] Human-readable command name (used in messages).

• name -- [in] Parameter name (normalized).

• value -- [in] Value provided for the parameter.

int _knit_check_command_arguments()

Check that the arguments expected by the command are provided. This function will fail with a fatal error (i.e. the script will stop) if a required argument is not provided, if an argument provided does not match any expected, or if a value does not conform to its parameter's declared type.

Parameters:

• cmd -- [in] Name of the command (mangled).

• ... -- [in] Arguments to pass to the command.

int _knit_check_constraints()

Evaluate all --when constraints declared for a command against the provided arguments. For each constrained parameter:

• If the condition evaluates to true and the parameter is required but absent from the original (user-provided) arguments, a fatal error is raised.

• If the condition evaluates to false and the parameter is present in the original arguments, a fatal error is raised. Returns 0 immediately when the experiment is not bootstrapped (jq unavailable).

Parameters:

• cmd -- [in] Mangled command name.

• orig_ref -- [in] Name of bash array holding the original (pre-expansion) args.

• exp_ref -- [in] Name of bash array holding the expanded args.

int _knit_checksum_inputs()

For a command about to run, verify existence of every file/directory "input" parameter and, unless it opted out with --no-checksum, stash its digest before the timed body starts. The value comes from the expanded invocation arguments. A required input, or an optional input given a non-empty value, that does not exist is fatal; an optional input with no value is skipped (no existence check, no checksum). Existence is enforced for every file/directory input; hashing is skipped for a --no-checksum one. Hashing happens here, before the body, so a body that overwrites its own input cannot corrupt the recorded input digest and the hash time is excluded from the run.

In a launched app worker the inputs were already verified and hashed once by the run dispatcher before launch; no rank hashes. Rank 0 then recovers the forwarded digests from the environment (other ranks suppress recording), so this returns after that instead of touching the filesystem.

Parameters:

• cmd -- [in] Mangled command name.

• ... -- [in] The expanded invocation arguments.

int _knit_checksum_is_app_worker()

Report whether the command about to run is an app executing as a launched worker rank, rather than an ordinary single-process command. An app's inputs are verified and hashed once by the run dispatcher on the login side, before any rank is spawned; the ranks must not touch the filesystem for checksums. The context is an app (the "app" command type) re-entered under a live run (KNIT_RUN_ID, exported by the dispatcher into the launcher environment).

Parameters:

• cmd -- [in] Mangled command name.

int _knit_checksum_outputs()

Verify existence of every file/directory "output" and, unless it opted out with --no-checksum, stash its digest for the row write. The output value (the path) is read from the in-memory output store set by knit_output, else the output's declared default; an output left with no value is skipped (no existence check, no checksum). Hashing is skipped for a --no-checksum one. Called after the run duration has been captured, so hashing is excluded from the measured run time.

When tolerant is "false" (a successful body) existence is enforced: a declared output whose path does not exist is fatal (a broken postcondition). When tolerant is "true" (a FAILED body) a missing output is skipped instead — a failed body may legitimately have produced only some of its outputs — while an output that IS present is still hashed and recorded, so a failure's partial results are captured.

In a launched app worker no rank hashes outputs: rank 0 records the output paths with empty checksum columns, and the run dispatcher verifies existence and hashes them once after the launcher returns, off every measured duration. So this returns early there, leaving the checksum columns for the dispatcher.

Parameters:

• cmd -- [in] Mangled command name.

• tolerant -- [in] "true" to skip (not fatal on) a missing output; default "false" enforces existence.

int _knit_checksum_require_exists()

Verify that a checksummed file/directory value refers to an existing target, fataling with a direction-specific message if it does not. A "file" must be a regular file (-f), a "directory" must be a directory (-d). An input that does not exist is a broken precondition; an output missing on a successful completion is a broken postcondition — both are fatal, so a run never records a checksum for an artifact that was not there.

Parameters:

• demangled_cmd -- [in] Human-readable command name (for the message).

• direction -- [in] "input" or "output" (shapes the error message).

• name -- [in] Normalized parameter/output name (for the message).

• kind -- [in] "file" or "directory".

• value -- [in] The path to check.

int _knit_checksum_stash()

Record a computed digest for the currently-recording command by setting the companion "<param>_checksum" output value in the command's in-memory output store, so the normal row-recording machinery writes it. The digest is stored algorithm-prefixed ("sha256:<hex>"). This is the source-agnostic seam every path feeds: the non-app input hook, the non-app output hook, and (later) the app dispatcher all land a digest here for the same column writer.

Parameters:

• cmd -- [in] Mangled command name.

• param -- [in] Normalized parameter/output name whose companion column to set.

• hex -- [in] The bare 64-hex digest (no algorithm prefix).

int _knit_checksum_stash_from_env()

Recover forwarded input digests for a launched app worker. The run dispatcher hashes each checksummed input once and forwards the bare digest to every rank through the launcher environment (KNIT_CHECKSUM_<param>). Rank 0 stashes those digests into its output store so its row records the same value the dispatcher computed, without any rank hashing. Other ranks suppress recording, so the stash is harmless there. An input with no forwarded digest (an absent optional, or one that opted out with --no-checksum) is skipped.

Parameters:

• cmd -- [in] Mangled command name.

int _knit_checksum_target_exists()

Test whether a checksummed file/directory value refers to an existing target: a "file" must be a regular file (-f), a "directory" must be a directory (-d). Returns success/failure without fataling, so a tolerant caller (an output hook after a FAILED body, which may legitimately have produced only some outputs) can skip a missing target rather than abort. _knit_checksum_require_exists is the fatal wrapper for the strict callers (inputs, and outputs of a success).

Parameters:

• kind -- [in] "file" or "directory".

• value -- [in] The path to test.

Returns:

0 if the target exists and matches the kind, 1 otherwise.

int _knit_command_check_usable()

Evaluate the usability predicates registered for a command (via knit_usable_if) in declaration order. On the first predicate that returns non-zero, set the reason output (nameref) to that predicate's parallel description and return 1. Return 0 if all predicates pass, or if the command declared none.

Each predicate is called as "<predicate> <demangled-cmd>" in the current shell (no subshell fork), so only its exit status is used. A predicate whose function does not exist is fatal: a usability guard that silently vanished would let an unusable command run.

Parameters:

• __knit_ret -- [out] Name of the variable to receive the failure reason (nameref).

• cmd -- [in] Command (mangled name) to check.

Returns:

0 if usable, 1 if a predicate failed (with the reason set).

int _knit_command_demangle()

Demangles a command, i.e. converts "command__1__subcommand__1__subcommand" back into "command:subcommand:subcommand".

Parameters:

• cmd -- [in] Command to demangle.

int _knit_command_display()

Rebuilds the human-readable name of a command from the registered spelling of each of its segments. Unlike _knit_command_demangle, which returns the canonical (underscore) identity, this restores the spelling the user wrote at registration (which may contain a hyphen), the same way a parameter keeps its registered spelling in "--help".

The mangled name is split into its "__1__"-separated prefixes; each prefix's stored display basename (KNIT_CMD<prefix>_display) is looked up and the basenames are joined with ":". A prefix with no stored basename (which should not happen, since a parent is always registered before its child) falls back to its canonical segment.

The result is returned through a caller-named variable (nameref) so the registration and "--help" paths pay no subshell fork.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the display name.

• cmd -- [in] Command to render (mangled name).

int _knit_command_get_last()

Takes a command in the form "aaa:bbb:ccc" or "aaa bbb ccc" or "aaa__1__bbb__1__cccc" and return the last command (e.g. "ccc" in all the cases above).

Parameters:

• __knit_ret -- [out] Name of the variable to hold the last command.

• cmd -- [in] Command name (colon/space/mangled).

int _knit_command_get_parents()

Takes a command in the form "aaa:bbb:ccc" or "aaa bbb ccc" or "aaa__1__bbb__1__cccc" and return the parent commands (e.g. "aaa:bbb" or "aaa bbb" or "aaa__1__bbb".

Parameters:

• __knit_ret -- [out] Name of the variable to hold the parent commands.

• cmd -- [in] Command name (colon/space/mangled).

int _knit_command_hidden()

The "--help" visibility test for a command. Return 0 (hidden) if the static _is_hidden boolean is true, or if any of the command's dynamic _hidden_pred predicates returns 0; return non-zero (shown) otherwise. Because knit_hidden and knit_hidden_if are mutually exclusive, at most one of the two arms is ever non-trivial for a given command.

Each dynamic predicate is called as "<predicate> <demangled-cmd>" in the current shell (no fork). A predicate whose function does not exist is a warning (not fatal): hiding is guidance, not access control, so a vanished predicate is treated as "no" (do not hide) and "--help" still renders.

Parameters:

• cmd -- [in] Command (mangled name) to test.

Returns:

0 if the command should be hidden from "--help", non-zero otherwise.

int _knit_command_highlighted()

The "--help" highlight test for a command. Return 0 (highlight) if any of the command's dynamic _highlight_pred predicates returns 0; return non-zero (plain) otherwise, including when the command declares no highlight predicates.

Each predicate is called as "<predicate> <demangled-cmd>" in the current shell (no fork). A predicate whose function does not exist is a warning (not fatal): highlighting is cosmetic, so a vanished predicate is treated as "no" (do not highlight) and "--help" still renders.

Parameters:

• cmd -- [in] Command (mangled name) to test.

Returns:

0 if the command name should be highlighted, non-zero otherwise.

int _knit_command_is_builtin()

Test whether a command is a framework builtin, i.e. it was marked with _knit_is_builtin during its registration.

Parameters:

• cmd -- [in] Command (mangled name) to test.

Returns:

0 if the command is a builtin, 1 otherwise.

int _knit_command_is_usable_before_bootstrap()

Test whether a command is usable before bootstrap, i.e. it was marked with knit_usable_before_bootstrap during its registration.

Parameters:

• cmd -- [in] Command (mangled name) to test.

Returns:

0 if the command is usable before bootstrap, 1 otherwise.

int _knit_command_is_wrapper()

Test whether a command is a wrapper, i.e. it was registered with knit_register_wrapper. A wrapper forwards its arguments verbatim to the underlying command and declares no parameters or outputs. Reads the command kind from the KNIT_CMD<cmd>_type field (see knit_register), which records the mutually-exclusive kind: "command" (plain, the default), "wrapper", "setup", "job", "app", or "resource".

Parameters:

• cmd -- [in] Command (mangled name) to test.

Returns:

0 if the command is a wrapper, 1 otherwise.

int _knit_command_mangle()

Mangles a command, i.e. converts "command:subcommand:subcommand" into "command__1__subcommand__1__subcommand" so the name can be used in variable names. Also converts spaces into 1.

Hyphens are folded to underscores first, so a hyphen and an underscore in a command name name the same command (the canonical, Bash-safe identity uses the underscore), the same way hyphens and underscores are interchangeable in parameter names.

The result is returned through a caller-named variable (nameref) so the registration hot path pays no subshell fork. The body is pure Bash (no sed), which is far cheaper: this runs once per command and parameter at load time.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the mangled command.

• cmd -- [in] Command to mangle (remaining arguments are joined with spaces).

int _knit_command_marker_var()

Resolve the global variable name backing a per-command marker, storing it in the caller-named variable. Markers live in their own KNIT_CMD<cmd>marker* namespace so a plugin cannot collide with core per-command state. The marker name is normalized (hyphens to underscores) and validated; a name that is not a valid identifier is fatal.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the resolved global name.

• name -- [in] The marker name.

int _knit_command_with_space()

Prints a mangled command (or a command with ":" in it) with spaces between subcommands.

Parameters:

• cmd -- [in] Command to print with spaces.

int _knit_decl_flag_present()

Return 0 if a bare declaration flag (e.g. "--no-checksum") appears among the remaining declaration arguments, 1 otherwise. Used to parse boolean flags such as --no-checksum that carry no value, so knit_get_parameter (which would read the following token as a value) is not suitable. Hyphens and underscores in the flag name are interchangeable, and anything from "--" onwards is ignored.

Parameters:

• flag -- [in] Flag name without the leading "--".

• ... -- [in] Declaration arguments to scan.

int _knit_declare_domain()

Record the search domain declared for a parameter by the "--domain <descriptor>" flag (and its companion "--log" scale hint) on knit_with_required / knit_with_optional. The raw descriptor token and the log flag are stored verbatim under the parameter's own namespace (${ns}_2_<param>_domain and _domain_log), beside the description/type/default fields. The descriptor is only parsed and validated against the parameter type later, on demand, by the parameter-space accessors — so registration does not pay to parse every parameter's domain. A cheap shape check rejects a gross typo here; a parameter with no "--domain" stores nothing, and "--log" with no "--domain" is a fatal.

Parameters:

• ns -- [in] Parameter namespace prefix (e.g. "_KNIT_CMD_<cmd>").

• param -- [in] Normalized parameter name.

• log -- [in] "true" when the --log scale hint was given, otherwise "false".

• domain -- [in] The descriptor token, or empty when --domain was not given.

int _knit_discover_ancestors()

Ensure lazy subcommand discovery has run for every ancestor of a command, so the command resolves even when it arrives as a single pre-mangled (or colon) token rather than as separate tokens. The per-token resolver walk (_knit_invoke_command) already discovers along a separate-token path; this is the fallback for an internal caller that passes a whole command name in one argument, e.g. _knit_invoke_command "job__1__show__1__stdout".

Walks the mangled prefix chain top-down (job -> job:show -> ...), running each existing ancestor's discovery function; each step may register the next level. A no-op for a top-level command (no ancestors) and for ancestors without a discovery function.

Parameters:

• cmd -- [in] Command (mangled name) whose ancestors to discover.

int _knit_ensure_discovered()

Run a command's subcommand discovery function (see knit_with_subcommand_discovery) if it has one and it has not run yet. A no-op for a command without a discovery function or whose function has already run, so every trigger (the resolver, "--help", "describe") may call it freely before reading the command's subcommands.

The "_discovered" guard is set to "true" BEFORE the function is called, so a discovery function that (directly or through a callback) causes the same command to be resolved again does not run the function a second time.

The function name is checked with "declare -F" when it is called (not at declaration time), because the function may be defined after the decorator. It is called in the current shell (never a subshell) with the command's display name as its first argument.

Parameters:

• cmd -- [in] Command (mangled name) whose discovery to ensure.

int _knit_execute_after_commands()

Evaluate the callbacks installed after a command. The callbacks are called with the calling command name (demangled) as context, as well as the list of parameters passed to the command.

Parameters:

• cmd -- [in] Command (mangled name) for which to execute the after callbacks.

• ... -- [in] Arguments of the command.

int _knit_execute_before_commands()

Evaluate the callbacks installed before a command. The callbacks are called with the calling command name (demangled) as context, as well as the list of parameters passed to the command.

A before-callback expresses a precondition for running the command (asserting an invocation context, activating a setup, building a Spack environment). If one fails (returns non-zero), the remaining callbacks are skipped and that status is returned so the caller (_knit_invoke_command) can abort the command rather than run its body against an unmet precondition.

Parameters:

• cmd -- [in] Command (mangled name) for which to execute the before callbacks.

• ... -- [in] Arguments of the command.

Returns:

0 if all callbacks succeeded, the first non-zero status otherwise.

int _knit_expand_command_arguments()

Adds optional arguments that are not provided in the arguments, and converts flags into --flag true or --flag false.

Parameters:

• name -- [in] Name of the command.

• ... -- [in] Arguments to pass to the command.

int _knit_find_flag()

This function takes a flag and checks if it appears in the remaining list of arguments, returning 0 if it does, 1 otherwise.

Example:

_knit_find_option "--help" aaa bbb ccc --help ddd

will return 0 because "--help" was found.

Parameters:

• flag -- [in] Flag to find.

• ... -- [in] List of arguments to search from.

Returns:

0 if the flag was found, 1 otherwise.

int _knit_format_option_alternatives()

Format the display spelling(s) of an option for a message. Options accept both hyphen and underscore spellings; this returns "--foo or --foo-bar" when the two spellings differ, and just "--foo" when they are identical (so a message about an option with no underscores does not read "--foo or --foo").

Parameters:

• __knit_ret -- [out] Name of the variable to hold the result.

• option -- [in] Parameter name (as declared).

int _knit_help_render_entry()

Render a single "--help" entry (an option row or a subcommand row) with an optional word-wrapped, hanging-indented description:

<head><first words of the description> <indent spaces><more words> ...

where <head> is the fully-formatted leading column (indentation, the padded option/subcommand name, and — for options — the "[annotation] " prefix) and continuation lines are indented to <indent> columns so the wrapped text forms a clean hanging indent under the description column.

Wrapping is applied only when a usable terminal width is known and the continuation column is wide enough; otherwise the entry falls back to a single line ("<head><description>"), byte-for-byte identical to the pre-wrapping output, so piped or redirected help stays unchanged.

Parameters:

• width -- [in] Terminal width in columns, or 0 for "no wrapping".

• head -- [in] Literal leading text printed before the description on the first line (may contain ANSI escape sequences).

• head_len -- [in] Display width of <head> (escape bytes excluded), i.e. the column at which the first-line description begins.

• indent -- [in] Number of spaces used to indent continuation lines.

• description -- [in] Description text, wrapped at word boundaries.

int _knit_hidden_if_not_usable_pred()

Internal hide predicate backing knit_hidden_if_not_usable. Receives the demangled command name, runs the command's usability check, and inverts it: returns 0 ("hide") when the command is not usable, and non-zero ("show") when it is usable (or declares no usability predicates).

Parameters:

• demangled -- [in] Demangled command name passed by _knit_command_hidden.

Returns:

0 if the command is not usable (hide it), non-zero otherwise.

int _knit_invoke_command()

Invoke a command.

Example:

_knit_invoke_command "say" "hello" "--name" "Matthieu"

Will invoke the command "say:hello" with arguments "--name" and "Matthieu".

Parameters:

• ...commands -- [in] Commands and subcommands.

• ...args -- [in] Arguments for the command.

int _knit_is_builtin()

Mark the item currently being defined as a framework builtin. This is called by knit's own source files immediately after a registration or enum definition, so that "describe" (and --exclude-builtins) can tell knit's own commands/enums apart from user-declared ones. It has two behaviors:

• Inside a command registration (KNIT_CURRENT_COMMAND set, i.e. between knit_register/knit_register_wrapper/... and knit_done), it marks the current command by setting _KNIT_CMD<cmd>_is_builtin=true.

• Otherwise it marks the most recently defined enum (_KNIT_LAST_ENUM) by adding it to the _KNIT_BUILTIN_ENUMS set.

int _knit_name_is_valid()

Checks that a parameter or command name is valid, i.e. it has to start with a letter, followed by any number of alphanumerical characters and hyphens and underscores. The names "true", "false", "null", "and", "or", and "not" are reserved for use in --when constraint expressions and are not allowed.

Parameters:

• param -- [in] Parameter name to normalize.

int _knit_name_normalize()

Normalizes a parameter or command name, i.e. converts its hyphens into underscores.

The result is returned through a caller-named variable (nameref) so the registration hot path (this runs once per parameter and command) pays no subshell fork. A plain temp holds the delegated result rather than passing the output nameref straight through, which would risk a circular name reference.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the normalized name.

• name -- [in] Name to normalize.

int _knit_output_default()

This function returns the default value of an output for a given command.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the default value.

• cmd -- [in] Command to which the output belongs (must be mangled).

• output -- [in] Name of the output (must be normalized).

int _knit_output_description()

This function returns the description of an output for a given command.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the description.

• cmd -- [in] Command to which the output belongs (must be mangled).

• output -- [in] Name of the output (must be normalized).

int _knit_output_type()

This function returns the type of an output for a given command.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the output type.

• cmd -- [in] Command to which the output belongs (must be mangled).

• output -- [in] Name of the output (must be normalized).

int _knit_param_check_declaration()

This function carries out all the checks for a parameter to be declared by knit_with_required/optional/flag. The parameter name must include a type annotation in the form "name:type" (e.g. "width:integer").

Parameters:

• suffix -- [in] Suffix ("required", "optional", or "flag") to use for variables.

• param -- [in] Parameter name followed by ":type".

• description -- [in] Description of the parameter.

int _knit_param_default()

This function returns the default value of a parameter for a given command.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the default value.

• cmd -- [in] Command to which the parameter belongs (must be mangled).

• param -- [in] Name of the parameter (must be normalized).

int _knit_param_description()

This function returns the description of a parameter for a given command.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the description.

• cmd -- [in] Command to which the parameter belongs (must be mangled).

• param -- [in] Name of the parameter (must be normalized).

int _knit_param_is_norecord()

Test whether a parameter or flag was declared with "--no-record": parsed, validated, and shown in "--help" as an ordinary option, but kept out of the command's table (no column) and out of every recorded row. This lets a command carry control options the user sets at the call site without widening its table schema. The marker lives in the parameter's own namespace (KNIT_CMD<cmd>2_norecord), set by knit_with_required / knit_with_optional / knit_with_flag.

Parameters:

• cmd -- [in] Mangled command name (as used in KNIT_CMD* variables).

• param -- [in] Normalized (underscored) parameter name.

Returns:

0 if the parameter is marked no-record, 1 otherwise.

int _knit_param_type()

This function returns the type of a parameter for a given command.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the type.

• cmd -- [in] Command to which the parameter belongs (must be mangled).

• param -- [in] Name of the parameter (must be normalized).

int _knit_print_command_usage()

Print the help message for a command/subcommand.

Parameters:

• ...cmds -- [in] Command and subcommand names

int _knit_print_options_block()

Print the "Options"-style listing for a single command: a titled section with an hrule, then (optionally) the "--help" entry, then the command's required parameters, optional parameters, and flags, each column-aligned. Extracted so it can be printed both for a command's own options and, for a subcommand invoked through a dispatcher, for the dispatcher's options as well.

Parameters:

• cmd -- [in] Mangled command name whose options to print.

• title -- [in] Section title (e.g. "Options" or "submit options").

• with_help -- [in] "true" to include the "--help" entry, "false" otherwise.

int _knit_provenance_enabled()

Decide whether an invocation of a command participates in the provenance graph: a participating command records a "call" edge and acts as an in-process source frame for the commands it invokes; a non-participating one is transparent (no edge, and skipped when a callee resolves its edge source — see _knit_resolve_source_context).

The effective setting is resolved in this order (innermost/most-specific wins):

1. the command's own explicit mark (knit_with_provenance -> "with", knit_without_provenance -> "without"), if any;

2. otherwise the nearest marked lexical ancestor's mark, walking the colon-nested command name up via _knit_command_get_parents (e.g. "a:b:c" -> "a:b" -> "a"), so one mark on a parent governs a whole subtree;

3. otherwise the default by visibility: hidden commands (marked via knit_hidden, e.g. the "_run" worker) are transparent so internal plumbing never shows up in the graph; every other (visible) command participates.

Inheritance is over the lexical command hierarchy (the names), not the runtime call stack. Data-row recording (knit_with_table) is orthogonal to this.

Returns 0 (success) when the command participates, 1 otherwise.

Parameters:

• cmd -- [in] Mangled command name.

int _knit_pset_filter_build()

Build the set of normalized parameter names named by a filter list, failing if any name is not in the source parameter set. Prevents a typo from silently importing the whole set. Names are normalized (hyphens and underscores are interchangeable).

Parameters:

• __knit_ret -- [out] Associative array to populate with the normalized names.

• pset_ns -- [in] Namespace prefix of the source set (e.g. "_KNIT_PSET_foo").

• set_name -- [in] Original set name, for error messages.

• mode -- [in] Filter mode ("exclude" or "only"), for error messages.

• list -- [in] Comma-separated list of parameter names.

int _knit_pset_import_skip()

Decide whether a parameter should be skipped while importing a parameter set, given the active filter. Returns 0 (skip) when the mode is "exclude" and the parameter is in the filter, or when the mode is "only" and the parameter is NOT in the filter. Returns 1 (import) otherwise, including when no filter is active.

Parameters:

• mode -- [in] Filter mode ("exclude", "only", or "" for no filter).

• filter_name -- [in] Associative array holding the filter's normalized names.

• param -- [in] Normalized parameter name under consideration.

Returns:

0 to skip the parameter, 1 to import it.

int _knit_record_invocation()

Record the just-completed invocation of a command: its data row (when the command declared a table with knit_with_table) and, when the command participates in the provenance graph (see _knit_provenance_enabled), a "call" edge from its source (the caller). Both are written only when the experiment is bootstrapped.

The row id is the id resolved when this frame was pushed (see _knit_resolve_row_id) and read back from the top of _KNIT_EXECUTING_ROW_ID, so it matches the id a nested callee already saw as its source. Recording therefore runs while the frame is still on the stacks (before it is popped).

The four cases:

• transparent (hidden) command with no table: nothing to record;

• transparent command with a table: the data row only (excluded from the graph — provenance participation and data-row recording are orthogonal);

• participating command with a table: the data row and the "call" edge in one transaction (_knit_db_record_invocation);

• participating command with no table: the "call" edge on its own (_knit_prov_record_edge), whose target id joins to no data row (dangling).

The edge's start_time was captured when the frame was pushed (top of _KNIT_EXECUTING_START_TIME); its end_time is captured here, after the body and after-callbacks.

Parameters:

• cmd -- [in] Mangled command name.

• ... -- [in] The expanded invocation arguments.

int _knit_record_row_now()

Record the current invocation's database row immediately, instead of waiting for the automatic post-invocation recording. Use this when a command must persist its row before doing blocking work whose side effects update that same row — e.g. knit submit records the submission before a --wait dispatch, so the job's own state transitions (running/completed) land on an existing row. Recording is idempotent: the automatic recording afterwards sees this one and does not insert a duplicate. Must be called from within an executing command.

Parameters:

• ... -- [in] The invocation arguments (params/flags to record).

int _knit_register_checksum()

Wire up existence checking and content checksums for a "file"/"directory" parameter or output that was just declared for the command being registered.

For every checksummable declaration (whether or not --no-checksum was given) it records a per-parameter marker — a KNIT_CMD<cmd>fileparams set plus _KNIT_CMD<cmd>fileparam holding "<direction>:<kind>:<checksum>", e.g. "input:file:yes" or "output:directory:no" — that the runtime reads to enforce direction-aware existence and to know whether and how to hash. Existence is a property of the type, so it is enforced even under --no-checksum; only the digest is opted out.

Unless --no-checksum was given it additionally registers an implicit recorded output column "<name>-checksum" (DB "<name>_checksum") of type string, so the digest is written by the normal row-recording machinery and appears in the command's table.

Passing --no-checksum on a non-checksummable type is a fatal declaration error (there is nothing to checksum). A synthesized companion name that collides with an already-declared parameter, flag, or output is a fatal declaration error so the companion can never overwrite a user-declared column.

Only meaningful in a command context; wiring is skipped in a parameter set (which has no outputs and no runtime existence hooks). The parameter must already have been added to its declaration set before this is called, so the collision check sees it.

Parameters:

• direction -- [in] "input" (parameter) or "output".

• type -- [in] The declared type (name or alias) of the parameter/output.

• name -- [in] The declared (un-normalized) parameter/output name.

• no_checksum -- [in] "true" if --no-checksum was given, "false" otherwise.

int _knit_register_fileparam()

Record the per-parameter existence/checksum marker for a "file"/"directory" declaration of the command being registered: a KNIT_CMD<cmd>fileparams set plus _KNIT_CMD<cmd>fileparam holding "<direction>:<kind>:<checksum>" (e.g. "input:file:yes" or "output:directory:no"). The runtime reads it to enforce direction-aware existence and to know whether and how to hash. The type alias is resolved to its canonical kind before storing.

This is the marker half of a file/directory declaration. _knit_register_checksum calls it and then also adds a companion "<name>-checksum" column; knit_with_output_artifact calls it on its own, because an artifact records its digest in the artifacts table rather than in a column of the command's own table.

Parameters:

• direction -- [in] "input" (parameter) or "output".

• type -- [in] The declared type (name or alias).

• name -- [in] The declared (un-normalized) parameter/output name.

• checksum -- [in] "yes" to also hash at runtime, "no" to only check existence.

int _knit_register_result()

Mark an output of the command being registered as a result: something that constitutes what the experiment was for, as opposed to an incidental output. The mark is importance only; it carries no runtime behavior beyond how the command is described. It is valid on an output of any type.

The output is added to a per-command KNIT_CMD<cmd>_results set, created as associative on first use (knit_register does not create it, since not every command has a result).

Only meaningful in a command context; a call in a parameter set is a no-op. The output must already have been added to the outputs set before this is called.

Parameters:

• name -- [in] The declared (un-normalized) output name.

int _knit_reserve_name()

Reserve a declared name in a command's (or parameter set's) name space, failing if the name is already taken. Parameters, outputs, artifacts, and synthesized checksum columns all share one name space per command: a parameter and an output map to the same table column, an artifact is referred to by name at runtime, and every declared name must be unambiguous. This is the single place that enforces that, so a new kind of declaration only has to call it rather than test every other declaration set by hand.

The name space lives in the associative array <ns>_names, mapping a normalized name to the lowercased kind that claimed it ("parameter", "output", "artifact", or "checksum column"). The map is created together with the command / parameter set, so it always exists here. On a clash the message distinguishes a same-kind redeclaration ("already declared") from a cross-kind collision ("collides with a

declared <kind>").

Parameters:

• ns -- [in] Namespace prefix ("_KNIT_CMD_<cmd>" or "_KNIT_PSET_<name>").

• context_name -- [in] Human-readable command / parameter set name for messages.

• kind -- [in] Capitalized kind of the incoming name ("Parameter", "Output", "Artifact", or "Checksum column").

• display -- [in] Un-normalized name as written, for messages.

• normalized -- [in] Normalized name to reserve.

int _knit_resolve_command_tokens()

Walk the leading command-name tokens of an argument list and resolve them to a mangled command name, running lazy subcommand discovery along the way (an idempotent side effect). It is greedy: it consumes non-"--" tokens one level at a time, registering each level's subcommands as it descends, and stops at the first "--"/"--option" token or once the accumulated name is a wrapper (a wrapper forwards everything after its name verbatim, so it resolves to itself). A single pre-mangled/colon token that the per-token walk did not split (e.g. an internal "job__1__cancel") is handled by an ancestor-discovery fallback.

This is the shared resolver behind both the command invoker (_knit_invoke_command) and the public dispatcher API (knit_resolve_subcommand and its companions), so all of them resolve and discover identically.

Parameters:

• __knit_ret_cmd -- [out] Nameref receiving the mangled command name (set even when the command is unknown, so a caller can message it).

• __knit_ret_consumed -- [out] Nameref receiving the count of leading tokens consumed as the command name.

• ... -- [in] The argument list (command tokens then options).

Returns:

0 when the resolved command exists, 1 when it does not.

int _knit_resolve_default()

Resolve a declared default value into the value actually used when an optional parameter is not provided. A default written as "ENV[NAME]" means "fall back to

the value of the NAME environment variable"; it resolves to that variable's current value (the empty string when the variable is unset). Any other string, including one that merely looks like ENV[...] but does not name a valid shell variable, is returned unchanged, so ordinary defaults keep their literal value.

Parameters:

• raw -- [in] Raw default value as declared with knit_with_optional.

int _knit_resolve_row_id()

Resolve the row id for an invocation of a command, used when its frame is pushed onto KNIT_EXECUTING_ROW_ID. The precedence is: an explicit id already set via knit_set_row_id (_KNIT_CMD<cmd>_row_id), otherwise a fresh uuidv7.

Every invocation gets its own distinct id: the historical couplings that made a job body reuse its submission's UUID (KNIT_JOB_PREFIX) and an app's rank-0 row reuse its run's UUID (KNIT_RUN_ID) are gone — the provenance edge, not a shared id, links a child back to its parent.

Parameters:

• cmd -- [in] Mangled command name.

int _knit_resolve_source_context()

Resolve the source of the currently-recording invocation's "call" edge (the caller), writing the source's row id and demangled command name into the two named output variables. The recording invocation is the edge's target. The precedence (innermost wins) is:

1. In-process caller frame — the nearest participating frame below the top of _KNIT_EXECUTING_COMMAND (transparent frames, per _knit_provenance_enabled, are skipped). Covers in-process nesting and setup dispatch (boundaries B1/B4). Its id comes from the parallel _KNIT_EXECUTING_ROW_ID stack.

2. Exported env — KNIT_SOURCE_ID / KNIT_SOURCE_COMMAND, set by a caller across a process boundary (a job's batch script, a run's launcher subshell; boundaries B2/B3, wired in later milestones) and read by the first command in the re-entered process, which has no in-process caller.

3. Root — neither available: both outputs are empty.

The current frame is the top of the stacks (it is recorded before being popped), so the in-process search starts one below the top.

Parameters:

• out_id -- [out] Name of the variable to receive the source row id.

• out_name -- [out] Name of the variable to receive the source command name.

int _knit_usable_before_bootstrap_validate()

knit_done callback registered by knit_usable_before_bootstrap. Enforces the three rules a command usable before bootstrap must satisfy. Each rule guards a behavior that would otherwise degrade silently (not crash) before bootstrap, so the point is to guarantee correct pre-bootstrap behavior, not to avoid a crash:

1. No database table: table row recording is skipped before bootstrap, so a usable command declaring a table would silently record nothing.

2. No "--when" constraint on any parameter: constraint evaluation (via jq) is skipped before bootstrap, so a usable command's constraint would be silently ignored.

3. Parent must also be usable before bootstrap: keeps the usable set a connected subtree rooted at the top level, which is what makes the "--help" filtering and runtime guard correct without extra reachability logic.

Any violation is fatal, naming the command and the specific reason.

Parameters:

• cmd -- [in] Command (mangled name) being validated.

int _knit_without_exit_status()

Mark the command being registered so its table does not get the reserved "__exit_status__" column. Used for a table whose status is tracked another way: the submissions "jobs" table records a job's lifecycle in its "state" column (written asynchronously, compute-side), so a numeric exit status recorded eagerly on the login node would be meaningless there. This is a framework internal, not part of the user-facing decorator surface.

Variables

String _KNIT_CALL_ALIAS

One-shot call-site alias for the next command invocation. knit_as sets it (to the user-supplied alias) immediately before delegating to knit; the first _knit_invoke_command it reaches captures it into _KNIT_EXECUTING_ALIAS and clears it, so exactly one edge — the directly named call — carries the alias.

AssociativeArray _KNIT_COMMANDS

List of registered commands.

Array _KNIT_EXECUTING_ALIAS

Stack of call-site aliases, parallel to _KNIT_EXECUTING_COMMAND: entry i holds the alias knit_as named frame i's invocation with, or empty for a plain call. Captured when the frame is pushed (from _KNIT_CALL_ALIAS, which knit_as sets just before delegating) and read back by _knit_record_invocation, which writes it to the frame's "call" edge alias column. Per-frame, so an alias on a dispatcher call never leaks onto the nested edges its body records.

Array _KNIT_EXECUTING_COMMAND

Stack of currently-executing command names (mangled). Used by knit_output.

Array _KNIT_EXECUTING_ROW_ID

Stack of resolved row ids, parallel to _KNIT_EXECUTING_COMMAND: entry i holds the id that frame i's invocation will record. The id is resolved when the frame is pushed (so a nested callee can read its caller's id while the caller's body still runs) and read back by _knit_record_invocation, so the id a callee saw as its edge source is exactly the id the caller records. knit_set_row_id updates the top entry so an explicit id and the recorded id never diverge.

Array _KNIT_EXECUTING_START_TIME

Stack of start timestamps, parallel to _KNIT_EXECUTING_COMMAND: entry i holds the epoch seconds captured when frame i was pushed (before its body ran). _knit_record_invocation reads the top entry as the call edge's start_time and pairs it with an end_time captured at record time (after the body and after-callbacks), so an invocation's duration is a plain subtraction.

Array _KNIT_INVOCATION_ARGS

The editable, EXPANDED argument list (defaults and flag values spliced in) of the command whose before-callbacks are currently running. The invoker publishes it just before the before-callback phase of a normal command and reads any edit back into the args it then passes to the body (and records), so a before-callback can edit what the body sees via knit_set_invocation_args / knit_set_parameter. Saved and restored around every invocation so a nested command does not corrupt an enclosing edit; empty outside a before-callback. Distinct from _KNIT_INVOCATION_RAW_ARGS (the raw, pre-expansion, read-only tokens).

String _KNIT_INVOCATION_CMD

The mangled name of the command whose before-callbacks are currently running, published alongside _KNIT_INVOCATION_ARGS. knit_set_parameter reads it to validate that a name is a declared parameter of that command. Empty outside a (non-wrapper) command's before-callback phase.

String _KNIT_INVOCATION_END_TIME

Completion timestamp of the current invocation, captured after its body and after-callbacks but before any output checksum is computed, so hashing a (possibly large) output is excluded from the recorded duration. When non-empty _knit_record_invocation uses it as the call edge's end_time instead of reading the clock itself; it is read once and cleared, so the eager and wrapper record paths (which never set it) keep capturing the clock at record time.

String _KNIT_INVOCATION_EXIT_STATUS

Exit status of the current invocation's body, captured after it returns. When non-empty _knit_record_invocation writes it into the row's reserved "__exit_status__" column (for a command whose table has that column); it is read once and cleared, so the eager record path (which never sets it) leaves the column NULL until the outcome is known and updated separately.

Array _KNIT_INVOCATION_RAW_ARGS

Raw (pre-expansion) arguments of the current command invocation. Set by _knit_invoke_command from the exact tokens the user typed, before optional defaults and flag values are spliced in, so a command body can tell an option the user typed apart from one that took its default. Read it with _knit_arg_was_provided.

The array is overwritten at the start of every invocation, including nested ones. A command body that runs nested commands (for example "bootstrap", which calls "knit setup" and "metadata store") MUST copy this global into a local at the very top of its body, before any nested call, or the nested call overwrites it. This is the copy-immediately contract.

String _KNIT_LAST_ROW_ID

Row id of the most recently recorded invocation. Exposed so a dispatcher can learn the id resolved for a body it invoked, after that body returns and its entry has been popped off _KNIT_EXECUTING_ROW_ID. knit setup uses it to write the setup body's row id to .setup.id, so a later consumer can record a "used_by" edge to the setup by id rather than by matching directory paths.

AssociativeArray _KNIT_PARAMETER_SETS

Set of defined parameter set names (normalized).

String _KNIT_RECORDING_SUPPRESSED

When non-empty, output recording is suppressed: knit_output discards its value (with a warning) and _knit_record_invocation records no row. This is a generic recording concept (the CLI layer stays unaware of MPI); the knit run per-rank worker sets it on every rank but rank 0, so a run's outputs and per-app row are recorded exactly once even though every rank re-enters the app command.

Array _KNIT_ROOT_COMMANDS

Top-level commands (those with no parent), in registration order. Together with the per-command "_KNIT_CMD_<cmd>_subcommands" arrays this forms an explicit command-tree adjacency so help/describe can traverse the tree in declaration order without deriving it from prefix-matching _KNIT_COMMANDS.

AssociativeArray _KNIT_USED_ALIASES

Set of call-site aliases already used within each invocation, so knit_as can reject a reused alias (which would make two edges indistinguishable in a query). Keyed by "<parent-row-id>:<alias>", where the parent row id scopes the alias to the calling invocation (empty for a root-level call).

db.sh

Functions

int _knit_db_check_table()

Check whether a table exists and already carries every declared column. Returns 0 if the table exists and every desired column is present, 1 if the table does not exist, or 2 if the table exists but is missing one or more of the desired columns.

The match is a superset test: columns present in the table but absent from the desired specification (e.g. a parameter later removed from the command) are ignored, and so are column order and column type. Migration is additive only (see _knit_db_migrate_table), so a present column of any type is treated as satisfying the specification and never triggers a change.

Example:

_knit_db_check_table "runs" "id:uuid" "duration:real"
# returns 0, 1, or 2

Parameters:

• table_name -- [in] Name of the table to check.

• ...specs -- [in] One or more "column-name:type" specifications.

Returns:

0 if every desired column is present, 1 if absent, 2 if any is missing.

int _knit_db_command_has_exit_status()

Decide whether a command's table carries the reserved "__exit_status__" column. A command has it unless it opted out of recording a failed invocation (knit_no_record_on_failure — it records only successes, so the status would always be 0) or its status is tracked another way (_knit_without_exit_status, e.g. the submissions "jobs" table's "state"). This is the single predicate used by both the schema builder (_knit_db_setup_table) and the row recorder (_knit_db_record_invocation) so the column set and the recorded columns cannot diverge.

Parameters:

• cmd -- [in] Mangled command name.

Returns:

0 if the table has the column, 1 otherwise.

int _knit_db_create_table()

Create a new table in the Knit database. Each column specification must be of the form "name:type" where type is a valid Knit type. Column names are normalized (hyphens converted to underscores). Fails with a fatal error if the table already exists, if no columns are specified, if a column spec is malformed, or if a type is unknown.

Example:

_knit_db_create_table "runs" "id:uuid" "duration:real" "label:string"

Parameters:

• table_name -- [in] Name of the table to create.

• ...specs -- [in] One or more "column-name:type" specifications.

int _knit_db_ensure_table()

Ensure a table exists and carries every column in the given "name:type" specification: create it if absent, add any missing columns if present, or do nothing if it is already a superset of the specification. This is the shared create-or-migrate step used by _knit_db_setup_table for both wrapper and ordinary command tables.

Parameters:

• table_name -- [in] Name of the table to ensure.

• ...specs -- [in] One or more "name:type" specifications.

int _knit_db_migrate_table()

Migrate an existing table so it carries every declared column. Migration is additive only: each desired "name:type" column that the table does not already have is appended with "ALTER TABLE ... ADD COLUMN". A column already present is left untouched (its type is never changed), and a column present in the table but absent from the specification (e.g. a parameter later removed from the command) is kept, preserving the values recorded for it. Column names are normalized (hyphens converted to underscores).

A newly added column carries no SQL default: existing rows read NULL for it (the run predates the column) and a future row that does not record the column is NULL too, rather than a stale copy of some default value. Dropping a column is never automatic; that is an explicit, separate operation.

Example:

_knit_db_migrate_table "runs" "id:uuid" "count:integer" "label:string"

Parameters:

• table_name -- [in] Name of the table to migrate.

• ...specs -- [in] One or more "name:type" specifications.

Returns:

0 if the migration was applied or no migration was needed.

int _knit_db_record_invocation()

Insert one row into a command's table, recording an invocation, and — when a provenance edge is requested — the matching edge into the provenance table, both in a single transaction. The row is built from the command's declared schema: the "id" column (a caller-supplied uuid), then the value of every required parameter, optional parameter, and flag (read from the expanded invocation arguments), then every output (read from the in-memory KNIT_CMD<cmd>_output_value store populated by knit_output, falling back to the output's declared default). Column names are the normalized (underscored) knit names, matching the schema created by _knit_db_setup_table.

The provenance edge, when requested, has the recorded row as its target (target_id = id, target_name = the demangled command name). An empty edge_type means "record no edge" — the row is inserted on its own, unchanged from before provenance existed. A non-empty edge_type (e.g. "call") writes both the row and the edge atomically, so a partial state cannot be observed.

Parameters:

• cmd -- [in] Mangled command name (as used in KNIT_CMD* variables).

• table -- [in] Table to insert into.

• id -- [in] Value for the "id" column (the target's uuid).

• source_id -- [in] Provenance edge source id (empty for a root); see prov.sh.

• source_name -- [in] Provenance edge source name (empty for a root).

• edge_type -- [in] Edge type (e.g. "call"), or empty to record no edge.

• start_time -- [in] Edge start_time (epoch seconds, empty -> NULL).

• end_time -- [in] Edge end_time (epoch seconds, empty -> NULL).

• alias -- [in] Edge call-site alias (empty -> NULL); see prov.sh.

• exit_status -- [in] Body exit status for the reserved "__exit_status__" column; empty leaves it NULL (the outcome is not yet known, e.g. the eager record path). Ignored for a command whose table has no such column.

• ... -- [in] The expanded invocation arguments (params/flags to read).

int _knit_db_setup_table()

Done callback installed by knit_with_table. Inspects the registered parameters, flags, and outputs of the command and ensures the database table carries a column for each — creating the table if absent or adding any missing columns if the schema has grown.

Column order at creation: "id" (uuid) first, the reserved "__exit_status__" column (unless the command opted out, see _knit_db_command_has_exit_status), then required parameters, optional parameters, flags, and outputs, each group sorted alphabetically. Migration is additive only (see _knit_db_migrate_table): a later-added column lands at the end of the table, and a column for a parameter that was removed from the command is kept, so physical column order need not match this declared order over the life of a table.

Parameters:

• cmd -- [in] Mangled command name (as used in KNIT_CMD* variables).

• table_name -- [in] Name of the database table to create or migrate.

int _knit_db_sql_ident()

Wrap an SQL identifier (table or column name) in double quotes, escaping any embedded double-quote characters by doubling them, per the SQL standard.

Example:

local q; _knit_db_sql_ident q "my_table"   # q == "my_table"
local q; _knit_db_sql_ident q 'a"b'        # q == "a""b"

Parameters:

• __knit_ret -- [out] Name of the variable to hold the quoted identifier.

• name -- [in] Identifier to quote.

Variables

AssociativeArray _KNIT_DB_REGISTERED_TABLES

Associative array mapping table name to the demangled command name that registered it. Used to detect duplicate table use across commands.

describe.sh

Functions

int _knit_describe()

Body of the "describe" command: read the requested filters and --format, then emit the description in that format. With "--output <file>" the document is written to that file instead of standard output (which also disables the default format's auto-color, since the destination is not a terminal).

Parameters:

• ... -- [in] Command arguments (expanded by the CLI framework).

int _knit_describe_children()

Return the mangled names of the direct children of a command as an array, in registration (declaration) order. The parent is given as a mangled name, or the empty string to list the top-level (root) commands. This reads the command tree adjacency built at registration time (KNIT_ROOT_COMMANDS and the per-command "_KNIT_CMD</em><cmd>_subcommands" arrays), so it needs no per-invocation build/teardown.

It first ensures the parent's lazy subcommands (if any) are discovered, so a describe walk expands exactly the part of the tree it visits. That step is fork-free once a subtree is discovered (the common case); the first walk into a lazy subtree pays its registration cost, which is the deferred work this feature is designed to avoid up front.

Parameters:

• __knit_ret -- [out] Name of the array variable to populate (nameref output).

• parent -- [in] Mangled parent command name, or "" for top-level commands.

int _knit_describe_command_kind()

Print the structural kind of a command, read from the _type field set at registration: "wrapper" (knit_register_wrapper), "job" (knit_register_job), "app" (knit_register_app), "setup" (a setup command), or "command" otherwise. This is the registered kind, not a guess from the parent: "submit" has both jobs and ordinary subcommands (submit prepared / submit next), so a child of a dispatcher is not necessarily a job.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the command kind.

• cmd -- [in] Mangled command name.

int _knit_describe_default()

Emit the human-readable description of the experiment: one titled block per command, walked depth-first so a subcommand follows its parent. Color is enabled only when stdout is a terminal and "--no-color" is not given.

Parameters:

• ... -- [in] Command arguments (expanded by the CLI framework).

int _knit_describe_default_artifacts()

Print a command's "Artifacts" section for the human-readable format (name, then "[type] description", with a trailing ", result" in the bracket for an artifact that carries "--result"). An artifact has no default, so none is shown. The artifacts are read from the command's artifacts set; the caller prints this section only for a command that declares one.

Parameters:

• cmd -- [in] Mangled command name.

• use_color -- [in] "true" to emit ANSI styling in the header.

• indent -- [in] Leading indentation for the section header (defaults to none); entries are indented two further spaces.

int _knit_describe_default_command()

Print one command as a titled block for the human-readable format, then recurse (flat, depth-first) into its emitted subcommands so each command is its own block titled by its full space-separated name. The Options and Outputs sections honor the "--no-input-params" / "--no-output-params" filters, an "Extra" section is printed when the command declares post-"--" arguments, and an "Implementation" section (the function body) is printed for a user command when "--include-implementation" is set.

Parameters:

• cmd -- [in] Mangled command name.

• use_color -- [in] "true" to emit ANSI styling.

• sel_ancestor -- [in] "true" if an ancestor of the command is in the "--only" selection.

int _knit_describe_default_heading()

Print a title or section header for the human-readable format. With color it is rendered bold and underlined on its own line; without color it is followed by an "---" hrule of matching width (the style "--help" uses).

Parameters:

• text -- [in] Header text.

• use_color -- [in] "true" to emit ANSI styling, "false" for an hrule.

• indent -- [in] Leading indentation (defaults to none).

int _knit_describe_default_options()

Print a command's "Options" section for the human-readable format, mirroring the "--help" layout: a header, the "--help" entry, then required, optional (with "default: '…'") and flag parameters, column-aligned, each annotated with its enum constraint, "resource:" type (for a knit_with_resource parameter), "artifact:" kind (for a knit_with_input_artifact parameter), and "when:" clause when present.

Parameters:

• cmd -- [in] Mangled command name.

• use_color -- [in] "true" to emit ANSI styling in the header.

• indent -- [in] Leading indentation for the section header (defaults to none); entries are indented two further spaces.

int _knit_describe_default_outputs()

Print a command's "Outputs" section for the human-readable format (name, then "[type, default: '…'] description", with a trailing ", result" in the bracket for an output that carries "--result"). An artifact is not an output column, so it is not listed here; it gets its own "Artifacts" section.

Parameters:

• cmd -- [in] Mangled command name.

• use_color -- [in] "true" to emit ANSI styling in the header.

• indent -- [in] Leading indentation for the section header (defaults to none); entries are indented two further spaces.

int _knit_describe_emit()

Emit the description in the requested format to standard output. Split out from _knit_describe so the caller can redirect the whole document to a file for "--output" without duplicating the format dispatch. "--compact" selects the single-line JSON variant and applies only to the "json" format.

Parameters:

• format -- [in] Output format ("default", "json", "yaml", or "markdown").

• compact -- [in] "true" to emit compact single-line JSON (json format only).

• ... -- [in] Command arguments (expanded by the CLI framework).

int _knit_describe_emit_array()

Emit a JSON array from a list of already-rendered elements. The opening bracket is printed inline; each element must already carry its own indentation; the closing bracket is printed at the given indent. An empty element list yields "[]".

Parameters:

• indent -- [in] Indentation string for the closing bracket.

• ...elements -- [in] Pre-rendered array elements (indented).

int _knit_describe_emit_object()

Emit a JSON object from a list of already-rendered entries. The opening brace is printed inline (the caller positions it, e.g. right after a "key": prefix); each entry must already carry its own indentation; the closing brace is printed at the given indent. An empty entry list yields "{}".

Parameters:

• indent -- [in] Indentation string for the closing brace.

• ...entries -- [in] Pre-rendered "key": value fragments (indented).

int _knit_describe_enum_constraint()

Return a "one of: a, b, c" constraint string for an enum-typed parameter, or the empty string when the parameter's type is not an enum. Values are sorted to match the other formatters.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the constraint string.

• cmd -- [in] Mangled command name.

• param -- [in] Normalized parameter name.

int _knit_describe_enum_values_json()

Return the values of an enum as an inline JSON array of strings (sorted for a stable order).

Parameters:

• __knit_ret -- [out] Name of the variable to hold the JSON array.

• name -- [in] Enum type name.

int _knit_describe_filter_on()

Return success if the named boolean filter is enabled for the current invocation. Unknown or unset keys are treated as disabled.

Parameters:

• name -- [in] Filter key (see _KNIT_DESCRIBE_FILTERS).

int _knit_describe_implementation()

Print a command's implementation — its registered function body as produced by "declare -f" — when "--include-implementation" is active and the command is a user (non-builtin) command. Prints nothing otherwise: without the flag no body is emitted, and a builtin's body is knit implementation detail that is never dumped. Every formatter consults this so the rule is applied in one place.

Parameters:

• cmd -- [in] Mangled command name.

int _knit_describe_is_result()

Return success if an output of a command carries the "--result" mark (it is a member of the command's results set). The set does not exist for a command with no result, in which case the membership test simply fails.

Parameters:

• cmd -- [in] Mangled command name.

• output -- [in] Normalized output name.

int _knit_describe_is_selected()

Decide whether a command's own block should be rendered, as opposed to the command being kept only as a path container for a selected descendant. Returns success when:

• no "--only" selection is active (every command is rendered), or

• the command is itself selected by "--only", or

• an ancestor is selected and "--recursive" is set. A command that is emitted only to preserve the path down to a selected descendant returns failure. The flat and Markdown formatters use this to skip a container's block (the child's full name already conveys the hierarchy) while still recursing into it; the tree formats (JSON/YAML) ignore it and emit every command _knit_describe_should_emit keeps, because a child cannot nest without its parent. Call only for a command already known to be emitted.

Parameters:

• cmd -- [in] Mangled command name.

• sel_ancestor -- [in] "true" if an ancestor of the command is in the "--only" selection, "false" otherwise.

int _knit_describe_json()

Emit the complete description of the experiment as a JSON document: knit version, experiment (script) name, format version, the full command tree (top-level commands with nested subcommands, hidden commands excluded), and the map of user-defined enums.

int _knit_describe_json_artifact()

Render one artifact as a JSON object (array element: leading indent included). An artifact is a produced entity, not an output column, so it carries no "default"; the "result" boolean says it carries the "--result" mark.

Parameters:

• cmd -- [in] Mangled command name.

• output -- [in] Normalized artifact name.

• indent -- [in] Indentation of the object's opening brace.

int _knit_describe_json_artifacts()

Render a command's "artifacts" array (one object per declared artifact). Printed inline (object value: no leading indent). Yields "[]" for a command that declares no artifact (the set is absent then).

Parameters:

• cmd -- [in] Mangled command name.

• indent -- [in] Indentation of the array's opening bracket.

int _knit_describe_json_command()

Render a single command (and, recursively, its subcommands) as a JSON object. The object is an array element, so its leading indent is included. Subcommands are pruned by the active filters (hidden/builtin/"--only"), matching the top-level command list.

Parameters:

• cmd -- [in] Mangled command name.

• indent -- [in] Indentation of the object's opening brace.

• sel_ancestor -- [in] "true" if an ancestor of the command is in the "--only" selection.

int _knit_describe_json_compact()

Emit the JSON description as a single compact line (no indentation, no inter-entry newlines, and no spaces after ":" or ","). It shadows the JSON builders' whitespace separators (_KNIT_DESCRIBE_JSON_NL / _CS / _IND) with the empty string for the duration of one _knit_describe_json call, so the compact form is produced directly by the same builder tree — no second minify pass. The locals are visible to the builders (and their command-substitution subshells) through bash dynamic scoping, and are restored automatically on return.

int _knit_describe_json_enums()

Render the top-level "enums" object: every user-defined (non-builtin) enum mapped to its values. Printed inline (object value: no leading indent).

Parameters:

• indent -- [in] Indentation of the object's opening brace.

int _knit_describe_json_escape()

Escape a string so it can be embedded inside a JSON string literal, without any external dependency (no jq), so "describe" works on a fresh checkout before bootstrap. Backslashes and double quotes are backslash-escaped; newlines, CR, and tabs become their short escapes; any remaining control character becomes a "\uXXXX" escape. The surrounding quotes are NOT added (see _knit_describe_json_str).

Parameters:

• __knit_ret -- [out] Name of the variable to hold the escaped string.

• string -- [in] String to escape.

int _knit_describe_json_output()

Render one output as a JSON object (array element: leading indent included). The "result" boolean says the output carries the "--result" mark. An artifact is not an output column, so it is not rendered here (see _knit_describe_json_artifact).

Parameters:

• cmd -- [in] Mangled command name.

• output -- [in] Normalized output name.

• indent -- [in] Indentation of the object's opening brace.

int _knit_describe_json_outputs()

Render a command's "outputs" array (value output columns only; artifacts get their own "artifacts" array). Printed inline (object value: no leading indent).

Parameters:

• cmd -- [in] Mangled command name.

• indent -- [in] Indentation of the array's opening bracket.

int _knit_describe_json_param()

Render one input parameter as a JSON object (array element: the leading indent is included). The rendered fields depend on the group: "required" has no default, "optional" adds a raw default, and "flags" are always boolean with no default or enum. An enum-typed parameter inlines its allowed values, a resource parameter (knit_with_resource) adds its "resource" type, an input-artifact parameter (knit_with_input_artifact) adds its required "artifact" kind, and a "--when" constraint is included when present.

Parameters:

• cmd -- [in] Mangled command name.

• group -- [in] Parameter group ("required", "optional", or "flags").

• param -- [in] Normalized parameter name.

• indent -- [in] Indentation of the object's opening brace.

int _knit_describe_json_params()

Render a command's "parameters" object (required, optional, flags arrays, and the extra description). Printed inline (object value: no leading indent).

Parameters:

• cmd -- [in] Mangled command name.

• indent -- [in] Indentation of the object's opening brace.

int _knit_describe_json_space()

Render a parameter's declared search domain as a JSON object (value: no leading indent; the closing brace is printed at the given indent). The object mirrors knit_parameter_space_json's shape — kind, type, lo, hi, step, glob (strings), log (boolean), and values (array of strings) — but is hand-built with the describe JSON helpers so it respects the pretty/compact whitespace and needs no jq. Reads the domain through knit_parameter_space (parse + validate on read).

Parameters:

• cmd -- [in] Mangled command name.

• param -- [in] Normalized parameter name.

• indent -- [in] Indentation of the object's closing brace.

int _knit_describe_json_str()

Return a value as a quoted, escaped JSON string literal.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the JSON string literal.

• string -- [in] Value to render.

int _knit_describe_markdown()

Emit the complete description of the experiment as a single Markdown document: a "#" title (the program description, or the script name when unset), a "##

Commands" wrapper, and one flat "###" section per command (depth-first, so a subcommand follows its parent). Depth does not consume heading levels, keeping the scheme within Markdown's six-level limit. Enum values are inlined in each parameter's Constraints column rather than a separate section.

int _knit_describe_md_artifacts()

Print a command's "#### Artifacts" sub-section as a Markdown table (one row per artifact). A "Result" column shows "result" for an artifact that carries "--result". An artifact has no default, so no Default column is shown. The artifacts are read from the command's artifacts set; the caller prints this sub-section only for a command that declares one.

Parameters:

• cmd -- [in] Mangled command name.

int _knit_describe_md_cell()

Escape a value so it is safe inside a Markdown table cell: pipes are backslash- escaped (they otherwise start a new column) and newlines/carriage returns are folded to spaces (a table row must stay on one line).

Parameters:

• __knit_ret -- [out] Name of the variable to hold the escaped value.

• value -- [in] Value to escape.

int _knit_describe_md_code()

Render a value as a Markdown inline-code span (escaped for a table cell). An empty value yields nothing, so this doubles as the "Default" column renderer: a declared value is shown as code and an empty-string default leaves a blank cell.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the rendered code span.

• value -- [in] Value to render.

int _knit_describe_md_command()

Render one command as a "### <full name>" Markdown section (an intro line with its kind/builtin note and description, an italic "Extra" line when declared, the "#### Parameters" / "#### Outputs" sub-sections honoring the omit flags, and a fenced "#### Implementation" block for a user command when "--include-implementation" is set), then recurse (flat, depth-first) into its emitted subcommands so each command is its own "###" section regardless of depth.

Parameters:

• cmd -- [in] Mangled command name.

• sel_ancestor -- [in] "true" if an ancestor of the command is in the "--only" selection.

int _knit_describe_md_constraints()

Build the "Constraints" column text for a parameter: the enum "one of: …" list (when the type is an enum), the "resource: <type>" annotation (when the parameter was declared with knit_with_resource), the "artifact: <kind>" annotation (when the parameter was declared with knit_with_input_artifact), and the "--when" clause (with the raw expression in inline code), joined by "; ". Returns the empty string when the parameter is unconstrained.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the constraints text.

• cmd -- [in] Mangled command name.

• param -- [in] Normalized parameter name.

int _knit_describe_md_outputs()

Print a command's "#### Outputs" sub-section as a Markdown table (one row per output). A "Result" column shows "result" for an output that carries "--result". An artifact is not an output column, so it is not listed here; it gets its own "#### Artifacts" sub-section. Prints "*None.*" when the command declares no output.

Parameters:

• cmd -- [in] Mangled command name.

int _knit_describe_md_params()

Print a command's "#### Parameters" sub-section as a Markdown table (one row per required, optional, and flag parameter, in that order; each group sorted), with a "Kind" column marking the group. Prints "*None.*" when the command declares no parameters. The universal "--help" flag is intentionally omitted.

Parameters:

• cmd -- [in] Mangled command name.

int _knit_describe_read_filters()

Populate the module-level filter state (_KNIT_DESCRIBE_FILTERS and _KNIT_DESCRIBE_ONLY) from the current invocation's arguments, so the model/traversal layer applies the requested filtering. Called by _knit_describe before any formatter runs.

Parameters:

• ... -- [in] Command arguments (expanded by the CLI framework).

int _knit_describe_should_emit()

Decide whether a command appears in the filtered command tree. A command is emitted when it is visible (passes the hidden/builtin filters) and either:

• no "--only" selection is active (every visible command is selected), or

• it is itself selected by "--only", or

• an ancestor is selected and "--recursive" is set, or

• it has a descendant that is itself emitted (so it is kept as a container that preserves the path down to a selected command).

Parameters:

• cmd -- [in] Mangled command name.

• sel_ancestor -- [in] "true" if an ancestor of the command is in the "--only" selection, "false" otherwise.

int _knit_describe_space_note()

Build the compact search-domain note for a parameter in the textual formats: "range [lo,hi]" (plus " step S" and/or " log" when set), "values {a,b,c}", or "glob <pattern>". Returns the empty string when the parameter declares no domain. The declared domain is read through knit_parameter_space, which parses and validates it, so an invalid declared domain is fatal here (on read).

Parameters:

• __knit_ret -- [out] Name of the variable to hold the note.

• cmd -- [in] Mangled command name.

• param -- [in] Normalized parameter name.

int _knit_describe_visible()

Return success if a command passes the hidden/builtin filters, independent of any "--only" selection: hidden commands are dropped unless "--include-hidden" is set, and builtin commands are dropped when "--exclude-builtins" is set.

Parameters:

• cmd -- [in] Mangled command name.

int _knit_describe_yaml()

Emit the complete description of the experiment as a YAML document, serializing the identical model as _knit_describe_json (same keys, nesting, and field semantics): knit version, experiment (script) name, format version, the filtered command tree, and the map of user-defined enums.

int _knit_describe_yaml_artifact()

Render one artifact as a YAML block-sequence element (the first key carries the "- " indicator at item_indent). An artifact is a produced entity, not an output column, so it carries no "default"; the "result" boolean says it carries the "--result" mark.

Parameters:

• cmd -- [in] Mangled command name.

• output -- [in] Normalized artifact name.

• item_indent -- [in] Indentation of the "- " sequence indicator.

int _knit_describe_yaml_command()

Render a single command (and, recursively, its subcommands) as a YAML block-sequence element (the first key carries the "- " indicator at item_indent). Subcommands are pruned by the active filters (hidden/builtin/"--only"), matching the top-level command list.

Parameters:

• cmd -- [in] Mangled command name.

• item_indent -- [in] Indentation of the "- " sequence indicator.

• sel_ancestor -- [in] "true" if an ancestor of the command is in the "--only" selection.

int _knit_describe_yaml_enums()

Render the top-level "enums:" mapping: every user-defined (non-builtin) enum mapped to its values as a flow sequence. Yields "enums: {}" when there are none.

int _knit_describe_yaml_flow_seq()

Print a YAML flow sequence ("[a, b, c]") of scalar values, each double-quoted when it would be unsafe or coerced as a plain flow scalar (the plain-scalar rules plus the flow indicators , [ ] { }). An empty list yields "[]".

Parameters:

• __knit_ret -- [out] Name of the variable to hold the flow sequence.

• ...values -- [in] Scalar values.

int _knit_describe_yaml_needs_quote()

Return success when a single-line string value must be double-quoted to survive a YAML round-trip as a string: leaving it as a plain scalar would either coerce it to another type (number, boolean, null) or be syntactically unsafe. Double-quoting an already-safe value is harmless, so the predicate errs toward quoting. Multi-line values are handled separately as block scalars.

Parameters:

• value -- [in] String value to test.

int _knit_describe_yaml_output()

Render one output as a YAML block-sequence element (the first key carries the "- " indicator at item_indent). The "result" boolean says the output carries the "--result" mark. An artifact is not an output column, so it is not rendered here (see _knit_describe_yaml_artifact).

Parameters:

• cmd -- [in] Mangled command name.

• output -- [in] Normalized output name.

• item_indent -- [in] Indentation of the "- " sequence indicator.

int _knit_describe_yaml_param()

Render one input parameter as a YAML block-sequence element (the first key carries the "- " indicator at item_indent). Mirrors the JSON parameter object: an enum type inlines its allowed values, "optional" carries a raw default, flags are boolean with no default, a resource parameter (knit_with_resource) adds its "resource" type, an input-artifact parameter (knit_with_input_artifact) adds its required "artifact" kind, and a "--when" constraint is included when present.

Parameters:

• cmd -- [in] Mangled command name.

• group -- [in] Parameter group ("required", "optional", or "flags").

• param -- [in] Normalized parameter name.

• item_indent -- [in] Indentation of the "- " sequence indicator.

int _knit_describe_yaml_params()

Render a command's "parameters:" mapping body: the required/optional/flags sequences (each "[]" when empty) and the "extra" scalar (or null), with the group keys at keys_indent. The "parameters:" key line itself is printed by the caller.

Parameters:

• cmd -- [in] Mangled command name.

• keys_indent -- [in] Indentation of the required/optional/flags keys.

int _knit_describe_yaml_scalar()

Render a string value as a YAML scalar to follow "key: ". A multi-line value becomes a literal block scalar ("|-") whose lines are indented by cont_indent; a single-line value that would be coerced or is unsafe as a plain scalar is double-quoted (reusing the JSON escaper, whose escapes YAML's double-quoted style shares); anything else is emitted verbatim.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the rendered scalar.

• value -- [in] String value to render.

• cont_indent -- [in] Indentation prepended to each line of a block scalar.

int _knit_describe_yaml_space()

Render a parameter's declared search domain as a nested YAML "space:" mapping, mirroring the JSON shape (kind, type, lo, hi, step, glob, log, values). The "space:" key is printed at key_indent; its members two spaces deeper. Reads the domain through knit_parameter_space (parse + validate on read).

Parameters:

• cmd -- [in] Mangled command name.

• param -- [in] Normalized parameter name.

• key_indent -- [in] Indentation of the "space:" key.

Variables

String _KNIT_DESCRIBE_EMITTED

Scratch flag the flat formatter uses to place inter-block blank lines: 0 until the first command block is printed, 1 afterwards, so a separator precedes every block except the first. Reset at the start of _knit_describe_default. It exists because a command kept only as a path container prints nothing, so the separator cannot be tied to the tree walk.

AssociativeArray _KNIT_DESCRIBE_FILTERS

Boolean filter state for the current "describe" invocation, populated by _knit_describe from the parsed flags and consulted by the model/traversal layer so every formatter inherits the same filtering. Keys: "exclude_builtins", "no_input_params", "no_output_params", "include_hidden", and "recursive"; each value is "true" or "false". A missing key defaults to "false", so the default (no filtering) also applies when the traversal is exercised directly.

String _KNIT_DESCRIBE_JSON_CS

Insignificant space the JSON builders insert after each ":" and inline ",". Defaults to a single space (pretty); shadowed with the empty string by _knit_describe_json_compact for compact output.

String _KNIT_DESCRIBE_JSON_IND

One level of indentation for the JSON builders. Defaults to two spaces (pretty); shadowed with the empty string by _knit_describe_json_compact so all indentation collapses for compact output.

String _KNIT_DESCRIBE_JSON_NL

Inter-entry newline the JSON builders insert inside objects and arrays. Its default (a newline) produces pretty-printed JSON; _knit_describe_json_compact shadows it with the empty string to emit single-line compact JSON without a second minify pass.

AssociativeArray _KNIT_DESCRIBE_ONLY

Set of mangled command names selected by "--only" for the current "describe" invocation. Empty means no selection was given, in which case every (visible) command is described.

detect.sh

Functions

int _knit_command_path()

Print the absolute path to a system executable if it exists on PATH, or nothing when it is absent. Thin wrapper around command -v that gives the bootstrap a single, easily stubbable resolution point for the system binaries (sqlite3, jq) it may symlink instead of building from source.

Parameters:

• name -- [in] Name of the executable to look up.

int _knit_detect_job_manager()

Detect which batch job manager is available in the current environment.

Outputs one of the following strings to stdout and returns 0:

• "flux" — FLUX_URI names a live Flux instance that owns this shell

• "slurm" — sbatch is present in PATH

• "pbs" — qsub is present in PATH (and sbatch is not)

• "flux" — flux is present in PATH (and no sbatch/qsub)

• "<unknown>" — none of the above

The result is cached in _KNIT_DETECTED_JOB_MANAGER so that subsequent calls within the same session return immediately without re-probing the PATH.

FLUX_URI is tested first because Flux is often deployed inside a Slurm or PBS allocation, so flux, sbatch, and qsub can all be on PATH at once. A live FLUX_URI means a Flux instance owns this shell, so Flux is the right backend; without it a Flux-under-Slurm login shell still detects as Slurm. When both sbatch and qsub are present (but no FLUX_URI), Slurm takes priority. A system flux with no live instance is the last resort before <unknown>.

int _knit_detect_launcher()

Detect which MPI launcher to use in the current environment.

Outputs one of the following strings to stdout and returns 0:

• "flux" — the detected scheduler is Flux (flux run wins regardless of any MPI-native launcher on PATH)

• "pals" — the MPI-native probe found PALS

• "openmpi" — the MPI-native probe found OpenMPI

• "mpich" — the MPI-native probe found MPICH

• "flux" — no MPI-native launcher, but flux is in PATH

• "<unknown>" — no recognised launcher found

Detection is scheduler-aware. When the detected scheduler is Flux, "flux run" is the reliable native launcher and wins even if an MPI-native launcher is on PATH: flux run bootstraps MPI through Flux's own PMI, and a bare mpirun may not reach the compute nodes inside a Flux allocation. Under any other scheduler (slurm, pbs, none, <unknown>) an MPI-native launcher on PATH wins, because a scheduler-integrated launcher such as srun needs the MPI built against that manager's PMI (which knit cannot verify), whereas mpirun/mpiexec always match their own MPI.

The MPI-native probe is delegated to _knit_detect_mpi_launcher(). Flux is the last resort when it is not the scheduler: an MPI-native launcher wins when present, so a system flux on PATH is chosen only after that probe finds nothing.

The result is cached in _KNIT_DETECTED_LAUNCHER so that subsequent calls within the same session return immediately without re-probing the PATH.

int _knit_detect_mpi()

Detect which MPI implementation is available in the current environment.

Outputs one of the following strings to stdout and returns 0:

• "openmpi" — mpirun is present and its --version output contains "Open MPI"

• "mpich" — mpirun is present and its --version output contains "HYDRA"

• "<unknown>" — mpirun is absent or its version string is not recognised

The result is cached in _KNIT_DETECTED_MPI so that subsequent calls within the same session return immediately without re-running mpirun.

int _knit_detect_mpi_launcher()

Detect which MPI-native launcher is available on PATH, ignoring the scheduler.

Outputs one of the following strings to stdout and returns 0:

• "pals" — mpiexec is present and its --help output starts with "Parallel Application Launch Service"

• "openmpi" — mpirun is present and its --version output contains "Open MPI"

• "mpich" — mpirun is present and its --version output contains "HYDRA"

• "<unknown>" — no recognised MPI-native launcher found

This probe is deliberately scheduler-unaware and never returns "flux": it reports only the MPI-native launcher that ships with an MPI on PATH. PALS is checked first because PALS provides mpiexec but not mpirun; OpenMPI and MPICH also provide mpiexec as an alias, so the first-line check distinguishes them.

It is the right probe for a setup that declares knit_provides_launcher: such a setup always installs an MPI-native launcher, so its contract must never freeze to "flux" even when the setup runs on a Flux login node.

The result is cached in _KNIT_DETECTED_MPI_LAUNCHER so that subsequent calls within the same session return immediately without re-probing the PATH.

int _knit_detect_node_ncpus()

Detect the per-node core count of the current cluster by querying the detected batch scheduler and taking the modal (most common) value across its nodes:

• slurm: sinfo -h -N -o 'c' (one line per node)

• pbs: pbsnodes -a resources_available.ncpus lines

• flux: flux resource list total cores divided by total nodes

Counting is node-weighted (-N on Slurm; one entry per node on PBS), so a minority outlier such as a login node listed alongside the compute nodes is ignored in favour of the value the bulk of the nodes report.

Prints the detected core count (a positive integer) to stdout, or nothing when the query yields no usable value. Knit allocates whole nodes, so this feeds node_ncpus (cpus-per-node) on machines that have no profile. When there is no batch scheduler the count of the local machine (nproc, or getconf as a portable fallback) is used so that node_ncpus is still populated on a workstation. The result is cached in _KNIT_DETECTED_NODE_NCPUS.

The local fallback is deliberately confined to the no-scheduler case: on a machine with a scheduler the login node's core count is not representative of the compute nodes, so a failed scheduler query returns nothing rather than the (misleading) local value.

Variables

String _KNIT_DETECTED_JOB_MANAGER

Cache for _knit_detect_job_manager(). Empty means "not yet detected"; one of "flux", "slurm", "pbs", or "<unknown>" after the first successful detection call.

String _KNIT_DETECTED_LAUNCHER

Cache for _knit_detect_launcher(). Empty means "not yet detected"; one of "pals", "openmpi", "mpich", "flux", or "<unknown>" after the first successful detection call.

String _KNIT_DETECTED_MPI

Cache for _knit_detect_mpi(). Empty means "not yet detected"; one of "openmpi", "mpich", or "<unknown>" after the first successful detection call.

String _KNIT_DETECTED_MPI_LAUNCHER

Cache for _knit_detect_mpi_launcher(). Empty means "not yet detected"; one of "pals", "openmpi", "mpich", or "<unknown>" after the first successful detection call.

String _KNIT_DETECTED_NODE_NCPUS

Cache for _knit_detect_node_ncpus(). Empty means "not yet detected" (or undetectable); a positive integer once detection succeeds.

drain.sh

Functions

int _knit_drain_dry_run()

List the prepared jobs that would be released, in release order, without claiming or releasing anything. The peek mirrors _knit_prepare_claim_next: the same state='prepared' filter (plus optional job / group), the same ORDER BY id ASC, and --count applied as a LIMIT. It is a read-only SELECT (no write lock), so it never advances a job's state. Each matching job prints as "<uuid> <job>" (with " [<group>]" appended when the job has a group); with json_summary the peeked list is printed as JSON instead. A count of matches is reported to stderr.

Parameters:

• count -- [in] Maximum jobs to list, or "" for no cap.

• json_summary -- [in] "true" to print the JSON list instead of the table.

• job -- [in] Optional job-name filter (empty means any).

• group -- [in] Optional group filter (empty means any).

int _knit_drain_dry_run_json()

Print the dry-run peek as a JSON object to stdout: {dry_run, count, jobs}, where jobs is the array sqlite emits in -json mode (id / job / group per row). An empty result set yields an empty array. Built with knit_jq so the wrapper object is well-formed.

Parameters:

• sql -- [in] The read-only SELECT to run in sqlite -json mode.

int _knit_drain_emit_json()

Print the end-of-run summary as a single JSON object to stdout, built with knit_jq so it is always well-formed and correctly typed: released is a number, completed / failed are numbers in waiting modes and null in no-limit mode, drained / stopped / dry_run are booleans. dry_run is always false here (the dry-run path emits its own summary).

Parameters:

• mode -- [in] "no-limit" or "waiting".

• released -- [in] Number of jobs released this run.

• completed -- [in] Number that completed successfully ("" in no-limit).

• failed -- [in] Number that failed ("" in no-limit).

• drained -- [in] "true" when the prepared queue was emptied.

• stopped -- [in] "true" when a failure halted the drain early.

int _knit_drain_nolimit()

No-limit draining (--max-inflight 0): release every matching prepared job, back to back, without waiting. Stops when the queue is drained or --count is reached. No per-job outcome is observed, so this always succeeds on a clean drain.

No-limit never waits for an outcome, so with --show-progress the bar degrades to a release bar: it fills the completed (dark-purple) section by released-so-far and leaves running and failed at 0 (neither is observable here). It is an honest, cheap "how fast the backlog is being fired off" signal, not a success meter.

Parameters:

• count -- [in] Maximum jobs to release, or "" for no cap.

• json_summary -- [in] "true" to also print the JSON summary to stdout.

• show_progress -- [in] "true" to draw a live release bar (TTY only).

• total -- [in] Backlog size for the bar (jobs this pass will act on).

• ... -- [in] Filter arguments forwarded to _knit_drain_release_next.

int _knit_drain_pool()

Throttled draining (--max-inflight N > 1): keep at most N jobs alive at once. The pool tops up to N background workers, then blocks on "wait -n" until any worker finishes and reaps every worker whose result file is ready. A worker that claimed no job marks the queue drained; a worker whose job failed bumps the failure count and, with stop_on_failure, stops further top-ups. The top-up guard "released + inflight < count" caps total claims at --count (each worker claims at most one job). All in-flight workers are drained before returning. Returns 0 when every released job succeeded, 1 otherwise.

Parameters:

• max_inflight -- [in] Maximum concurrent workers (> 1).

• stop_on_failure -- [in] "true" to stop topping up after the first failure.

• count -- [in] Maximum jobs to release, or "" for no cap.

• json_summary -- [in] "true" to also print the JSON summary to stdout.

• show_progress -- [in] "true" to draw a live progress bar (TTY only).

• total -- [in] Backlog size for the bar (jobs this pass will act on).

• ... -- [in] Filter arguments forwarded to the workers.

int _knit_drain_pool_worker()

A single pool worker: release one job (waiting for it) and write its outcome as a "<uuid>\t<rc>" line to the given result file. Run in the background by _knit_drain_pool; the result file is how the pool reaps the worker and learns whether it claimed a job (non-empty uuid) and how it fared (rc). A worker always writes a result, so the pool never blocks on a worker that produced nothing.

Parameters:

• result_file -- [in] Path the worker writes its "<uuid>\t<rc>" line to.

• ... -- [in] Filter arguments forwarded to _knit_drain_release_next.

int _knit_drain_release_next()

Release the next prepared job by re-executing the experiment as submit next (without --wait), then, when asked to wait, block on job wait for its outcome. This is the single, stubbable point through which draining releases a job, so the loop logic can be unit-tested against a simulated queue without a live scheduler.

The released job's UUID is printed to stdout, and the exit status reflects its outcome when waiting: submit next prints the UUID and returns non-zero (empty stdout) once the queue is drained, so the caller distinguishes "drained" from "released" by an empty UUID; job wait then returns non-zero for a failed or killed job. submit --wait is deliberately not used — it returns 0 even for a job whose body failed, so it cannot detect failure (see features/submit-drain.md §4.2).

Parameters:

• wait_flag -- [in] "true" to wait for the released job's outcome (via job wait), else "false".

• ... -- [in] Filter arguments forwarded verbatim to submit next (for example --type / --group).

int _knit_drain_report()

Print the end-of-run summary. The human-readable line always goes to stderr (via the logging system): with no jobs released it reports that the queue was empty; in "no-limit" mode it reports only the count (no per-job outcome was observed); otherwise it reports the completed / failed breakdown. When json_summary is "true" a machine-readable JSON object is additionally printed to stdout (see _knit_drain_emit_json). Always returns 0 so it does not disturb the caller's exit status.

Parameters:

• mode -- [in] "no-limit" or "waiting".

• json_summary -- [in] "true" to also print the JSON object to stdout.

• released -- [in] Number of jobs released this run.

• completed -- [in] Number that completed successfully ("" in no-limit).

• failed -- [in] Number that failed ("" in no-limit).

• drained -- [in] "true" when the prepared queue was emptied.

• stopped -- [in] "true" when a failure halted the drain early.

int _knit_drain_serial()

Serial draining (--max-inflight 1): release one job, wait for it, release the next, until the queue is drained or --count is reached. Counts completed and failed jobs; with stop_on_failure it stops releasing new jobs after the first failure. Returns 0 when every released job succeeded, 1 otherwise.

Parameters:

• stop_on_failure -- [in] "true" to stop after the first failed job.

• count -- [in] Maximum jobs to release, or "" for no cap.

• json_summary -- [in] "true" to also print the JSON summary to stdout.

• show_progress -- [in] "true" to draw a live progress bar (TTY only).

• total -- [in] Backlog size for the bar (jobs this pass will act on).

• ... -- [in] Filter arguments forwarded to _knit_drain_release_next.

int _knit_drain_total()

Count the prepared jobs this drain pass will act on and return the count by nameref. Uses the same state='prepared' (+ optional job / group) filter as _knit_drain_dry_run, capped at --count when set, so it anchors the progress bar to exactly the backlog the loop will release. A single read-only SELECT COUNT(*); it advances no state. Run after any --reconcile / --retry pass so re-queued rows are already counted. Returns 0 when there is nothing to drain.

Parameters:

• __knit_ret -- [out] Name of the variable to receive the count.

• count -- [in] Maximum jobs this run, or "" for no cap.

• job -- [in] Optional job-name filter (empty means any).

• group -- [in] Optional group filter (empty means any).

int _knit_drain_validate_retry_states()

Check that a --retry list names only states drain may re-queue, and abort (knit_fatal) otherwise. Drain retries only the terminal-failed states "killed" and "failed"; the non-terminal "running"/"submitting" are never reclaimed (orphan detection is a separate feature) and "completed" is never redone from a drain. Any other value, or an empty list, is rejected.

Parameters:

• csv -- [in] The comma-separated --retry list (non-empty).

env.sh

Functions

int _knit_env_env_built()

Return 0 when at least one setup instance has a built environment for the given manager, non-zero otherwise. A built environment is a "<nm>-env/<lockfile>" file under a setup instance directory (see _knit_env_setup_before_cb). Used by the update-mode freeze rule: once any environment has been concretized, the manager's version is frozen for reproducibility. A missing setup root reports "none built". (Generalizes what was Spack's own env-built check.)

Parameters:

• name -- [in] Manager name (any spelling).

int _knit_env_is_manager_shorthand()

Return 0 when the token is the "with_<nm>_env", "with_<nm>_env_from", "with_<nm>_specs", or "with_<nm>_specs_from" shorthand of a registered environment manager. The source-time shorthand generator uses it so a KNIT_WITHOUT_SHORTHAND opt-out of such a token is recognised rather than warned about as an unknown token.

Parameters:

• token -- [in] Shorthand token (the part after "@").

int _knit_env_is_registered()

Return 0 when a manager with the given normalized name is in the registry, non-zero otherwise.

Parameters:

• nm -- [in] Normalized (underscore) manager name.

int _knit_env_need_provision()

Return 0 when a manager must be provisioned: either any of the given option refs is non-empty (the user asked for a specific version), or the manager is flagged required (a setup declared its environment, see later milestones).

Parameters:

• name -- [in] Manager name (any spelling).

• ... -- [in] The option refs, positional in declared order (may be empty).

int _knit_env_option_names()

Store, in the caller-named array, the declared bootstrap-option names of a manager (the "name" half of each "name:type" pair), in declared order.

Parameters:

• __knit_ret -- [out] Name of the array to hold the option names.

• nm -- [in] Normalized (underscore) manager name.

int _knit_env_provision_all()

Provision every registered environment manager that is needed, in registration order. Called from the first-bootstrap body (after sqlite/jq so latest-release resolution and provenance metadata work). A no-op when no manager is registered.

Parameters:

• ... -- [in] The expanded bootstrap invocation arguments.

int _knit_env_provision_one()

Provision one manager at bootstrap time when it is needed: read each declared bootstrap option off the invocation arguments (in declared order), decide via _knit_env_need_provision, and on a yes call the provision verb with those refs positionally. A no-op when the manager is not needed.

Parameters:

• nm -- [in] Normalized (underscore) manager name.

• ... -- [in] The expanded bootstrap invocation arguments.

int _knit_env_record_contribution()

Generic recorder shared by every setup environment directive. It (1) validates the A/B mode rule against what the setup has already declared for this manager, (2) appends the contribution to the per-command, per-manager contribution list, and (3) on the FIRST contribution for a manager, installs the build/activation callbacks, declares the __<nm>_manifest__ / __<nm>_lock__ provenance outputs and the --help note, and flags the manager required so bootstrap provisions it.

A setup declares, per manager, EITHER world A (one complete manifest, mode "env") OR world B (an accumulating spec list, mode "specs"); the two cannot be mixed, and world A is exactly one contribution. The caller passes the mode its directive belongs to; a conflicting or duplicate world-A declaration is fatal.

The contribution list _KNIT_CMD_<cmd>_envcontrib_<nm> holds two array elements per contribution — a tag then its payload — so a payload (such as an inline manifest) may carry spaces and newlines without any escaping. The mode is kept in the scalar _KNIT_CMD_<cmd>_envmode_<nm>.

Parameters:

• nm -- [in] Normalized (underscore) manager name.

• mode -- [in] "env" (world A, exclusive) or "specs" (world B, additive).

• tag -- [in] Contribution tag (literal-spec / fn-spec / file-manifest / stdin-manifest / fn-manifest).

• ... -- [in] One payload per remaining argument; each appended as its own contribution with the same tag (so knit_with_<nm>_specs can pass several specs at once).

int _knit_env_register_shorthands()

Register the "@with_<nm>_env", "@with_<nm>_env_from", "@with_<nm>_specs", and "@with_<nm>_specs_from" shorthands for a manager, each forwarding to the directive knit_env_done generated. KNIT_WITHOUT_SHORTHAND and collision rejection are handled by knit_register_shorthand. Called from knit_env_done for a plugin's manager (declared after the shorthand layer exists) and from the source-time shorthand generator for a core manager (declared before it).

Parameters:

• nm -- [in] Normalized (underscore) manager name.

int _knit_env_render_tokens()

Render the "{{parameter-name}}" placeholders in a manifest (or spec) text, replacing each with the value of the named setup parameter. The parameter values are read with knit_get_parameter from the setup's expanded invocation arguments (passed as the trailing arguments). This is the one rendering point shared by the specs sugar, here-doc/stdin manifests, and file manifests.

A token is "{{" + optional whitespace + a parameter name + optional whitespace

• "}}". The name may use hyphens or underscores interchangeably (as everywhere in Knit). Substitution is textual and literal: the value is inserted verbatim, with no recursion over an inserted value and no shell evaluation. Text with no "{{" is returned unchanged by a cheap early check, so the common, non-parameterized case pays almost nothing.

A token whose name does not resolve to a declared parameter of the setup (knit_get_parameter returns non-zero) is almost always a typo; it is fatal, naming the unresolved placeholder and the manager.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the rendered text.

• display -- [in] Manager display name, used only in the fatal message.

• text -- [in] The manifest/spec text that may contain "{{...}}" tokens.

• ... -- [in] The setup's expanded invocation arguments.

int _knit_env_require_context()

Guard shared by the knit_env_* setters and knit_env_done: fatal unless a manager registration is open (knit_register_env_manager was called and knit_env_done has not yet closed it). Returns normally when a context is open.

Parameters:

• fn -- [in] Name of the calling function, used in the error message.

int _knit_env_require_setup_context()

Guard shared by the setup directives (knit_with_<nm>_env / _specs / _env_from / _specs_from): fatal unless a setup registration is currently open. Returns normally when the context is a setup.

Parameters:

• directive -- [in] Full directive name, used in the error message.

int _knit_env_resolve_manifest()

Resolve the setup's contribution list for one manager into the concrete manifest text, at invocation time. World A (a single file / stdin / function manifest) yields the document verbatim (rendered); world B (a spec list) runs any spec-generating functions, collects every spec, and synthesizes one manifest via the manager's specs verb. "{{parameter}}" placeholders are rendered against the setup's invocation arguments throughout.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the rendered manifest text.

• nm -- [in] Normalized (underscore) manager name.

• cmd -- [in] Mangled command name of the executing setup (holds the list).

• ... -- [in] The setup's expanded invocation arguments.

int _knit_env_setup_after_cb()

After-callback installed by _knit_env_with_env, registered AFTER the generic _knit_setup_after_cb so it runs last and its appended block is authoritative when a job sources .activate.sh. It (1) appends the activate verb's lines to .activate.sh so a dependent job re-enters the exact environment, and (2) records the concrete manifest and lockfile as provenance outputs, gzip+base64.

The provenance is emitted with knit_output: after-callbacks run while the command is still on the executing-command stack, so knit_output resolves and type-checks the outputs normally.

The manifest is captured from <nm>-env/<manifest> (the install-placed copy, like Spack's concretized spack.yaml) and falls back to the materialized ${KNIT_SETUP_PREFIX}/<manifest>; the lockfile from <nm>-env/<lockfile>.

Parameters:

• nm -- [in] Normalized (underscore) manager name. Trailing arguments are the setup's runtime arguments and are ignored.

int _knit_env_setup_before_cb()

Before-callback installed by the first environment contribution. Runs as the setup's first step (before-cbs execute in the setup's own shell, so the activated environment persists into the setup body and the after-callbacks). It sources the platform, provisions the manager on demand, resolves the setup's contribution list into a concrete manifest, writes it to ${KNIT_SETUP_PREFIX}/<manifest>, builds the environment into ${KNIT_SETUP_PREFIX}/<nm>-env via the install verb, and activates it in the current shell (by eval-ing the activate verb's lines) so the body builds against the installed packages.

On an install failure (or a spec/manifest function exiting non-zero) it returns non-zero WITHOUT activating: the non-zero status propagates so the setup body never runs and the setup dispatcher removes the half-built directory.

The resolved specs/manifest may contain "{{parameter}}" placeholders; they are rendered against the setup's expanded invocation arguments (the trailing arguments) while the contribution list is resolved, so a spec or manifest can depend on a setup parameter. A file manifest is rendered on read (the file on disk is never modified).

Parameters:

• nm -- [in] Normalized (underscore) manager name.

• ... -- [in] The setup's expanded invocation arguments, used to run spec / manifest functions and resolve "{{parameter}}" placeholders.

int _knit_env_update_all()

Update-mode dispatcher: run _knit_env_update_one for every registered manager, in registration order. Called from the bootstrap update body. Returns 0 when any manager was provisioned or re-provisioned, 1 when none changed.

Parameters:

• raw_name -- [in] Name of the array holding the raw invocation arguments.

int _knit_env_update_one()

Update-mode handler for one manager (the generalization of what was Spack's own update-mode handler). The manager's version anchors reproducibility, so it is frozen once an environment is built:

• Tool not provisioned yet: provision now when a ref was typed or the manager is required (the first-bootstrap condition).

• Already provisioned and a typed ref differs from the stored ref:

  • no environment built anywhere -> re-provision at the effective ref(s) (a typed ref replaces its stored value; an untyped one keeps the stored value, so changing one knob does not move another to latest);

  • an environment is built -> fatal, for reproducibility.

• A typed ref equal to the stored ref, or no typed ref -> no-op.

Typed-ness is read from the raw invocation arguments (an option's value, when typed, is present there); the stored value of each option <opt> is the metadata key __<opt>_ref__ (normalized), which the provision verb must write. The re-provision removes .knit/<name> first; a manager owning extra directories cleans them in its own provision verb.

Parameters:

• nm -- [in] Normalized (underscore) manager name.

• raw_name -- [in] Name of the array holding the raw invocation arguments.

Returns:

0 when the manager was provisioned or re-provisioned, 1 when nothing changed.

int _knit_env_with_env()

Generic implementation of knit_with_<nm>_env: declare that the setup currently being registered needs a <nm> environment described by a complete manifest (world A). The manifest is a file (non-empty first argument, resolved absolute at registration and recorded with knit_bundle_auto_require) or, with no argument, a here-doc / stdin manifest consumed at registration. The stdin form fails fast on an interactive terminal (nothing to read) and rejects an empty manifest.

Valid only inside a setup registration (fatal otherwise). World A is exactly one contribution per manager and mutually exclusive with the world-B spec directives (enforced by the shared recorder).

Parameters:

• nm -- [in] Normalized (underscore) manager name (bound by the wrapper).

• file -- [in] Optional manifest path; if omitted, read from stdin.

int _knit_env_with_env_from()

Generic implementation of knit_with_<nm>env_from: the programmatic twin of knit_with<nm>_env (world A). Instead of a file or an inline document, a user function produces the complete manifest at setup-invocation time. The function is called as <fn> "$@" with the setup's expanded invocation arguments, reads its parameters with knit_get_parameter, and prints the whole manifest (spack.yaml / pyproject.toml) on stdout; the document is then rendered for "{{parameter}}" and materialized.

Valid only inside a setup registration (fatal otherwise). World A is exactly one contribution per manager and mutually exclusive with the world-B spec directives (enforced by the shared recorder). The function name is captured here and resolved when the setup runs; the function must be defined by then.

Parameters:

• nm -- [in] Normalized (underscore) manager name (bound by the wrapper).

• fn -- [in] Name of the manifest-generating function.

int _knit_env_with_specs()

Generic implementation of knit_with_<nm>_specs: the common "just install these

specs" case (world B). Each spec is appended to the setup's accumulating spec list; at build time the manager's specs verb synthesizes one manifest from the whole list, so provenance capture, activation, and auto-provisioning all apply identically. Valid only inside a setup registration; additive with other world-B directives and mutually exclusive with the world-A manifest directives (enforced by the shared recorder).

Parameters:

• nm -- [in] Normalized (underscore) manager name (bound by the wrapper).

• ... -- [in] One or more specs.

int _knit_env_with_specs_from()

Generic implementation of knit_with_<nm>specs_from: like knit_with<nm>specs (world B) but the specs are produced by a user function at setup-invocation time instead of being literal. The function is called as <fn> "$@" with the setup's expanded invocation arguments, reads its parameters with knit_get_parameter, and prints newline-separated specs on stdout; each non-blank line is rendered for "{{parameter}}" and appended to the setup's accumulating spec list. It composes freely with knit_with<nm>specs and other knit_with<nm>_specs_from calls (any order), and is mutually exclusive with the world-A manifest directives.

The function name is captured here and resolved when the setup runs; the function must be defined by then.

Parameters:

• nm -- [in] Normalized (underscore) manager name (bound by the wrapper).

• fn -- [in] Name of the spec-generating function.

Variables

Array _KNIT_ENV_MANAGERS

Registry of registered environment managers, holding each manager's normalized (underscore) name in registration order. Appended by knit_env_done; iterated by the bootstrap provisioning dispatcher. Declared global (-g) so it survives knit.sh being sourced from within a function (as the bats tests do).

frame.sh

Functions

int _knit_progress_bar_counts()

Build the counts/percent text for one bar and return it by nameref. Reads "<c> done <r> run [<f> fail] <rem> left (<total>) <pct>%" where the fail field is omitted when there are no failures and the percent is terminal progress floor(100 * (completed + failed) / total). With --label the text is prefixed by a fixed-width label column (used by the two-bar form). No color is used here.

Parameters:

• __knit_ret -- [out] Name of the variable to receive the counts string.

• completed -- [in] Number of completed items.

• running -- [in] Number of running items.

• failed -- [in] Number of failed/killed items.

• remaining -- [in] Number of not-yet-started items.

• --label -- [in] <text> Optional: prefix with a fixed-width label column.

int _knit_progress_bar_legend()

Build the one-line glyph key drawn under the frame and return it by nameref. The order matches the bar: failed, done, running. With --no-color the glyphs are ASCII (failed and done both render as #, so the legend labels carry the distinction).

Parameters:

• __knit_ret -- [out] Name of the variable to receive the legend string.

• --no-color -- [in] Optional: emit ASCII glyphs and no color escapes.

int _knit_progress_bar_render()

Build the interior fill string for one bar row of exactly <width> display cells and return it by nameref. No frame, no counts, no terminal I/O. Section order is failed | completed | running | remaining(space); whole cells only (no partial blocks). Cell counts use largest-remainder rounding so the runs sum to exactly <width>, and a non-zero section keeps at least one cell while the field has room (a single failed/running job is always visible). Colors are the brand accents via _knit_brand_color; with --no-color the glyphs become ASCII (# for failed/completed, : for running, space for remaining).

Parameters:

• __knit_ret -- [out] Name of the variable to receive the fill string.

• width -- [in] Interior width in cells.

• completed -- [in] Number of completed items.

• running -- [in] Number of running items.

• failed -- [in] Number of failed/killed items.

• remaining -- [in] Number of not-yet-started items.

• --no-color -- [in] Optional: emit ASCII glyphs and no color escapes.

int _knit_progress_clip()

Clip a line to <max> display columns and return it by nameref, counting visible characters only — ANSI SGR escapes are copied through but do not count toward the width. Keeps a long line from wrapping onto a second physical row, which would desync the cursor-up redraw. A reset is appended if any color was still open at the cut.

Parameters:

• __knit_ret -- [out] Name of the variable to receive the clipped line.

• line -- [in] The line to clip (may contain ANSI escapes).

• max -- [in] Maximum display columns.

int _knit_progress_colored()

Return success when the bar should be colorized: stderr is a TTY and NO_COLOR is unset. (The brand/16-color choice is then made by _knit_brand_color.)

int _knit_progress_draw()

Erase the previously drawn progress block and repaint <block> to stderr. Every line is clipped to the current terminal width so none wraps (printed-line count equals physical-row count). The cursor is moved up by the previous block's physical-row count — computed as its logical lines times ceil(prev_width / cur_width), to account for the terminal reflowing already-printed rows when it is made narrower — then everything below is erased (\033[J) before the new block is printed. Re-measuring and clipping each draw means a resize (grow or shrink) repaints at the new size without stacking or tearing.

Parameters:

• block -- [in] The multi-line block to paint (each line without a clear-to-eol).

int _knit_progress_gate()

Return success when a live progress bar should be drawn: stderr is a TTY and KNIT_LOG_LEVEL is at or below "info" (a warning-or-higher, i.e. quieter, run suppresses the bar). Failure otherwise.

int _knit_progress_hbar()

Print a run of <n> horizontal rule glyphs (no newline).

Parameters:

• n -- [in] Number of rule cells.

int _knit_progress_inner_width()

Set the nameref given as the first argument to the interior width for the bar: the current terminal width (via _knit_terminal_width) minus the two frame sides, re-measured each call so the frame tracks the terminal. Falls back to a 40-cell interior when the width is unknown, and never goes below 10.

Parameters:

• __knit_ret -- [out] Name of the variable to receive the interior width.

int _knit_progress_rule_bottom()

Print the frame's bottom rule for an interior width of <inner> cells. Dimmed when color is enabled.

Parameters:

• inner -- [in] Interior width in cells.

int _knit_progress_rule_top()

Print the frame's top rule for an interior width of <inner> cells, with an optional <title> embedded in the rule. The interior between the corners is one leading rule cell, the padded title, then the remaining rule cells, summing to exactly <inner>. Dimmed when color is enabled.

Parameters:

• inner -- [in] Interior width in cells.

• title -- [in] Optional title text.

int _knit_progress_side()

Print one vertical frame side glyph. Dimmed when color is enabled.

Variables

String _KNIT_PROGRESS_BL

Bottom-left frame corner glyph (U+2514) for the progress bar.

String _KNIT_PROGRESS_BR

Bottom-right frame corner glyph (U+2518) for the progress bar.

String _KNIT_PROGRESS_FULL

Progress bar

A full-width, framed progress bar for long, batch-of-jobs commands (submit drain, ax optimize). The bar interior fills left to right with one run of block cells per section, in the order failed | completed | running | remaining, with the remaining portion left as whitespace inside the frame. Failed sits first so a failure is the first thing the eye lands on. Section colors are knit brand accents (magenta / dark purple / light purple) with a 16-color fallback.

The surface is split into pure string renderers (no terminal I/O, below) and a TTY draw layer (knit_progress_bar / knit_progress_bar2, added separately). The renderers take an explicit width and an explicit --no-color so they are deterministic and unit-testable; the draw layer owns the terminal.

Full block glyph (U+2588) for the completed and failed progress-bar sections; color distinguishes the two.

String _KNIT_PROGRESS_H

Horizontal frame rule glyph (U+2500) for the progress bar.

String _KNIT_PROGRESS_LINES

Number of logical lines the last progress block printed (0 before the first paint). Drives the cursor-up on the next redraw.

String _KNIT_PROGRESS_RUN

Medium-shade block glyph (U+2592) for the running progress-bar section, so it is distinct from the completed section even without color.

String _KNIT_PROGRESS_TL

Progress bar — TTY draw layer

knit_progress_bar / knit_progress_bar2 draw a framed bar live to stderr; they are a no-op unless stderr is a TTY and KNIT_LOG_LEVEL is at or below "info" (a quiet run gets no bar). On each call the terminal width is re-measured and the whole block (titled frame + interior bar row(s) + legend/counts) is rebuilt and repainted with a reflow-aware redraw, so a terminal resize — grow or shrink — repaints cleanly rather than stacking. knit_progress_bar_finish resets the redraw state and drops to a fresh line so a following knit_info prints below the frame.

Top-left frame corner glyph (U+250C) for the progress bar.

String _KNIT_PROGRESS_TR

Top-right frame corner glyph (U+2510) for the progress bar.

String _KNIT_PROGRESS_V

Vertical frame side glyph (U+2502) for the progress bar.

String _KNIT_PROGRESS_WIDTH

Terminal width the last progress block was printed at, used to compute how many physical rows it occupies now if the terminal has since been resized.

github.sh

Functions

int _knit_github_resolve_commit_light()

Resolve a ref (tag, branch, or commit) to its commit SHA via the GitHub API, returning it through a nameref and WITHOUT jq. The plugin import path runs at source time, before bootstrap, when the provisioned jq (knit_github_resolve_commit's parser) does not exist yet; this reads the top-level "sha" field (the commit's own, first in the commits response) with a plain grep instead. Returning by nameref (not stdout) also keeps a failure fatal in the caller's shell rather than lost inside a command substitution.

Parameters:

• commit_out -- [out] Nameref receiving the resolved 40-hex commit SHA.

• org -- [in] Repository owner (organization or user).

• repo -- [in] Repository name under that owner.

• ref -- [in] Tag, branch, or commit SHA to resolve.

Returns:

0 and sets the output when resolved; 1 (output untouched) otherwise.

global.sh

Functions

int _knit_brand_color()

Set the nameref given as the first argument to the escape sequence for a brand color, resolved to truecolor when the terminal supports it (see _knit_truecolor) or the nearest 16-color ANSI otherwise. The brand key is one of: magenta, purple, purple_dark.

Parameters:

• __knit_ret -- [out] Name of the variable to receive the escape sequence.

• key -- [in] Brand color key (magenta | purple | purple_dark).

int _knit_stdout_is_terminal()

Return success when standard output is a terminal. Factored into its own function so callers that colorize or wrap output based on the terminal (e.g. "--help" highlighting and describe) can stub it in tests to force the terminal path on or off deterministically.

int _knit_terminal_width()

Set the nameref given as the first argument to the width of the terminal (in columns), or to "0" when the width cannot be used for wrapping — i.e. when standard output is not a terminal or "stty size" yields no usable value. Callers treat a "0" result as "no wrapping". Factored into its own function (mirroring _knit_stdout_is_terminal) so tests can stub it to force a width deterministically.

The result is returned by nameref rather than printed for capture with command substitution: the terminal check (_knit_stdout_is_terminal, i.e. "[[ -t 1 ]]") must be evaluated in the caller's context, where file descriptor 1 is the real help destination. A command substitution would replace fd 1 with a pipe and so always report "not a terminal", disabling wrapping even on a real terminal.

Parameters:

• __knit_ret -- [out] Name of the variable to receive the width (or "0").

int _knit_truecolor()

Return success (0) when the terminal supports 24-bit truecolor, failure (1) otherwise. COLORTERM (truecolor/24bit) is the explicit signal, but many terminals that support truecolor leave COLORTERM empty, so a modern TERM is also accepted. KNIT_FORCE_16COLOR forces failure and KNIT_TRUECOLOR forces success (both for testing the two code paths deterministically).

Variables

AssociativeArray _KNIT_BRAND_FALLBACK

Nearest basic 16-color ANSI escape for each brand color, used when the terminal does not support 24-bit truecolor. Chosen so colorized output stays legible and the brand colors remain distinguishable from one another.

AssociativeArray _KNIT_BRAND_TRUECOLOR

Knit brand accent colors as 24-bit truecolor escape sequences, taken from the landing-page palette (docs/pygments-knit/knit_pygments.py, mirroring web/styles.css). Used where knit wants its own identity rather than one of the basic ANSI colors (e.g. the progress bar sections). Keys: magenta #b83a83 (--knit-magenta, red-magenta) purple #9e7fd4 (--knit-purple, light purple) purple_dark #5f4c7f (dark purple; brand purple shifted darker, same hue)

AssociativeArray _KNIT_COLORS

Associative array mapping color and style names to their ANSI escape sequences. Use these to colorize terminal output. The "reset" entry clears all active attributes.

Foreground colors: black, red, green, yellow, blue, magenta, cyan, white, bright_black, bright_red, bright_green, bright_yellow, bright_blue, bright_magenta, bright_cyan, bright_white. Background colors: bg_black, bg_red, bg_green, bg_yellow, bg_blue, bg_magenta, bg_cyan, bg_white, bg_bright_black, bg_bright_red, bg_bright_green, bg_bright_yellow, bg_bright_blue, bg_bright_magenta, bg_bright_cyan, bg_bright_white. Styles: bold, dim, italic, underline, blink, reverse, hidden, strikethrough.

String _KNIT_IS_BOOTSTRAPPING

Set to "true" when the first argument passed to the experiment script is "bootstrap", i.e. when the user is running the bootstrap command. Functions that require a bootstrapped experiment use this to distinguish between a legitimate pre-bootstrap invocation (during bootstrap itself) and an erroneous one (calling a DB-backed command before bootstrap has run).

Array _KNIT_ON_FATAL

List of callback command strings run by knit_fatal just before it exits. Each entry is a single command, pre-escaped with printf q (as knit_run_before / knit_run_after store their callbacks), so it can be re-run with eval. A callback registers via knit_on_fatal.

The list is empty in almost every context (the login side, bootstrap, any ordinary command), so knit_fatal's drain is a no-op there. It exists so a compute-side job can record a terminal "failed" state when its body or a before-callback aborts with knit_fatal, which exits the process before the normal after-callback (that would record the state) can run.

Array _KNIT_SCRIPT_ARGV

The positional parameters the experiment script was invoked with, captured at the very top of the concatenated knit.sh (which is sourced with no arguments of its own, so "$@" here is the experiment's own argv). Plugin import and bootstrap materialization consult this to tell the current command apart (for example "bootstrap", which must also materialize into .knit/, from an ordinary run) without re-parsing the command line.

String _KNIT_SCRIPT_PATH

Absolute, resolved path to the experiment script that sourced knit.sh. Unlike KNIT_SCRIPT_NAME (a base name for messages), this is a runnable path, captured at load time before any directory change, so commands that re-execute the experiment (for example "submit drain" releasing jobs in a loop) can invoke it reliably regardless of the current working directory.

job.sh

Functions

int _knit_command_is_job()

Test whether a command is a job, i.e. it was registered with knit_register_job. A job's row carries a "state" column written from its callbacks and signal traps (running / killed / completed), so a directive that would suppress that row can consult this to reject a job. Reads the command kind from the KNIT_CMD<cmd>_type field (see knit_register).

Parameters:

• cmd -- [in] Command (mangled name) to test.

Returns:

0 if the command is a job, 1 otherwise.

int _knit_declare_submit_options()

Declare the scheduler options shared by the submit and prepare dispatchers, so the two surfaces cannot drift. Must be called between knit_register and knit_done while registering either dispatcher. It declares the optional submission parameters (setup, name, job-name, account, project, queue, nodes, walltime, gpus-per-node) and the free-form --group label, plus the "job" dispatch marker and the "Jobs" subcommand title.

Each dispatcher declares the rest itself: submit adds the submit-only --wait flag and owns the "jobs" table and its outputs; prepare declares neither a table nor outputs (it records under the submit identity, see _knit_prepare_build).

int _knit_job_after_cb()

After-callback installed on every submit subcommand by knit_register_job. Records the job's terminal state from the body's exit status: "failed" when the body returned non-zero, "completed" when it returned zero. A signal death (a cancel, an out-of-memory or walltime kill) routes through the TERM/USR1 trap (_knit_job_set_state "killed"), which runs instead of this callback, so "killed" takes precedence over "failed".

int _knit_job_before_cb()

Before-callback installed on every setup subcommand by knit_register_job. Verifies that KNIT_JOB_PREFIX is set, ensuring the job was invoked through knit submit rather than called directly, installs the pre-termination signal handler, marks the job "running", registers the on-fatal state recorder, and records the allocated hostnames.

It does NOT source the setup environment: the jobscript already sources the setup's .activate.sh before it re-enters the experiment (see _knit_sched_write_jobscript), and the exports survive the exec, so the environment is already active here. Sourcing again would re-apply the composable env_append/env_prepend lines and double their entries.

int _knit_job_fatal_record()

On-fatal callback (registered via knit_on_fatal in _knit_job_before_cb) that records the running job as "failed" when it aborts through knit_fatal. A knit_fatal in the job body or in a before-callback (e.g. a missing resource or setup) exits the process before the normal after-callback can record a terminal state, which would otherwise strand the row at "running". Recording "failed" here mirrors _knit_job_after_cb's "non-zero body => failed" rule.

It records "failed" ONLY when the row is still in a non-terminal state (running/submitted): a job that already recorded a terminal state — most importantly "killed" via the signal trap — keeps it, so a kill followed by a fatal cannot downgrade "killed" to "failed". Best-effort and guarded like _knit_job_set_state, so it never itself aborts the already-aborting process.

int _knit_job_killed_trap()

Signal handler installed while a job runs on the compute node. Schedulers warn a job before killing it: Slurm can send a chosen signal a configurable time before the walltime limit (requested via --signal in the batch directives) and sends SIGTERM before SIGKILL; PBS likewise sends SIGTERM before SIGKILL. Flux cancels a batch job by shutting down its instance, which sends SIGHUP to the batch initial program. This handler records the job as "killed" so its row does not stay stuck at "running", then exits so the after-callback (which would mark it "completed") does not run.

int _knit_job_record_hostnames()

Record the nodes the running job was allocated into its jobs-table row. Called on the compute side, once the job is running, where the scheduler has populated the node allocation and the experiment's .knit is shared over the parallel file system (so the row inserted by knit submit on the login node can be updated in place). The stored value is the deduplicated, comma-separated host list, i.e. the output of knit_job_hostnames --separator ,. The row id is the job UUID, i.e. the basename of KNIT_JOB_PREFIX. Best-effort: like state tracking, a failure is downgraded to a warning so it never takes down the job.

int _knit_job_set_state()

Update the lifecycle state of the running job's jobs-table row. Called on the compute side, where the experiment's .knit is shared over the parallel file system, so the row inserted by knit submit on the login node can be updated in place. The row id is the job UUID, i.e. the basename of KNIT_JOB_PREFIX (the job directory). Best-effort: status tracking must never take down the job itself, so a failure is downgraded to a warning.

Parameters:

• state -- [in] New state value (e.g. running, completed, failed, killed).

int _knit_prepare_build()

Build a submission without dispatching it: the shared first phase of submit and prepare. It validates the request (job known, usability, setup type, argument check), creates the job directory <job-root>/<uuid> (and any --name alias), records the jobs row in the target lifecycle state with an empty native-cmd, emits the setup "used_by" provenance edge, resolves the scheduler options and backend, generates the batch script .job.sh, and freezes the resolved backend and options into a .submit metadata file so a later release (_knit_submit_dispatch) is deterministic and needs no re-resolution.

The four output arguments receive the job UUID, its directory, the --name alias symlink path (empty when no --name), and the job name. Callers MUST pass output variables whose names do not clash with this function's internal locals (uuid/jobdir/alias_link/job_name) — see the nameref-shadow-collision rule.

Output namerefs come first, then the target state, then the submit CLI args.

Parameters:

• __knit_ret1 -- [out] Out: the job UUID (the jobs row id).

• __knit_ret2 -- [out] Out: the job directory <job-root>/<uuid>.

• __knit_ret3 -- [out] Out: the --name alias symlink path (empty when none).

• __knit_ret4 -- [out] Out: the job name (the token after --).

• target_state -- [in] Lifecycle state to record ("submitted" or "prepared").

• ... -- [in] The submit CLI arguments (everything, including -- job args).

int _knit_submit()

Entry point for the submit CLI command. A submission is built and then dispatched: _knit_prepare_build validates the request, creates the job directory, records the jobs row, generates the batch script, and freezes the resolved backend and options into a .submit metadata file; _knit_submit_dispatch then issues the scheduler command. submit runs both back to back, so its behaviour is a build immediately followed by a release. prepare runs only the build (see prepare.sh).

Usage:

./exp.sh submit [--setup <setup-name>] [--name <alias>] [sched-args...] \
    -- job-name [args...]

int _knit_submit_cleanup_rejected()

Undo the eager bookkeeping of a submission the scheduler rejected, so a job that never ran leaves no trace. knit submit records the jobs row (and, for a job with a setup, a "used_by" provenance edge) and creates the job directory before dispatching, because a blocking --wait job needs the row to exist so the compute side can transition it. When the submission command itself fails, none of that should survive: this removes the jobs row, any provenance edge pointing at it, the --name alias symlink (if any), and the job directory. Each step is best-effort (the provenance table may not exist for a setup-less job).

Parameters:

• uuid -- [in] The submission's job UUID (its row id and edge target_id).

• jobdir -- [in] The job directory to remove.

• alias_link -- [in] Path to the --name alias symlink, or empty when none.

int _knit_submit_dispatch()

Release a built submission to the scheduler: the second phase of submit, and the whole of submit prepared / submit next. It reads the resolved backend and options from the job directory's .submit metadata (frozen by _knit_prepare_build), builds the (non-blocking) scheduler submission command, records it as the row's native-cmd, advances the row to "submitted", issues the command, and handles a scheduler rejection (removing the never-run job, exactly as a direct submit does). On success it records the backend job id in .job.id, prints the job UUID to stdout, and — when the resolved "wait" option is true — blocks until the job reaches a terminal state and returns non-zero if that state is "failed" or "killed" (via _knit_job_wait_one), so a blocking submit honors the --wait contract of returning the job's exit code.

The submission is always issued non-blocking so the UUID can be printed as soon as the scheduler accepts the job, before any wait blocks; a caller who captures stdout still receives exactly the one UUID line.

Parameters:

• uuid -- [in] The job UUID (the jobs row id).

• jobdir -- [in] The job directory holding .submit and .job.sh.

• job_name -- [in] The job name (for log and error messages).

• alias_link -- [in] The --name alias symlink path, or empty when none (removed on rejection).

• wait_override -- [in] Optional: "true"/"false" to override the frozen "wait" option at release time (empty keeps the value frozen at build time). submit freezes --wait into .submit, so it passes nothing; submit prepared/submit next accept --wait at release and pass it here (prepared jobs freeze "wait" as false, since prepare has no --wait).

int _knit_submit_meta_read()

Restore the scheduler backend and resolved options a submission was built with from its .submit metadata file (see _knit_submit_meta_write). The backend name is written to the first named variable; each recorded option is written into the named associative array, which the caller declares.

Parameters:

• file -- [in] Path of the .submit file to read.

• out_backend -- [out] Name of the variable to receive the backend name.

• arr_name -- [out] Name of the associative array to populate with the options.

int _knit_submit_meta_write()

Freeze the resolved scheduler backend and options of a built submission into its .submit metadata file, so a later release reconstructs them without re-resolving. The format is one "key=value" per line: a single "backend=<name>" line, then one "opt:<key>=<value>" line per resolved option. A value is the rest of its line, so it may contain "=" or spaces but not a newline (no scheduler option does).

Parameters:

• file -- [in] Path of the .submit file to write.

• backend -- [in] Scheduler backend name.

• arr_name -- [in] Name of the resolved-options associative array to persist.

Variables

AssociativeArray _KNIT_JOBS

Associative array mapping registered job names to 1. Used to validate that a job names passed to knit submit is known.

String _KNIT_JOBS_TABLE

Name of the table recording every job and its lifecycle state. The row id is the job UUID; the "state" column moves submitted -> running -> completed, or -> killed when the scheduler terminates a job before it finishes. A submission the scheduler rejects never becomes a job and leaves no row at all (see _knit_submit_cleanup_rejected).

job_cli.sh

Functions

int _knit_job_cancel()

Cancel a job that is still running by asking its scheduler backend to terminate it, then recording the job's row as "killed". The backend job id is read from the job directory's .job.id (as job wait does) and handed to the backend's cancel primitive (local -> kill, slurm -> scancel, pbs -> qdel).

A prepared job (state "prepared") was never dispatched, so there is no scheduler job to cancel: it is simply removed (row, job directory, and any --name alias), the same teardown as a rejected submission. That branch runs before the .job.id lookup, since a prepared job has no launcher id.

Cancelling a job that has already reached a terminal state (completed or killed) is a no-op with an informational message. An unknown id is fatal, and a still-running job whose .job.id is missing errors rather than reporting a cancellation that did not happen.

The compute side may also record "killed" via its pre-termination signal trap, but the row is updated here as well so it stays consistent for backends or races where that handler never runs (e.g. a hard kill, or a remote node whose write has not yet landed).

int _knit_job_dir()

Print the working directory of a job given its UUID. Every job lives in one job root, <job-root>/<id>, regardless of which setup (if any) it used: the job root is resolved from bootstrap metadata (job_path) against the experiment root (see _knit_job_root).

Parameters:

• id -- [in] Job UUID.

int _knit_job_discover()

Register the job subcommands (status / list / wait / cancel / resubmit / show, and show's own stdout / stderr / script). Called lazily by the framework the first time a job subcommand is resolved, listed in "--help", or walked by "describe". "job:show" is registered before its own children so the parent exists first.

Parameters:

• parent -- [in] The parent command's display name (unused).

int _knit_job_follow_file()

Stream one of a job's output files live, like tail -f, and stop cleanly once the job finishes. Shared by job show stdout --follow and job show stderr --follow. An unknown id is fatal (as for the non-following form).

Behaviour depends on the job's lifecycle state:

• Already terminal: nothing more will be written, so print the file once and return (a missing file is fatal, exactly as job show ${label}).

• Still running: wait for the file to appear (it is created on the job's first write), then follow it from the beginning. A bare tail -f would hang forever after the job exits, so it runs in the background and is stopped once the job's jobs-table row reaches a terminal state. That state is polled rather than tied to a local pid so following works for remote scheduler backends where the job runs on another node. If the job finishes without ever producing the file, that is fatal (as for the non-following form).

Parameters:

• id -- [in] Job UUID.

• filename -- [in] Name of the file within the job directory (".stdout"/".stderr").

• label -- [in] Human-readable name of the file for error messages.

int _knit_job_in_clause()

Build a SQL "<column> IN (...)" condition from a comma-separated list of values, each escaped as a single-quoted literal. Empty entries are dropped; nothing is printed when the list has no non-empty value, so the caller can test for an empty result and skip the condition entirely.

Parameters:

• column -- [in] Column name to match (interpolated verbatim, so a trusted name).

• csv -- [in] Comma-separated list of values.

int _knit_job_list()

Print a table of submitted jobs (id, job name, state), one row per job.

--status adds an equality filter on the lifecycle state. The setup filter is the OR of two optional parts: --no-setup matches jobs with no setup (setup column NULL or empty), and --setup takes a comma-separated list of setup paths matched with SQL IN. Given both, a job qualifies if it has no setup OR its setup is in the list, e.g. "--no-setup --setup a,b,c". When neither is given no setup filter is applied. --types takes a comma-separated list of job types (the job name recorded in the "job" column) matched with SQL IN. All three filters (status, setup, types) are AND-ed together.

By default the result is rendered with aligned columns and a header row. With --json the same filtered query is emitted as a JSON array (one object per job, or [] when nothing matches), using sqlite's -json output; the filters apply identically in both modes.

int _knit_job_reconcile()

Entry point for the job reconcile CLI command. For each job whose row is in state running or submitting, ask the scheduler for its true fate and correct the row to the right terminal state when the scheduler confirms the job has stopped. A job the scheduler still reports as active is left unchanged.

With no selectors the command walks all active jobs. The id / group / type selectors AND-ed together narrow the scope. An empty match is a no-op (not an error). With --dry-run each would-be change is reported but no write is made.

Usage:

./exp.sh job reconcile [--id <uuid>] [--group <g>] [--type <job>] [--dry-run]

int _knit_job_reconcile_apply()

Shared worker behind job reconcile and submit drain --reconcile. For each job row in state running or submitting that matches the optional selectors, ask the scheduler for the job's true state and move the row to the correct terminal state when the scheduler confirms the job has stopped. A running verdict leaves the row unchanged. An unknown verdict (the scheduler has no record of a job the row calls active) is mapped to killed.

The .job.id-less rule: when the job directory has no launcher id file, a submitting row without an explicit --id is skipped with a warning (a live drainer may be mid-dispatch); any other active row without a launcher id (including a running row or a submitting row addressed by explicit --id) is treated as unknown, which maps to killed.

Each changed row is printed to stdout as "<id> <new_state>". Under dry-run the same output is produced but no write is made. A summary count goes to stderr. Reconciling nothing is not an error.

Parameters:

• dry_run -- [in] "true" to report without writing.

• sel_id -- [in] Optional job UUID to match (empty means any).

• sel_group -- [in] Optional group to match (empty means any).

• sel_type -- [in] Optional job type to match (empty means any).

int _knit_job_reconcile_set_state()

Update a job row to a new state using a conditional SQL UPDATE that only matches rows still in an active state (running or submitting). This guard prevents a reconcile write from overwriting a terminal state that the job's own kill trap wrote a moment earlier: whichever writer runs second finds the row already left active states and makes no change.

Route through _knit_sqlite3_write (the flock serializer) so the conditional check and the write are atomic with respect to other concurrent writers.

Parameters:

• id -- [in] Job UUID.

• new_state -- [in] Terminal state to record (completed, failed, or killed).

int _knit_job_reconstruct_args_from_db_row()

Append CLI arguments reconstructing a command's recorded parameters from one row of a table, so a past invocation can be replayed. Iteration is driven by the command's declared schema (not the raw table columns) so that only genuine parameters are emitted and outputs are ignored: each required or optional parameter with a non-empty recorded value is appended as "--name value", and each flag recorded as "true" is appended as a bare "--name". A column named in the space-separated skip list is left out (e.g. "setup", handled specially by the caller). A parameter whose column is absent from the table is skipped, so a job whose per-job table predates a newly added parameter still replays.

Parameters:

• out_name -- [out] Name of the array variable to append the reconstructed args to.

• cmd -- [in] Mangled command whose schema drives the reconstruction.

• table -- [in] Table holding the recorded row.

• id -- [in] Value of the row's "id" column.

• skip -- [in] Space-separated normalized names to omit.

int _knit_job_reset()

Entry point for the job reset CLI command. Re-queue terminal-failed jobs (state "killed" or "failed") back to "prepared" so a later submit drain retries them. The source states default to "killed,failed" and are restricted to those two (plus "completed" under --force) by _knit_job_reset_validate_states; the non-terminal "running"/"submitting" are never touched (orphan reclaim is a separate feature). The id / group / type selectors narrow which jobs match (all AND-ed), and --max-retries caps how many times a repeatedly-failing job is retried. The actual re-queue is done by the shared worker _knit_job_reset_apply.

Usage:

./exp.sh job reset [--id <uuid>] [--group <g>] [--type <job>] \
    [--state killed,failed] [--max-retries <n>] [--force]

int _knit_job_reset_apply()

Re-queue every job matching the selectors whose state is in states_csv back to "prepared" for a retry (see _knit_job_reset_one). This is the shared worker behind job reset and submit drain --retry, so the two surfaces cannot drift. The selectors are AND-ed: states_csv (a validated non-empty list of source states), plus an optional id / group / job-type equality. A job whose "retries" count already reached max_retries is left terminal and warned about, so a poison job cannot be retried without bound; an empty max_retries means no cap. Each re-queued id is printed to stdout (in id order); a one-line count goes to stderr. Resetting nothing is not an error (idempotent): a scope with no matching terminal job simply reports zero.

Parameters:

• states_csv -- [in] Comma-separated source states to reset (already validated).

• max_retries -- [in] Retry cap, or "" for no cap.

• sel_id -- [in] Optional job UUID to match (empty means any).

• sel_group -- [in] Optional group to match (empty means any).

• sel_type -- [in] Optional job type (the "job" column) to match (empty means any).

int _knit_job_reset_one()

Re-queue a single job row for a retry: move it back to state "prepared" and clear the artifacts of its previous run, so the next submit drain claims and re-runs it from the same recorded inputs. The durable folder, the batch script, and the recorded arguments are preserved (prepare wrote them and they do not change); only the run-specific fields are cleared, restoring the row to the shape a freshly-prepared job has: "state" -> prepared, "hostnames" and "native-cmd" emptied, and the "retries" counter incremented. The stale backend launcher id (<jobdir>/.job.id) is removed so a re-dispatch records a fresh one. The job keeps its id and its provenance; no new row or edge is created.

Parameters:

• id -- [in] The job UUID (its jobs-row id).

• current -- [in] The job's current "retries" count (NULL coalesced to 0 by the caller), bumped by one here.

int _knit_job_reset_validate_states()

Check that a comma-separated --state list names only states that job reset may re-queue, and abort (knit_fatal) otherwise. The terminal-failed states "killed" and "failed" are always allowed; "completed" is allowed only when the caller passed --force (a deliberate redo). Any other state (including the non-terminal "running"/"submitting"/"prepared"/"submitted", which reset never touches) or an unknown value is rejected, as is an empty list.

Parameters:

• csv -- [in] The comma-separated --state list.

• force -- [in] "true" when --force was given (allows "completed").

int _knit_job_resubmit()

Re-submit an existing job by replaying what was recorded for it. The jobs row supplies the setup path, the job name (the "job" output column, used as the token after --) and the submission options; the per-job <jobname> table supplies the arguments passed to the job itself. These are reconstructed into a fresh knit submit --setup <setup> [options] -- <job> [args] invocation, which mints a new job UUID — the old id is never reused.

The submission options and job arguments are rebuilt by _knit_job_reconstruct_args_from_db_row from each command's declared schema, so bookkeeping columns (id, state, and the "job" name itself) and job outputs are left out; the setup path is skipped there too and passed as --setup instead. A job that was submitted but never ran has no per-job table, so it is resubmitted with its submission options only. An unknown id is a fatal error.

int _knit_job_show()

Show everything recorded about one job. Two rows describe a job: the submission row in the jobs table (the scheduler options passed to knit submit plus the lifecycle state), and, once the job has actually run, the row it recorded in its own per-job table (the arguments passed to the job after --). The per-job table is named after the job, read from the submission row's "job" column.

Text mode prints two labelled, column-aligned sections (Submission and Parameters). --json emits { "submission": <row>, "parameters": <row> }, each a single object (or null when absent), built from sqlite's -json output merged with the bundled jq. The per-job table exists only after the job has run at least once; until then the parameters section is empty (null in JSON).

An unknown id is a fatal error.

int _knit_job_show_file()

Print the contents of one of the files in a job's working directory. Shared by the job show stdout, job show stderr and job show script subcommands. The job must exist in the jobs table (an unknown id is fatal), and the requested file must be present in the job's working directory (a missing file is fatal — for the output streams it usually means the job has not produced that output yet).

Parameters:

• id -- [in] Job UUID.

• filename -- [in] Name of the file within the job directory (e.g. ".stdout", ".stderr", ".job.sh").

• label -- [in] Human-readable name of the file for error messages (e.g. "stdout", "stderr", "script").

int _knit_job_show_script()

Print the batch script knit generated for a job (the .job.sh file in the job's working directory). An unknown id or an absent file is a fatal error.

int _knit_job_show_stderr()

Print the standard error a job captured while running (the .stderr file in the job's working directory). An unknown id or an absent file is a fatal error. With --follow, stream the output live and stop once the job finishes (see _knit_job_follow_file).

int _knit_job_show_stdout()

Print the standard output a job captured while running (the .stdout file in the job's working directory). An unknown id or an absent file is a fatal error. With --follow, stream the output live and stop once the job finishes (see _knit_job_follow_file).

int _knit_job_state_is_terminal()

Succeed (return 0) if the job's recorded lifecycle state is terminal (completed, failed, or killed), meaning no more output will be written. Any other state (including an unknown id, which yields an empty result) returns non-zero. Used by the follow loop as a backend-agnostic "job is done" signal.

Parameters:

• id -- [in] Job UUID.

int _knit_job_status()

int _knit_job_wait()

Block until one or more jobs reach a terminal state, printing each job's terminal lifecycle state (one per line, in the order given).

The --id argument accepts a comma-separated list of job UUIDs. Each job is waited for in turn via _knit_job_wait_one. The exit code aggregates the single-job convention: it is non-zero if ANY job ended "failed" or "killed", zero otherwise. A missing/unknown id anywhere in the list is a fatal error. A single id behaves exactly as waiting for that one job.

int _knit_job_wait_one()

Block until a single job finishes, then print its terminal lifecycle state; a "killed" or "failed" state yields a non-zero exit so callers can detect failure.

Rather than busy-polling the database, this blocks on the scheduler itself: once the job's row is not already terminal, it looks up the backend job id (.job.id in the job directory) and calls the backend's native wait (_knit_sched_wait), which returns when the scheduler stops running the job. The job's terminal state (completed/failed/killed) is recorded by its compute-side callbacks over the shared filesystem; that write is read back afterwards, with a short reconciliation window since it may land a moment after the scheduler reports the job gone. An unknown id is a fatal error rather than an endless wait.

Parameters:

• id -- [in] The job UUID to wait for.

launch.sh

Functions

int _knit_launch_backend()

Resolve which launcher backend to use, by precedence:

per-run --launcher -> concrete launcher metadata -> KNIT_PROVIDED_LAUNCHER (a setup's frozen contract) -> none

Every launcher value is frozen ahead of the run — no run-time detection. The launcher metadata is the launcher integrated into the machine, resolved once at bootstrap from the profile or bootstrap-time detection; it is preferred over a setup's contract because a site's launcher cooperates with the resource manager as intended (§3 of the design). A "<unknown>" launcher (detection found no MPI at bootstrap, no profile) or an explicit "none" (the machine was told it offers no integrated launcher, via bootstrap --launcher none or a profile launcher.type of "none") is skipped so a providing setup's frozen KNIT_PROVIDED_LAUNCHER — set by knit_provides_launcher when the setup built or module-loaded its own MPI — is used instead. When nothing is configured, the "none" backend runs the app as a single rank-0 process. This mirrors how _knit_sched_backend degrades an undetected scheduler to the local background-process backend. Note that a run-time --launcher of "none" (the override argument, tier 1) is the opposite: an explicit, terminal choice of the "none" backend that does NOT fall through to a setup contract. Returns one of "none", "openmpi", "mpich", "pals", "slurm", "pbs", "flux".

Parameters:

• __knit_ret -- [out] Name of the variable to hold the resolved backend name.

• override -- [in] Optional explicit launcher name (a per-run --launcher value); empty to fall back to the metadata then the setup contract.

int _knit_launch_bind_value()

The launcher layer is the app-side mirror of the scheduler layer (src/sched.sh). Where the scheduler places a job on an allocation, the launcher places the ranks of an app across the nodes of that allocation (MPI-style). A launcher is a separate axis from the scheduler: a Slurm job may run its ranks under OpenMPI's mpirun, under srun, etc. Each launcher backend lives in its own src/launch_<name>.sh with the same function set, dispatched here by name.

Backends:

• none — run the worker directly, no launcher: a single rank-0 process.

• openmpi — mpirun (Open MPI)

• mpich — mpiexec/Hydra (MPICH)

• slurm — srun (scheduler-integrated)

• pbs — the PBS mpiexec wrapper

• pals — mpiexec (HPE Cray PALS)

• flux — flux run (scheduler-integrated)

Translate knit's normalized --bind vocabulary into the spelling a given backend expects. knit defines a small set of portable binding units — none, core, socket, numa, thread — and each backend family renders them differently: Slurm's --cpu-bind uses the plural/locality-domain spellings (cores, sockets, ldoms, threads); PALS's --cpu-bind uses the singular spellings (core, socket, numa, thread); the OpenMPI/Hydra family (openmpi, mpich, pbs) matches PALS except that a hardware thread is spelled hwthread. Flux's -o cpu-affinity is a distribution policy (off | per-task | map:LIST), not a granularity, so all four levels map to per-task and none maps to off. An unrecognized value is not an error — it is passed through verbatim (with a warning) so a user can still reach a launcher-specific binding this vocabulary does not cover. Returns the translated value.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the translated value.

• backend -- [in] Launcher backend name ("openmpi", "mpich", "slurm", "pals", "flux").

• value -- [in] The knit --bind value to translate.

int _knit_launch_cmdline()

Dispatch to the configured backend's command-line builder, which fills a caller-provided array (passed by name) with the launcher argument vector (the launcher executable and its placement flags) for the resolved placement options. The "none" backend has no launcher, so it leaves the array empty.

Parameters:

• backend -- [in] Launcher backend name ("none", "openmpi", "mpich", ...).

• opts_name -- [in] Name of the resolved placement-options associative array (keys: procs, procs-per-node, hostnames, launcher-args).

• argv_name -- [in] Name of the array to fill with the launcher argument vector.

int _knit_launch_exec()

Dispatch to the configured backend's exec function, which runs the launcher with the translated placement flags followed by the worker command, and returns the launcher's exit status. The worker command is everything after a literal "--". The "none" backend runs the worker command directly, with no launcher.

Parameters:

• backend -- [in] Launcher backend name ("none", "openmpi", "mpich", ...).

• arr_name -- [in] Name of the resolved placement-options associative array.

• -- -- [in] Literal separator.

• ... -- [in] The worker command and its arguments.

launch_flux.sh

Functions

int _knit_launch_flux_cmdline()

The "flux" launcher backend places an app's ranks with Flux's flux run. Like the slurm/pbs backends — and unlike the MPI-native openmpi/mpich/pals backends — it is scheduler-integrated: flux run places tasks in the surrounding Flux instance directly. It is auto-detectable when a Flux instance owns the shell (FLUX_URI is set). flux run is task-based, which matches knit's normalized placement options directly:

--procs N -> -n N (--ntasks) --procs-per-node M -> -N <nnodes> (nnodes = ceil(procs / M)) --cpus-per-proc N -> -c N (--cores-per-task, native) --gpus-per-proc N -> -g N (--gpus-per-task, native) --hostnames h0,h1 -> --requires=host:h0,h1 --bind V -> -o cpu-affinity=<v> (V normalized by _knit_launch_bind_value) --gpu-bind V -> -o gpu-affinity=<V> (value passed through verbatim) --launcher-args … -> appended verbatim after the placement flags

flux run splits its placement flags into two families that must not be mixed: per-task (-n/-c/-g) and per-resource (--tasks-per-node/--gpus-per-node). knit keeps the per-task family so cpus-per-proc and gpus-per-proc stay native, and expresses procs-per-node as a node count (-N) instead of --tasks-per-node: with -n tasks over -N nodes, flux run fills the nodes evenly, so the ranks per node equal procs-per-node. nnodes is ceil(procs / procs-per-node).

flux run is a strong launcher fit: cpus-per-proc and gpus-per-proc are both native, where the Hydra-family backends warn and skip them. Binding is coarser: Flux's -o cpu-affinity is a distribution policy (off | per-task | map:LIST), not a granularity, so knit's core/socket/numa/thread all map to per-task (see _knit_launch_bind_value); reach exact placement through --launcher-args -o cpu-affinity=map:LIST.

Placement is resolved and validated upstream by the run dispatcher; this backend only formats the resolved triple into an argument vector. flux run executes inside the surrounding job's Flux instance and forwards its environment to every task, so the ranks inherit the job's KNIT_* variables and the setup env.

Build the launcher argument vector for the flux backend (the flux run executable followed by its placement flags) into a caller-provided array, passed by name. Each placement flag is added only when the corresponding option is set; any --launcher-args string is word-split and appended verbatim.

Parameters:

• argv_name -- [out] Name of the array to fill with the launcher argument vector.

• opts_name -- [in] Name of the resolved placement-options associative array (keys: procs, procs-per-node, hostnames, launcher-args).

int _knit_launch_flux_exec()

Run flux run with the translated placement flags followed by the worker command, and return its exit status. The worker command is everything after a literal "--". The launcher argv is built by _knit_launch_flux_cmdline.

Parameters:

• arr_name -- [in] Name of the resolved placement-options associative array.

• -- -- [in] Literal separator.

• ... -- [in] The worker command and its arguments.

launch_mpich.sh

Functions

int _knit_launch_mpich_cmdline()

The "mpich" launcher backend places an app's ranks with MPICH's mpiexec (the Hydra process manager). It translates knit's minimal placement options into Hydra flags:

--procs N -> -n N --procs-per-node M -> -ppn M --hostnames h0,h1 -> -hosts h0,h1 --bind V -> -bind-to <V> (V normalized by _knit_launch_bind_value) --launcher-args … -> appended verbatim after the placement flags

Hydra has no native flag for CPUs-per-rank (--cpus-per-proc) or GPU placement (--gpus-per-proc / --gpu-bind), so those are warned about and skipped; reach them through --launcher-args.

Placement is resolved and validated upstream by the run dispatcher; this backend only formats the resolved triple into an argument vector. Hydra forwards the submitting environment to every rank by default, so the ranks inherit the surrounding job's environment (KNIT_* variables and the setup env).

Build the launcher argument vector for the mpich backend (the mpiexec executable followed by its placement flags) into a caller-provided array, passed by name. Each placement flag is added only when the corresponding option is set; any --launcher-args string is word-split and appended verbatim.

Parameters:

• argv_name -- [out] Name of the array to fill with the launcher argument vector.

• opts_name -- [in] Name of the resolved placement-options associative array (keys: procs, procs-per-node, hostnames, launcher-args).

int _knit_launch_mpich_exec()

Run mpiexec with the translated placement flags followed by the worker command, and return its exit status. The worker command is everything after a literal "--". The launcher argv is built by _knit_launch_mpich_cmdline.

Parameters:

• arr_name -- [in] Name of the resolved placement-options associative array.

• -- -- [in] Literal separator.

• ... -- [in] The worker command and its arguments.

launch_none.sh

Functions

int _knit_launch_none_cmdline()

Build the launcher argument vector for the none backend into a caller-provided array, passed by name. There is no launcher, so the array is left empty. The placement is validated first so an over-specified request is still rejected on this path.

Parameters:

• argv_name -- [out] Name of the array to fill with the launcher argument vector.

• opts_name -- [in] Name of the resolved placement-options associative array.

int _knit_launch_none_exec()

Run the worker command directly, with no launcher, yielding a single rank-0 / size-1 process. The worker command is everything after a literal "--". The placement is validated first (single local rank only). Returns the worker's exit status.

Parameters:

• arr_name -- [in] Name of the resolved placement-options associative array.

• -- -- [in] Literal separator.

• ... -- [in] The worker command and its arguments.

int _knit_launch_none_validate()

The "none" launcher backend is the graceful-degradation case: no MPI launcher is detected or configured, so an app runs as a single rank-0 / size-1 process directly on the host, with no launcher in front of it. It is not a standalone path — a run is always inside a job's allocation — it just means "this

allocation has no launcher, run one rank here."

Because it can only ever run one rank on the local host, it rejects any placement that asks for more: the only accepted request is a single local rank (procs 1, procs-per-node 1, hostnames = this host), or the equivalent with the options left to their defaults. Anything else (a second rank, a remote host) is a configuration error the user should see immediately rather than have silently collapsed to one local rank.

Fail fast unless the resolved placement is a single rank on the local host. The none backend has no launcher and cannot spread ranks, so a request for more than one process, more than one process per node, or a host other than this machine is fatal. Empty (defaulted) options are accepted — they resolve to the single local rank the backend runs. --launcher-args is ignored (there is no launcher to pass it to).

Parameters:

• arr_name -- [in] Name of the resolved placement-options associative array (keys: procs, procs-per-node, hostnames).

launch_openmpi.sh

Functions

int _knit_launch_openmpi_cmdline()

Build the launcher argument vector for the openmpi backend (the mpirun executable followed by its placement flags) into a caller-provided array, passed by name. Each placement flag is added only when the corresponding option is set; the --host value is slot-annotated by _knit_launch_openmpi_host_slots, and any --launcher-args string is word-split and appended verbatim.

Parameters:

• argv_name -- [out] Name of the array to fill with the launcher argument vector.

• opts_name -- [in] Name of the resolved placement-options associative array (keys: procs, procs-per-node, hostnames, launcher-args).

int _knit_launch_openmpi_env_forward()

Build the mpirun "-x NAME" flags that forward the current environment to every rank, into a caller-provided array passed by name. mpirun does not forward the environment to ranks it starts on other nodes over SSH (see the file header), so the surrounding job's environment (KNIT_* variables and the setup env) is forwarded explicitly here to match srun/Hydra behavior.

Every exported variable is forwarded except two classes that must not be copied from the launching shell to the ranks:

• the launcher's own per-rank and resource-manager variables (OMPI_*, PMI_*, PMIX_*, and the Slurm/PBS/PALS/Flux/Hydra/PRTE families), which each rank or backend sets for itself — copying the launching shell's values would corrupt a rank's identity or confuse the backend; and

• shell- and host-local values (HOSTNAME, PWD, TMPDIR, SSH_* and the like), which are meaningful only on the launching node. The per-rank KNIT_MPI_RANK/SIZE/LOCAL_RANK are likewise skipped; the worker derives them from the launcher-native variables on each rank.

Parameters:

• argv_name -- [out] Name of the array to fill with the "-x NAME" flags.

int _knit_launch_openmpi_exec()

Run mpirun with the translated placement flags followed by the worker command, and return its exit status. The worker command is everything after a literal "--". The launcher argv is built by _knit_launch_openmpi_cmdline; the "-x" environment-forwarding flags (built by _knit_launch_openmpi_env_forward) are spliced in right after the mpirun executable so remote ranks inherit the job environment. The forwarding flags are intentionally left out of the recorded native command (which stays the clean placement command).

Parameters:

• arr_name -- [in] Name of the resolved placement-options associative array.

• -- -- [in] Literal separator.

• ... -- [in] The worker command and its arguments.

int _knit_launch_openmpi_host_slots()

The "openmpi" launcher backend places an app's ranks with Open MPI's mpirun. It translates knit's minimal placement options into mpirun flags:

--procs N -> -n N --procs-per-node M -> --npernode M --hostnames h0,h1 -> --host h0:S,h1:S (S = per-host slot count) --cpus-per-proc N -> --map-by slot:PE=N --bind V -> --bind-to <V> (V normalized by _knit_launch_bind_value) --launcher-args … -> appended verbatim after the placement flags

GPU placement (--gpus-per-proc / --gpu-bind) has no portable mpirun flag, so it is warned about and skipped; reach GPU affinity through --launcher-args. Note that --map-by slot:PE=N may conflict with --npernode (--procs-per-node), which is itself a mapping directive; this translation is best-effort (there is no live CI for it) and --launcher-args is the escape hatch if a site needs a different mapping.

The slot suffix on each --host entry is essential: mpirun's bare "--host h0,h1" advertises only one slot per host (a documented Open MPI default), so "-n 4" over two such hosts fails with "not enough slots" even inside a matching Slurm/PBS allocation, because an explicit --host overrides the resource manager's slot counts. Annotating each host with S slots (S = --procs-per-node when set, else ceil(procs / nhosts)) states the intended per-host capacity so the requested rank count fits. This is also what a plain SSH/none-scheduler cluster needs, where there is no resource manager to supply slots at all.

Exception — Open MPI built with PBS TM integration (--with-tm): inside a PBS job Open MPI's tm RAS already knows the allocated nodes and their slot counts, and an explicit --host (with or without slot suffixes) conflicts with it — the mapper rejects the placement ("requested more processes than the ppr ... can

support", or "not enough slots"). So when $PBS_NODEFILE is set the --host flag is omitted entirely and the allocation is left to TM; -n / --npernode still control the rank count and per-node distribution. (Slurm's PLM, by contrast, tolerates the redundant --host and needs it to honour a strict host subset, so it is kept there.) A strict host

subset of a PBS allocation via --hostnames is therefore not expressible to this backend; use MPICH there.

Placement is resolved and validated upstream by the run dispatcher; this backend only formats the resolved triple into an argument vector.

Environment forwarding: unlike srun and MPICH's Hydra, which pass the launching environment to every rank, mpirun forwards nothing to ranks it starts on other nodes over SSH. The ranks on the launching node inherit the job's environment by fork, but the remote ranks would start with a bare login environment and lose the surrounding job's environment (KNIT_* variables and the setup env). So _knit_launch_openmpi_exec forwards the environment explicitly with mpirun's -x, skipping the variables that must stay per-rank or per-host (the launcher's own rank/PMI variables and shell-local values); see _knit_launch_openmpi_env_forward.

Build the value of mpirun's --host flag from a comma-separated host list, annotating each host with a per-host slot count so the requested rank count fits (see the file header for why bare --host is not enough). The slot count is --procs-per-node when set; otherwise it is derived as ceil(procs / nhosts) so procs ranks spread across the hosts. When neither procs-per-node nor procs is known, the hosts are emitted unannotated (nothing to size them by).

Parameters:

• hosts -- [in] Comma-separated host list (the resolved --hostnames value).

• ppn -- [in] Resolved --procs-per-node value (may be empty).

• procs -- [in] Resolved --procs value (may be empty).

launch_pals.sh

Functions

int _knit_launch_pals_cmdline()

The "pals" launcher backend places an app's ranks with the HPE Cray PALS (Parallel Application Launch Service) mpiexec, as found on ALCF Polaris and Aurora. Like openmpi and mpich — and unlike the scheduler-integrated slurm/pbs backends — it is MPI-native and auto-detectable (_knit_detect_launcher recognises PALS by its mpiexec --help banner). It translates knit's minimal placement options into PALS mpiexec flags:

--procs N -> -n N --procs-per-node M -> --ppn M --hostnames h0,h1 -> --hosts h0,h1 --cpus-per-proc N -> --depth N --bind V -> --cpu-bind <V> (V normalized by _knit_launch_bind_value) --launcher-args … -> appended verbatim after the placement flags

PALS uses the long spellings --ppn and --hosts (double dash), distinguishing it from the Hydra-based mpich/pbs backends that use -ppn/-hosts. To bind each rank to the cores reserved by --depth, pass --bind (e.g. --bind core) or --launcher-args "--cpu-bind depth"; knit does not auto-inject a --cpu-bind. GPU placement (--gpus-per-proc / --gpu-bind) has no PALS mpiexec flag — GPU affinity on PALS systems is set by a wrapper script — so it is warned about and skipped; reach it through --launcher-args.

Placement is resolved and validated upstream by the run dispatcher; this backend only formats the resolved triple into an argument vector. PALS mpiexec forwards the submitting environment to every rank by default, so the ranks inherit the surrounding job's environment (KNIT_* variables and the setup env).

Build the launcher argument vector for the pals backend (the mpiexec executable followed by its placement flags) into a caller-provided array, passed by name. Each placement flag is added only when the corresponding option is set; any --launcher-args string is word-split and appended verbatim.

Parameters:

• argv_name -- [out] Name of the array to fill with the launcher argument vector.

• opts_name -- [in] Name of the resolved placement-options associative array (keys: procs, procs-per-node, hostnames, launcher-args).

int _knit_launch_pals_exec()

Run the PALS mpiexec with the translated placement flags followed by the worker command, and return its exit status. The worker command is everything after a literal "--". The launcher argv is built by _knit_launch_pals_cmdline.

Parameters:

• arr_name -- [in] Name of the resolved placement-options associative array.

• -- -- [in] Literal separator.

• ... -- [in] The worker command and its arguments.

launch_pbs.sh

Functions

int _knit_launch_pbs_cmdline()

The "pbs" launcher backend places an app's ranks with the PBS mpiexec wrapper (Hydra-based, as shipped with PBS Pro / OpenPBS). Unlike the MPI-native backends (openmpi/mpich), it is scheduler-integrated: the PBS mpiexec wrapper reads $PBS_NODEFILE from the surrounding allocation. It is never auto-detected — it is selectable only via --launcher pbs, the launcher metadata, or a machine profile. It translates knit's minimal placement options into Hydra flags:

--procs N -> -n N --procs-per-node M -> -ppn M --hostnames h0,h1 -> -hosts h0,h1 --bind V -> -bind-to <V> (V normalized by _knit_launch_bind_value) --launcher-args … -> appended verbatim after the placement flags

The Hydra-based wrapper has no native flag for CPUs-per-rank (--cpus-per-proc) or GPU placement (--gpus-per-proc / --gpu-bind), so those are warned about and skipped; reach them through --launcher-args.

Placement is resolved and validated upstream by the run dispatcher; this backend only formats the resolved triple into an argument vector. The wrapper runs inside the surrounding job's allocation and forwards its environment to every rank, so the ranks inherit the job's KNIT_* variables and the setup env.

Build the launcher argument vector for the pbs backend (the mpiexec executable followed by its placement flags) into a caller-provided array, passed by name. Each placement flag is added only when the corresponding option is set; any --launcher-args string is word-split and appended verbatim.

Parameters:

• argv_name -- [out] Name of the array to fill with the launcher argument vector.

• opts_name -- [in] Name of the resolved placement-options associative array (keys: procs, procs-per-node, hostnames, launcher-args).

int _knit_launch_pbs_exec()

Run the PBS mpiexec wrapper with the translated placement flags followed by the worker command, and return its exit status. The worker command is everything after a literal "--". The launcher argv is built by _knit_launch_pbs_cmdline.

Parameters:

• arr_name -- [in] Name of the resolved placement-options associative array.

• -- -- [in] Literal separator.

• ... -- [in] The worker command and its arguments.

launch_slurm.sh

Functions

int _knit_launch_slurm_cmdline()

The "slurm" launcher backend places an app's ranks with Slurm's srun. Unlike the MPI-native backends (openmpi/mpich), it is scheduler-integrated: srun talks to the surrounding Slurm allocation directly. It is never auto-detected — it is selectable only via --launcher slurm, the launcher metadata, or a machine profile. It translates knit's minimal placement options into srun flags:

--procs N -> --ntasks N --procs-per-node M -> --ntasks-per-node M --hostnames h0,h1 -> --nodelist h0,h1 --nodes k (k = number of hosts) --cpus-per-proc N -> --cpus-per-task N --bind V -> --cpu-bind=<V> (V normalized by _knit_launch_bind_value) --gpus-per-proc N -> --gpus-per-task N --gpu-bind V -> --gpu-bind=V (value passed through verbatim) --launcher-args … -> appended verbatim after the placement flags

Placement is resolved and validated upstream by the run dispatcher; this backend only formats the resolved triple into an argument vector. srun runs inside the surrounding job's allocation and forwards its environment to every task, so the ranks inherit the job's KNIT_* variables and the setup env.

Build the launcher argument vector for the slurm backend (the srun executable followed by its placement flags) into a caller-provided array, passed by name. Each placement flag is added only when the corresponding option is set. When --hostnames is given, the node count for --nodes is derived from the number of comma-separated hosts. Any --launcher-args string is word-split and appended verbatim.

Parameters:

• argv_name -- [out] Name of the array to fill with the launcher argument vector.

• opts_name -- [in] Name of the resolved placement-options associative array (keys: procs, procs-per-node, hostnames, launcher-args).

int _knit_launch_slurm_exec()

Run srun with the translated placement flags followed by the worker command, and return its exit status. The worker command is everything after a literal "--". The launcher argv is built by _knit_launch_slurm_cmdline.

Parameters:

• arr_name -- [in] Name of the resolved placement-options associative array.

• -- -- [in] Literal separator.

• ... -- [in] The worker command and its arguments.

local.sh

Functions

int _knit_submit_local()

Submit a command as a background process, acting as a minimal local job scheduler for development and testing on machines without an HPC scheduler.

The process is launched with nohup so it survives terminal disconnects. nohup exec-replaces itself with the target command without forking, so the returned PID refers directly to the running command process.

Prints the PID of the background process on stdout and returns 0. Returns 1 on argument errors.

Everything after a literal -- is the command (and its arguments) to run and is not validated as a submission option. Options may be given either as "--name value" or "--name=value".

Parameters:

• ... -- [in] Submission options followed by -- and the command to run: --stdout <file> Redirect command stdout to <file> (default: /dev/null) --stderr <file> Redirect command stderr to <file> (default: /dev/null) --stdin <file> Redirect command stdin from <file> (default: /dev/null) --walltime HH:MM:SS Kill the command after this wall-clock time.

int _knit_wait_local()

Wait until a locally submitted background process has finished.

Uses kill -0 polling because wait(1) only works for child processes of the current shell, and _knit_submit_local detaches the process via nohup.

Parameters:

• pid -- [in] PID returned by _knit_submit_local.

int _knit_walltime_to_seconds()

Convert a wall-clock time string in HH:MM:SS format to an integer number of seconds, printed to stdout. The hours field may have any number of digits; the minutes and seconds fields are two digits in the range 00-59. Fields are read as decimal (10#) so leading zeros are not taken as octal.

This is the single walltime parser shared across the scheduler backends (the local timeout, the Flux FSD conversion, queue selection). A value that is not in HH:MM:SS form is rejected: nothing is printed and the function returns 1, so a caller can fall back (e.g. pass a site-specific duration through verbatim).

Parameters:

• walltime -- [in] Wall-clock time in HH:MM:SS format.

log.sh

Functions

int _knit_ensure_trace_file()

Create the trace file on first use and cache its path in _KNIT_TRACE_FILE. Must be called in the current shell (not a subshell) before reading _KNIT_TRACE_FILE.

int _knit_log()

This function acts like printf but takes a log level first, and adds [knit:<level>] in front and

after the text. It outputs to stderr.

Example:

_knit_log info "Hello, Matthieu"

Parameters:

• level -- [in] Logging level.

• ... -- [in] Arguments for printf.

int _knit_log_level_to_int()

Convert a log level string to its integer value, storing it in the caller-named variable. trace=0, debug=1, info=2, warning=3, error=4, critical=5. Yields 2 (info) for unrecognized values.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the integer value.

• level -- [in] Log level string.

Variables

AssociativeArray _KNIT_LOG_LEVELS

Maps each log level name to its integer value: trace=0, debug=1, info=2, warning=3, error=4, critical=5. The log functions run once per registered command and parameter at load time, so each looks the current level up here directly (an "${_KNIT_LOG_LEVELS[${KNIT_LOG_LEVEL:-info}]:-2}" arithmetic test) rather than calling a helper: the lookup reads KNIT_LOG_LEVEL fresh (so a direct assignment still takes effect) with no per-call function overhead, and an unknown level yields 2 (info). The subscript defaults to "info" ("${KNIT_LOG_LEVEL:-info}") because an empty subscript is a Bash error, which the "${...:-2}" value default cannot suppress; a non-empty but unknown level is a valid (absent) key and does fall through to 2.

String _KNIT_TRACE_FILE

The trace file is the file used to redirect the output of programs. It is created lazily by _knit_ensure_trace_file() on first use rather than eagerly here: creating it at source time ran an mktemp on every "source knit.sh" and leaked a temporary file each time (the test suite sources knit.sh hundreds of times). Empty until first needed.

main.sh

Variables

ExportedString _KNIT_JUMP_TO_DIR

Opt-in, internal environment variable used by the re-entry paths (submit, run) to relocate the shell after the framework has been sourced. Those paths arrange for the experiment to be sourced from the directory holding knit.sh (so a bare source knit.sh resolves) and set this variable to the directory the body should actually run in. When it is unset (the normal top-level invocation) nothing happens: the feature is fully inert for ordinary runs.

metadata.sh

Functions

int _knit_metadata_discover()

Register the metadata subcommands (store / load / show). Called lazily by the framework the first time a metadata subcommand is resolved, listed in "--help", or walked by "describe".

Parameters:

• parent -- [in] The parent command's display name (unused).

int _knit_metadata_get()

Look up the value associated with a key in the metadata table and store it in the caller-named variable (empty when the key is absent). This is the nameref-returning counterpart of the metadata load command body, for internal hot-path callers that would otherwise capture the value with a forking command substitution.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the value.

• key -- [in] Metadata key to look up.

int _knit_metadata_is_json()

Return success when the given text is valid JSON, using Knit's jq.

Parameters:

• value -- [in] Text to test.

int _knit_metadata_load()

Load the value associated with a key from the metadata table (the CLI command body; prints the value to stdout). Internal callers should use _knit_metadata_get instead to avoid a command substitution.

int _knit_metadata_render_value()

Render a metadata value for a single line of at most a given width. The rules, in order:

• a value with no newline that fits in the budget is shown as-is;

• otherwise a value that is valid JSON is shown as "<json>";

• otherwise a value whose first line fits is shown as that first line followed by "...";

• otherwise the first line is truncated to fit and followed by "...".

Parameters:

• __knit_ret -- [out] Name of the variable to hold the rendered value.

• value -- [in] The raw metadata value.

• avail -- [in] Number of columns available for the value.

int _knit_metadata_show()

Show the content of the metadata table, one key/value pair per line. Values are rendered by _knit_metadata_render_value so that long or multi-line values do not take over the screen.

int _knit_metadata_store()

pilot.sh

Functions

int _knit_pilot_after_cb()

After-callback installed by knit_with_pilot, applying the body-exit policy for sub-jobs still outstanding when the body returns. Default (drain): ask the daemon to run every submitted sub-job to completion, blocking until it has, then exit — the safe "don't lose work" default, so each sub-job records its own terminal state. With --kill-on-exit: instead kill outstanding (running + queued) sub-jobs and mark them killed (_knit_pilot_teardown_kill). Either way the daemon is then reaped. A no-op when no daemon was started (KNIT_PILOT_SOCK unset). A walltime kill mid-drain does not reach here: the kill-trap (_knit_pilot_killed_trap) runs instead.

Parameters:

• kill_on_exit -- [in] Non-empty to kill outstanding sub-jobs instead of draining.

int _knit_pilot_before_cb()

Before-callback installed by knit_with_pilot, run after the generic job before-callback (so the allocation's hostnames are already discoverable). It builds the resource inventory — the node pool (knit_job_hostnames, less the launch host when --exclude-launch-host) plus the per-node core/GPU counts from bootstrap metadata — starts knit-pilotd on a control socket under the job directory, waits for the socket to appear, and exports KNIT_PILOT / KNIT_PILOT_SOCK so the body's knit submit calls reach the daemon. The node pool is built before KNIT_PILOT is exported so host discovery still resolves the real scheduler backend. Daemon output goes to <jobdir>/pilot.log.

Parameters:

• exclude_launch_host -- [in] Non-empty to drop the local host from the pool.

• ... -- [in] The job's own runtime arguments (ignored).

int _knit_pilot_build_nodelist()

Print the comma-separated node list for the pilot daemon's resource pool: the deduplicated hostnames the pilot job was allocated (knit_job_hostnames). When exclude_launch_host is non-empty, the daemon's own host is dropped from the pool if present — compared on the short (pre-dot) name so it matches whether the allocation reports short names or FQDNs. The drop is a no-op when the host is not in the pool (a separate launch/service node). Prints the empty string when no node remains.

Parameters:

• exclude_launch_host -- [in] Non-empty to drop the local host from the pool.

int _knit_pilot_killed_trap()

Kill-trap for a pilot job, replacing the generic _knit_job_killed_trap on HUP/TERM/USR1 (a cancel, an out-of-time or walltime kill). It first tears the daemon down and marks all outstanding sub-jobs killed, so a killed pilot never leaves a sub-job "running" or lost, then chains to the generic trap to record the pilot itself "killed" and exit. This is also the path a walltime kill arriving mid-drain falls through to: the signal interrupts the blocking drain in the after-callback and runs this instead.

int _knit_pilot_mark_outstanding_killed()

Mark every sub-job the pilot submitted that is not already in a terminal state as "killed" in the jobs table. Called from the kill paths (a killed pilot, or --kill-on-exit) after the daemon has been told to kill/drop outstanding sub-jobs: it closes the gap for queued sub-jobs that never started a process of their own (and so never recorded a terminal state) and is a harmless idempotent re-write for running sub-jobs whose own kill-trap already recorded "killed". Best-effort and gated on bootstrap; a sub-job with no row (unknown state) is skipped. Reads the tracking file written by _knit_pilot_record_subjob.

int _knit_pilot_record_subjob()

Record a sub-job UUID the pilot submitted, one per line, in a tracking file under the pilot job directory. Teardown reads this list to reconcile sub-jobs the daemon killed or dropped: a running sub-job records its own "killed" when signalled, but a still-queued sub-job never started a knit process and would otherwise stay "submitted", so the pilot marks it killed from this list. A no-op when not inside a pilot job directory (e.g. under unit tests).

Parameters:

• id -- [in] The sub-job UUID just queued with the daemon.

int _knit_pilot_reject_nested_submit()

The nested-submit guard. knit submit is allowed on the login node and, inside a pilot job, retargeted to the in-allocation daemon; everywhere else inside a job body it is rejected. Returns 0 (allow) when not inside a job body (KNIT_JOB_PREFIX unset — the ordinary login-side submission) or when a pilot is active (KNIT_PILOT set); otherwise it is a nested submission from a non-pilot job body and fatals, naming knit_with_pilot. Called at the top of _knit_submit, which only the real submit -- <job> dispatch reaches — the compute-side body re-entry runs submit <job> (no --) straight into the job function, so this never blocks a job from starting.

int _knit_pilot_require_sock()

Fatal unless KNIT_PILOT_SOCK names a control socket to talk to the daemon. Guards the pilot backend's submit/wait/cancel paths, which are only reachable from inside a pilot job body; a missing socket means one of those was reached outside a pilot, which is a programming error worth a clear message.

int _knit_pilot_teardown_kill()

Tear the pilot down the hard way: tell the daemon to kill every running sub-job's process group and drop every queued one (shutdown --kill, which also makes the daemon exit), then reconcile the sub-job rows so none is left "submitted" or "running". Shared by the --kill-on-exit body-exit policy and the pilot's kill-trap. A no-op when no daemon was started (KNIT_PILOT_SOCK unset).

int _knit_sched_pilot_cancel()

Cancel a pilot sub-job by asking the daemon to terminate it: it kills a running sub-job's process group and drops a still-queued one, marking it killed. Cancelling a sub-job that is already gone is not an error. Knit's terminal state (killed) is recorded by the sub-job's own kill-trap / the daemon.

Parameters:

• payload -- [in] The pilot payload "<parent-uuid>:<sub-job-id>" (a bare id is accepted too). Stripped to the sub-job id for the daemon.

int _knit_sched_pilot_directives()

Emit the batch directives for the pilot backend. A pilot sub-job is launched by the daemon as a plain process, not by a batch scheduler, so there are no directives and this prints nothing — it exists so every backend honours the same contract.

Parameters:

• arr_name -- [in] Name of the resolved-options associative array (unused).

• jobdir -- [in] Job directory (unused).

int _knit_sched_pilot_hostfile()

Print the host list for a pilot sub-job: the node subset the daemon assigned it, named in KNIT_PILOT_HOSTFILE (one host per line). This is the "raw" host list per the dispatcher contract — callers such as knit_job_hostnames deduplicate and clean it. Blank lines are dropped. When no readable hostfile is present, warn and fall back to this machine's hostname. Reached from _knit_sched_hostfile whenever KNIT_PILOT_HOSTFILE is set, so a sub-job places its work onto its own subset without needing any pilot environment of its own.

int _knit_sched_pilot_status()

Return the status of a pilot sub-job. The payload is "<parent-uuid>:<sub-job-id>".

Precise path: when a live daemon is reachable (KNIT_PILOT_SOCK points at an existing socket, e.g. reconcile run from inside the pilot body), ask it with the non-blocking STATUS verb for the sub-job's actual state and map it: queued or running -> 'running' (left alone); completed -> 'completed'; failed -> 'failed'; killed or unknown -> 'killed'. A daemon that cannot be reached (stale socket, non-zero exit, empty/unrecognized answer) falls through to the parent walk.

Fallback (no reachable daemon, e.g. reconcile from the login node after a hard-killed pilot): a pilot sub-job lives and dies with its pilot allocation, and the pilot job is itself an ordinary machine-scheduler job, so its fate is resolved from the parent pilot's true state as the parent's own scheduler reports it. Verdict: the parent still running -> 'running' (left alone); the parent terminal/gone, or no parent to ask -> 'killed' (the allocation is gone, so a sub-job stuck non-terminal died with it; same unknown->killed policy as the other backends). The walk is transitive: a parent that is itself a pilot recurses one more hop through _knit_sched_status.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the verdict string.

• payload -- [in] The pilot payload "<parent-uuid>:<sub-job-id>".

int _knit_sched_pilot_submit()

Submit a built sub-job to the in-allocation knit-pilotd daemon: queue its batch script (run as "bash <script>") with the node count resolved for this submission, keyed by the sub-job UUID (the job directory's basename), with stdout/stderr redirected to the job directory. The request is non-blocking — the daemon queues it and returns at once — matching the dispatcher contract (the optional --wait is a separate _knit_sched_pilot_wait step). On the daemon accepting the request, "<parent-uuid>:<sub-job-id>" is printed as the backend job id: the daemon tracks the sub-job by <sub-job-id> (wait/cancel strip to it) and <parent-uuid> links the sub-job to the pilot job that submitted it, so reconcile can resolve a stuck sub-job's fate from the parent pilot's state. A daemon rejection propagates as a non-zero status, which the dispatcher treats as a scheduler rejection.

Parameters:

• arr_name -- [in] Name of the resolved-options associative array.

• script -- [in] Path to the batch script to queue.

• jobdir -- [in] Job directory (its basename is the sub-job UUID).

int _knit_sched_pilot_submit_cmdline()

Build the pilot backend's submission command into a caller-provided array, passed by name, for recording as the row's native-cmd: the daemon submit with the node count, launching "bash <script>". The socket, sub-job id, and stdout/stderr redirection are knit-managed plumbing (and the id/socket are not available here), so — like the local backend's "bash <script>" — they are omitted from the recorded command, which captures the essential shape.

Parameters:

• argv_name -- [out] Name of the array to fill with the submission argv.

• arr_name -- [in] Name of the resolved-options associative array.

• script -- [in] Path to the batch script to queue.

int _knit_sched_pilot_wait()

Block until a pilot sub-job reaches a terminal state, by asking the daemon to wait on its id. The daemon's client exits 0 when the sub-job completed and non-zero when it failed or was killed, which this returns unchanged — the same contract the other backends' wait functions honour (knit's own terminal state is read from the jobs table afterwards by the caller).

Parameters:

• payload -- [in] The pilot payload "<parent-uuid>:<sub-job-id>" (its id field is the sub-job id the daemon tracks); a bare id is accepted too. Stripped to the sub-job id before the daemon call.

Variables

_KNIT_PILOTD_VERSION

Version of knit-pilotd to provision.

String _KNIT_PILOT_DAEMON_PID

PID of the knit-pilotd daemon the pilot before-callback started, kept so the after-callback can reap it. Lives only within the pilot job body's process.

plugin.sh

Functions

int _knit_plugin_bootstrap()

Bootstrap-time integration for imported plugins, run from _knit_bootstrap (both first bootstrap and update mode). For every plugin imported by the experiment this invocation it writes the committed lockfile, ensures the plugins table, records a provenance row (idempotent), and materializes the pinned tree under .knit for a self-contained experiment directory. A no-op when nothing was imported.

int _knit_plugin_cached_commit()

Find a usable cached commit for a plugin id when its ref cannot be resolved upstream (rate-limited or a transient network error). The content-addressed cache keys trees by commit SHA with no record of which ref produced which, so this returns the most recently fetched commit tree (the best guess for the ref's current tip) that still carries a plugin.sh. It looks in the bootstrapped ".knit" tree and the shared cache, since knit_import can source from either. Returns 1 (output untouched) when nothing is cached.

Parameters:

• commit_out -- [out] Nameref receiving the cached commit SHA.

• id -- [in] Canonical plugin id ("<host>/<org>/<repo>").

Returns:

0 and sets the output when a cached commit exists; 1 otherwise.

int _knit_plugin_defer()

Record a plugin whose code is unavailable this invocation (offline, uncached, unpinned) so it shows in the describe/--help footer, and install the command_not_found safety net so the experiment source still loads when it calls the plugin's (now undefined) symbols under "set -e".

Parameters:

• id -- [in] Canonical id ("<host>/<org>/<repo>").

• ref -- [in] Requested ref (may be empty).

int _knit_plugin_ensure_table()

Ensure the plugins provenance table exists and carries every column, creating or additively migrating it on first use this process (guarded by _KNIT_PLUGIN_TABLE_ENSURED). The schema records, per import: the uuid row id, the spec as written, the resolved host/org/repo, the requested ref, the resolved commit, the tarball checksum, the importing plugin (empty for the experiment), and the epoch second it was recorded.

int _knit_plugin_fetch()

Fetch a plugin commit into the content-addressed cache. The tarball is downloaded and extracted into a temporary directory; its sha256 is verified against an expected digest when one was given; the tree is checked for a top-level plugin.sh; then it is published atomically into the cache directory. The computed sha256 is returned so the caller can record it in the lockfile. Fatal on download failure, checksum mismatch, or a missing plugin.sh.

Parameters:

• sha256_out -- [out] Nameref receiving the fetched tarball's sha256.

• host -- [in] Repository host.

• org -- [in] Repository owner.

• repo -- [in] Repository name.

• commit -- [in] Commit SHA to fetch.

• dest -- [in] Cache directory to publish the extracted tree into.

• expected -- [in] Expected tarball sha256 (empty to skip verification).

int _knit_plugin_install_cnf_net()

Install, at most once, a command_not_found_handle that turns a call to an undefined knit_*‍/@* symbol into a no-op returning 0 (so a deferred plugin's decorators/commands do not abort a "set -e" experiment at registration time), while any other unknown command keeps the default "command not found" failure.

int _knit_plugin_lock_get()

Look up a plugin's pin in the lockfile. Sets the commit and checksum outputs to the recorded values (checksum may be empty) and returns 0 when the canonical id is pinned; returns 1 (clearing both outputs) when it is not pinned or the lockfile is absent. Blank lines and "#" comment lines are ignored.

Parameters:

• commit_out -- [out] Nameref receiving the pinned commit SHA.

• checksum_out -- [out] Nameref receiving the pinned tarball checksum.

• id -- [in] Canonical id ("<host>/<org>/<repo>").

int _knit_plugin_lock_set()

Record (or update) a plugin's pin in the lockfile, then rewrite the file with its entries sorted by canonical id for a stable, review-friendly diff. An empty checksum is written as a two-field line; a present checksum as three fields.

Parameters:

• id -- [in] Canonical id ("<host>/<org>/<repo>").

• commit -- [in] Resolved commit SHA.

• checksum -- [in] Tarball sha256 (may be empty).

int _knit_plugin_materialize()

Copy an imported plugin's pinned source tree from the shared content-addressed cache into the experiment's own ".knit/plugins/<host>/<org>/<repo>/<commit>/", so the experiment directory carries a self-contained, offline copy that knit_import sources once the experiment is bootstrapped. The destination is keyed by commit (mirroring the cache), so changing a plugin's pin materializes the new commit beside the old instead of silently reusing it. A no-op when the destination already holds this commit, and a no-op when the cache tree is absent (e.g. the import was deferred offline), so bootstrap does not fail on it.

Parameters:

• id -- [in] Canonical id ("<host>/<org>/<repo>") of a plugin in the order set.

int _knit_plugin_parse_spec()

Parse an import spec into its host, org, repo, and ref parts. Accepted forms:

• "[https://]<host>/<org>/<repo>[@<ref>]" — a full spec.

• "<org>/<repo>[@<ref>]" — host defaults to github.com.

• "<name>[@<ref>]" — the blessed-namespace shorthand, expanding to github.com/knit-sh/knit-<name>. A missing ref is returned empty (the caller treats it as the default branch). Fatal on a malformed spec.

Parameters:

• host_out -- [out] Nameref receiving the host.

• org_out -- [out] Nameref receiving the owner (org or user).

• repo_out -- [out] Nameref receiving the repository name.

• ref_out -- [out] Nameref receiving the ref (empty when none was given).

• spec -- [in] The import spec as written by the user.

int _knit_plugin_print_deferred_footer()

Print, to standard error, the footer listing plugins deferred this invocation (empty list -> nothing). describe and --help call it after their output so a user sees which plugin commands/decorators are missing and how to get them, instead of a silently truncated command tree. Standard error is used so the footer never corrupts a machine-readable describe format or piped help text.

int _knit_plugin_record_row()

Insert one provenance row for an imported plugin into the plugins table. The insert is idempotent across repeated bootstraps: a row for this host/org/repo at this exact commit is written at most once, so re-running bootstrap (or bootstrap update mode) never duplicates an unchanged plugin.

Parameters:

• id -- [in] Canonical id ("<host>/<org>/<repo>") of a plugin in the order set.

int _knit_plugin_resolve_commit()

Resolve a plugin ref to an exact commit SHA, honoring the lockfile. Precedence: a full 40-hex-character SHA resolves to itself with no network access; else a lockfile pin for this canonical id is authoritative (deterministic and offline-friendly); else the ref is resolved through the host API (an empty ref means the default branch, resolved as "HEAD") and the writeback flag is set so the caller records the fresh pin. When KNIT_OFFLINE=1 blocks a needed network resolution, the function returns 1 (the caller defers the import) rather than failing.

Parameters:

• commit_out -- [out] Nameref receiving the resolved commit SHA.

• writeback_out -- [out] Nameref set to "true" when a fresh (unpinned, network-resolved) commit should be written back to the lockfile.

• host -- [in] Repository host (only github.com is supported today).

• org -- [in] Repository owner.

• repo -- [in] Repository name.

• ref -- [in] Ref to resolve (empty for the default branch).

Returns:

0 when resolved; 1 when it cannot be resolved offline (defer).

int _knit_plugin_write_lockfile()

Write every imported plugin's resolved pin (commit + checksum) to the committed lockfile, so a fresh checkout resolves the same commits before bootstrap. Each entry is upserted (see _knit_plugin_lock_set), leaving an unrelated pin already in the file untouched. A no-op when nothing was imported.

int _knit_plugins_discover()

Register the "plugins" subcommands. Called lazily by the framework the first time a plugins subcommand is resolved, listed in "--help", or walked by "describe".

Parameters:

• parent -- [in] The parent command's display name (unused).

int _knit_plugins_list()

Body of "knit plugins list": print the plugins recorded in the database, one per line, with the canonical id, the requested ref, the resolved commit (short) and the importer, aligned into columns and ordered chronologically. Reports (and succeeds) when no plugin has been recorded.

int _knit_plugins_lock()

Body of "knit plugins lock": (re)write the committed lockfile from the plugins imported this invocation, without a full bootstrap. Useful on a fresh checkout to freeze the resolved commits before anything else. Reports, but does not fail, when no plugin was imported.

int _knit_plugins_show()

Body of "knit plugins show --id <selector>": print the full recorded metadata and provenance of a plugin. The selector matches a plugin's row id, its canonical id ("host/org/repo"), or its repository short name (with or without the "knit-" prefix of the blessed namespace), so "spack", "knit-spack", and "github.com/knit-sh/knit-spack" all select the same plugin. A plugin recorded at more than one commit over the experiment's life prints one block per recording, oldest first. No match is fatal.

int _knit_plugins_table_exists()

Return 0 when the plugins provenance table exists in the database, 1 otherwise. The table is created at bootstrap only when the experiment imported at least one plugin, so its absence means "no plugins recorded" rather than an error.

Variables

String _KNIT_PLUGIN_CACHE_ROOT

Root of the content-addressed plugin cache, shared across experiments and invocations. A plugin commit is fetched once into "<root>/<host>/<org>/<repo>/<commit>/" and every later import of that commit is an offline cache hit. Defaults under XDG_CACHE_HOME (falling back to ~/.cache); a test or a reproducer may override it.

AssociativeArray _KNIT_PLUGIN_CHECKSUM

Map of canonical id to the fetched tarball's sha256 (empty when the commit came from the cache with no recorded pin). Written to the lockfile and the plugins-table "checksum" column.

String _KNIT_PLUGIN_CNF_INSTALLED

"true" once the command_not_found_handle safety net has been installed (on the first deferral), so it is installed at most once. The net keeps a "set -e" experiment source loading when it calls a deferred plugin's missing knit_*‍/@* symbol at registration time.

AssociativeArray _KNIT_PLUGIN_COMMITS

Map of canonical id to the exact commit SHA its ref resolved to. Populated as each plugin is imported; the seed of the provenance recorded at bootstrap.

Array _KNIT_PLUGIN_DEFERRED

Labels ("<id>" or "<id>@<ref>") of plugins that could not be made available in the current invocation (offline, with no cached commit and no lockfile pin). Their import is deferred rather than fatal, and they are listed in the describe/--help footer so the gap is explicit rather than a silent truncation.

AssociativeArray _KNIT_PLUGIN_IMPORTED_BY

Map of canonical id to the id of the plugin that imported it, or the empty string when the experiment imported it directly. This is the dependency edge recorded in the plugins-table "imported_by" column.

Array _KNIT_PLUGIN_IMPORT_STACK

Stack of plugin ids currently being sourced, pushed around each "source

plugin.sh" so a transitive knit_import done from a plugin's own

plugin.sh knows which plugin imported it (the stack top) and records the dependency edge.

String _KNIT_PLUGIN_LOCKFILE

Path to the committed lockfile that pins every imported plugin (canonical id -> resolved commit + checksum). It lives next to the experiment script and ships in the experiment's VCS, so a fresh checkout resolves the exact commits before bootstrap, for both introspection and reproduction. A resolution prefers a lockfile pin over the ref; a mutable ref with no pin is resolved once and written back here. A test or reproducer may override the path.

Array _KNIT_PLUGIN_ORDER

Canonical ids ("<host>/<org>/<repo>") of the plugins imported this invocation, in first-import order (a dependency, imported transitively from a dependent's plugin.sh, sorts before the dependent's remaining imports but after the dependent's own entry). It drives the lockfile write, the plugins-table rows, and the management commands; a repeated import of the same id is recorded once.

AssociativeArray _KNIT_PLUGIN_REFS

Map of canonical id ("<host>/<org>/<repo>") to the ref it was imported at, recorded the moment an import begins (before sourcing) so a transitive or cyclic re-import of the same id dedups instead of recursing. An empty value means the import pinned no ref (default branch). Used for dedup and for the version-conflict check.

AssociativeArray _KNIT_PLUGIN_SPEC

Map of canonical id to the import spec exactly as the user wrote it, kept for the plugins-table "spec" column (the intent, next to the resolved commit).

String _KNIT_PLUGIN_TABLE

Name of the provenance table recording which plugins an experiment imported and at which exact commit. A normal knit table in .knit/knit.db, visible to "knit query" like any other.

String _KNIT_PLUGIN_TABLE_ENSURED

"1" once the plugins table has been ensured this process, so the idempotent create-or-migrate runs at most once per invocation.

prepare.sh

Functions

int _knit_prepare()

Entry point for the prepare CLI command. Builds a submission and records it with state "prepared" without contacting the scheduler (see _knit_prepare_build): the batch script and the frozen .submit metadata are written, but no scheduler command is issued and no .job.id is recorded. The job stays queued until released by submit prepared or submit next. Prints the job UUID (the canonical, scheduler-independent identifier).

Usage:

./exp.sh prepare [--setup <setup-name>] [--name <alias>] [--group <name>] \
    [sched-args...] -- job-name [args...]

int _knit_prepare_claim_id()

Atomically claim one prepared job by its UUID, moving it from state "prepared" to the transient "submitting". Like _knit_prepare_claim_next but keyed on an explicit id: a single conditional UPDATE under the write lock whose guard ("AND state='prepared'") both prevents a double claim and yields nothing when the row is no longer prepared. Prints the claimed UUID, or nothing when the row is absent or not in state "prepared".

Parameters:

• uuid -- [in] The job UUID to claim.

int _knit_prepare_claim_next()

Atomically claim the oldest prepared job matching optional filters, moving it from state "prepared" to the transient "submitting" so no concurrent releaser can grab the same row. The whole pick-and-mark is one conditional UPDATE under the write lock (see _knit_sqlite3_write): the inner SELECT picks the lowest id in state "prepared" (job UUIDs are time-ordered uuidv7, so lowest id is oldest prepared), and the outer guard re-checks the state so a row another releaser claimed first is not taken twice. Job UUIDs order by prepare time, so this releases in prepare order. Prints the claimed UUID, or nothing when no prepared job matches (queue drained, or the race was lost).

Parameters:

• job -- [in] Optional job-name filter (the "job" column); empty means any.

• group -- [in] Optional group filter (the "group" column); empty means any.

int _knit_prepare_release()

Release a claimed prepared job (already moved to state "submitting" by one of the claim helpers) to the scheduler. The submission spec was frozen at prepare time, so this reconstructs the job name and any --name alias from the jobs row, then hands off to _knit_submit_dispatch, which builds the submit command, advances the row "submitting" -> "submitted", and issues it (cleaning up on a scheduler rejection exactly as a direct submit does). _knit_submit_dispatch prints the released job's UUID (before any --wait blocks) and then waits when asked.

Parameters:

• uuid -- [in] The claimed job UUID.

• wait_flag -- [in] "true"/"false": block until the job completes (see _knit_submit_dispatch's wait override).

int _knit_prepare_remove()

Remove a prepared job that was never dispatched: delete its jobs row, its job directory, and any --name alias. A prepared job has contacted no scheduler, so there is nothing to cancel; the teardown is exactly the rejection cleanup a failed submit performs (_knit_submit_cleanup_rejected), reached here by the "prepared" branch of job cancel. The --name alias is reconstructed from the row's "name" column, as _knit_prepare_release reconstructs it on dispatch.

Parameters:

• uuid -- [in] The prepared job UUID.

profile.sh

Functions

int _knit_load_profile()

Extract all portable fields from a profile's JSON and store them in global variables. Called by bootstrap after jq is available, with the resolved JSON content (not a name).

Sets (empty string when a field is absent): _KNIT_PROFILE_SCHEDULER_TYPE _KNIT_PROFILE_SCHEDULER_COMMAND _KNIT_PROFILE_SCHEDULER_DEFAULT_QUEUE _KNIT_PROFILE_SCHEDULER_DEFAULT_ARGS (space-joined from JSON array) _KNIT_PROFILE_LAUNCHER_TYPE _KNIT_PROFILE_LAUNCHER_COMMAND _KNIT_PROFILE_LAUNCHER_DEFAULT_ARGS (space-joined from JSON array) _KNIT_PROFILE_CORES_PER_NODE _KNIT_PROFILE_GPUS_PER_NODE

Parameters:

• json -- [in] The resolved profile JSON content.

int _knit_profile_admin_entries()

List the admin-provided profiles under _KNIT_PROFILE_ADMIN_DIR as "<name><TAB><description><TAB><hidden>" lines, one per profile, where <name> is the path relative to the admin directory minus the .json suffix and <hidden> is "true" when the profile sets "_hide": true (mirroring the GitHub index's hidden field). Empty when the directory is absent.

Every profile is listed, including hidden ones; knit_list_profiles does the filtering. Both the hidden check and the description read are jq-free (a grep/sed for the marker) so listing keeps working before bootstrap.

Parameters:

• __knit_ret1 -- [out] Name of the variable to hold the newline-separated entries.

int _knit_profile_file_description()

Extract a profile file's description field jq-free, so knit profile list can show it before bootstrap (jq may be absent). Uses a sed for the first "description": "..." occurrence; relies on the value containing no literal '"'. The result is empty when the field is absent.

Parameters:

• __knit_ret1 -- [out] Name of the variable to hold the description.

• file -- [in] Path to the profile JSON file.

int _knit_profile_github_url()

Build the raw.githubusercontent.com URL for a profile path at a ref. Uses the raw host (not github.com/.../blob/...) so the response is the JSON itself, not an HTML page.

Parameters:

• path -- [in] Profile path under src/profiles, e.g. "anl/improv".

• ref -- [in] Git ref (tag, branch, or SHA).

int _knit_profile_http_get()

Fetch a URL and, on HTTP 200, store the body in the named variable. Records the HTTP status in _KNIT_PROFILE_LAST_HTTP. Returns 0 only on HTTP 200.

Parameters:

• __knit_ret1 -- [out] Name of the variable to hold the response body.

• url -- [in] URL to fetch.

int _knit_profile_is_hidden()

Return success when the profile file names its _hide field as true. Uses a jq-free grep so it works before bootstrap (jq may be absent), tolerating the usual JSON spacing around the colon.

Parameters:

• file -- [in] Path to the profile JSON file.

int _knit_profile_latest_ref()

Resolve the latest knit release tag via the GitHub API. Prints the tag; fatal if none can be resolved.

Warning

doxygenfunction: Unable to resolve function "_knit_profile_list" with arguments "None". Candidate function could not be parsed. Parsing error is Error when parsing function declaration. If the function has no return type: Error in declarator or parameters-and-qualifiers Invalid C++ declaration: Expecting "(" in parameters-and-qualifiers. [error at 14] knit_register profile knit_empty List and inspect machine profiles _knit_is_builtin knit_usable_before_bootstrap knit_with_subcommand_discovery _knit_profile_discover knit_done _knit_profile_list () --------------^ If the function has a return type: Error in declarator or parameters-and-qualifiers If pointer to member declarator: Invalid C++ declaration: Expected '::' in pointer to member (function). [error at 22] knit_register profile knit_empty List and inspect machine profiles _knit_is_builtin knit_usable_before_bootstrap knit_with_subcommand_discovery _knit_profile_discover knit_done _knit_profile_list () ----------------------^ If declarator-id: Invalid C++ declaration: Expecting "(" in parameters-and-qualifiers. [error at 22] knit_register profile knit_empty List and inspect machine profiles _knit_is_builtin knit_usable_before_bootstrap knit_with_subcommand_discovery _knit_profile_discover knit_done _knit_profile_list () ----------------------^

int _knit_profile_parse_index()

Parse a profile index into one "<name><TAB><description><TAB><hidden>" line per entry. The index is a JSON array of one-line objects, e.g. { "name": "anl/aurora", "description": "...", "hidden": true } (see gen-profile-index.sh). Extraction is jq-free (a single sed) so it works before bootstrap, when jq is not yet available; it relies on the description containing no literal '"', which the generator guarantees for shipped descriptions.

Parameters:

• __knit_ret1 -- [out] Name of the variable to hold the newline-separated list.

• body -- [in] The index.json content.

_knit_profile_show ()

int _knit_render_platform_files()

Materialize the profile's platform artifacts under _KNIT_PREFIX: platform.sh (modules + environment) and spack-config.json (the profile's spack object). Either file is left absent when the profile omits the corresponding fields. Called by bootstrap after the profile is resolved.

Parameters:

• json -- [in] The resolved profile JSON content.

int _knit_render_platform_sh()

Render the platform shell fragment (§5.3) to a file: an optional module-init source line, an optional module purge, one module use <path> per module_use entry, a single module load of the profile's modules, and one export KEY=VALUE per environment entry. The file is left absent (not created) when the profile has neither modules nor environment. Fatal when modules is present but no module init resolves.

Parameters:

• json -- [in] The resolved profile JSON content.

• outfile -- [in] Path of the platform.sh file to write.

int _knit_render_spack_config()

Render the profile's spack object as a single Spack environment-config fragment. The keys under spack are Spack config section names (packages, mirrors, concretizer, config, compilers, ...) and each value is that section's content exactly as it appears under the corresponding top-level key in a Spack config file. Knit does not interpret the contents: it writes the whole object back wrapped under a top-level spack key, i.e. { "spack": <value> }, which is the Spack environment-manifest form. Because JSON is a subset of YAML, Spack ingests the file directly: it is spack config added into the environment (always -e <env>) at env-install time, where the spack wrapper lets a single file carry every section at once. Nothing is written when the profile declares no spack object (or an empty one).

Parameters:

• json -- [in] The resolved profile JSON content.

• outfile -- [in] Path of the spack-config file to write.

int _knit_resolve_module_init()

Resolve the environment-module init script to source when materializing the platform (§5.2), trying in order: an explicit profile "module_init" override; the MODULESHOME-derived path and the standard candidate list (_KNIT_MODULE_INIT_CANDIDATES); finally, if module is already a function or command in this environment, no init line is needed (empty result). Returns non-zero when none applies, so the caller can fatal.

Parameters:

• __knit_ret1 -- [out] Name of the variable to hold the resolved init path (empty when module is already available and no script is needed).

• json -- [in] The resolved profile JSON content.

int _knit_resolve_profile()

Resolve a --profile <spec> to its JSON content and a canonical label, trying each source in order (first match wins):

1. URL (http:// or https://) fetched verbatim

2. local file <spec> or <spec>.json on disk (the offline story)

3. admin profile /etc/knit/profiles/<spec>.json

4. GitHub shorthand <namespace>/<machine>[/<variant>...][<ref>]; the path may be two or more segments (e.g. nersc/perlmutter/cpu). A bare ref defaults to the default branch (most up-to-date profiles); @latest resolves via the releases API

On success sets the JSON content and the resolved label (a URL, path, or "<path>@<ref>"). If nothing resolves, fatal with a message enumerating every source that was tried.

Parameters:

• __knit_ret1 -- [out] Name of the variable to hold the resolved JSON content.

• __knit_ret2 -- [out] Name of the variable to hold the resolved label.

• spec -- [in] The --profile argument.

Variables

Array _KNIT_MODULE_INIT_CANDIDATES

Ordered list of environment-module init scripts searched (after an optional profile "module_init" override and the MODULESHOME-derived path) when materializing .knit/platform.sh. The first that exists is sourced to make the module shell function available. Overridable for testing.

String _KNIT_PROFILE_ADMIN_DIR

Directory of admin-provided (site) profiles, tried before the GitHub store so a machine's own copy wins and resolves offline. Overridable for testing.

String _KNIT_PROFILE_DEFAULT_REF

Git ref the GitHub shorthand (and the profile index) resolve against when no explicit is given. This is the default branch, not the running knit version tag, because the profile store on the default branch carries the most up-to-date machine profiles.

String _KNIT_PROFILE_LAST_HTTP

HTTP status of the most recent _knit_profile_http_get call ("200" on success, the returned status otherwise, empty when curl itself failed). Used to build the enumerating "profile not found" error.

String _KNIT_PROFILE_REPO

GitHub repository serving the in-repo profile store, and the org/repo the shorthand form resolves against.

prov.sh

Functions

int _knit_produced_edge_sql()

Build (print, without executing) the INSERT for a "produced" provenance edge: a producing invocation (the source) produced an artifact (the target, a row of the artifacts table). It is a thin wrapper over _knit_prov_edge_sql that bakes in the produced-edge shape: the target_name is the artifacts node label, the edge_type is "produced", and — a produced edge has no duration and no call site — the two timestamps and the alias are NULL. Meant to be composed into the producing invocation's record-time transaction (see the artifacts write path).

Parameters:

• source_id -- [in] UUID of the producing invocation (empty for a root).

• source_name -- [in] Demangled command name of the producer.

• artifact_id -- [in] UUID of the produced artifact (the artifacts row id).

int _knit_prov_create_table()

Create the provenance edge table if it does not already exist. Each row is one directed relationship, "source --edge_type--> target", of one of three kinds: a "call" edge (source invoked target), a "used_by" edge (target references a setup or resource, which is the source, built by an earlier invocation), or a "produced" edge (source invocation produced target, an artifacts-table row). The source is always the antecedent (the caller, the setup, the producer) and the target the dependent (the callee, the consumer, the artifact). Node identity is the pair (id, name); the timestamps are REAL epoch seconds and are NULL for "used_by" and "produced" edges. The nullable "alias" column holds the call-site name recorded by knit_as (NULL for a plain edge). Called at bootstrap alongside the metadata table.

int _knit_prov_edge_sql()

Build (print, without executing) the INSERT statement for one provenance edge. The statement is meant to be run through _knit_sqlite3_write, either on its own (see _knit_prov_record_edge) or inside a transaction next to a data-row insert (see _knit_db_record_invocation). Timestamps are rendered as NULL when empty.

Parameters:

• source_id -- [in] UUID of the source (caller for "call"; setup for "used_by"); empty for a root invocation.

• source_name -- [in] Demangled command name of the source (empty for a root).

• target_id -- [in] UUID of the target (callee for "call"; consumer for "used_by").

• target_name -- [in] Demangled command name of the target.

• edge_type -- [in] "call" (source invoked target), "used_by" (target references a setup/resource, which is the source), or "produced" (source produced target, an artifact).

• start_time -- [in] Epoch seconds when the call started (empty -> NULL).

• end_time -- [in] Epoch seconds when the call returned (empty -> NULL).

• alias -- [in] Call-site name from knit_as (empty -> NULL).

int _knit_prov_ensure_table()

Ensure the provenance edge table exists before an edge is written, creating it lazily on first use. A freshly bootstrapped experiment already has the table (created at bootstrap), but a database bootstrapped before this feature shipped does not; ensuring it here lets a new invocation record edges (and keeps its data-row-plus-edge transaction from rolling back) rather than failing. The create is idempotent and runs at most once per process, guarded by _KNIT_PROV_TABLE_ENSURED.

int _knit_prov_now()

Print the current time as a REAL number of seconds since the Unix epoch, at the best precision available. Used for a call edge's start_time (captured when the frame is pushed) and end_time (captured at record time), so a duration is a plain subtraction.

int _knit_prov_nullable_literal()

Render an argument as a SQL literal for a nullable column: an empty argument becomes a bare NULL; a non-empty argument becomes a single-quoted, escaped literal. Used for the two REAL timestamp columns (empty for "used_by" edges, which have no duration; SQLite's type affinity coerces the quoted number to a REAL) and for the TEXT "alias" column (empty for a plain, unaliased edge).

Parameters:

• value -- [in] Column value, or empty for NULL.

int _knit_prov_record_edge()

Insert a single provenance edge into the edge table, serialized through the advisory-locked writer. Used on its own for a target that records no data row (a table-less command) and for "used_by" edges; a target that also records a data row writes both in one transaction via _knit_db_record_invocation instead.

Parameters:

• source_id -- [in] See _knit_prov_edge_sql.

• source_name -- [in] See _knit_prov_edge_sql.

• target_id -- [in] See _knit_prov_edge_sql.

• target_name -- [in] See _knit_prov_edge_sql.

• edge_type -- [in] See _knit_prov_edge_sql.

• start_time -- [in] See _knit_prov_edge_sql.

• end_time -- [in] See _knit_prov_edge_sql.

int _knit_record_used_by_edge()

Record a "used_by" provenance edge from an already-resolved source node (a setup or a resource instance) to a consuming invocation (the target). A "used_by" edge has no duration, so both timestamps are NULL. Shared by the setup-dependency after-callback (setup:<type> source) and the resource- dependency after-callback (resource:<type> source); each caller reads its own on-disk id/type markers and passes the resolved node identity here.

Best-effort and gated with the other provenance writes: it records nothing when recording is disabled, on a suppressed rank, before bootstrap, when the target does not participate in the graph, or when the source id is empty (e.g. a setup or resource materialized before provenance shipped).

Parameters:

• source_id -- [in] Resolved row id of the source node (empty -> no edge).

• source_name -- [in] Node name of the source ("setup:<type>" / "resource:<type>").

• target_cmd -- [in] Mangled command name of the consumer (the edge target).

• target_id -- [in] Resolved row id of the consumer (the edge target).

Variables

String _KNIT_PROV_TABLE

Name of the framework-owned edge table that records the relationships between invocations (the provenance graph). The "__" prefix/suffix marks it as reserved (not a user-provided command table), like other reserved names.

String _KNIT_PROV_TABLE_ENSURED

Set to "1" once the provenance table has been ensured in this process (see _knit_prov_ensure_table), so the idempotent CREATE runs at most once per run.

query.sh

Functions

int _knit_query_annotate_catalog()

Filter the raw query catalog listing read from standard input, annotating it with two things read from the live schema: each table <name> line whose table has a distinct command-name alias gains " (command: <name>)" so users discover both spellings of a label, and each column <name> line gains " (<TYPE>)" with the column's SQL storage type. Column types are looked up once per table (through the stubbable _knit_query_column_types helper) as the table line is seen. A TABLE.COLUMN validation line (a single line with no table / column prefix) is passed through unchanged.

int _knit_query_build_lens_preamble()

Build the SQL preamble that turns a _knit_sqlite3 session into the query lens: a read-only union over the resolved lens databases. The first database is the session's own main schema (it is _KNIT_DATABASE, which _knit_sqlite3 opens), so only the extra databases are ATTACHed, each read-only as p1, p2, ... . For every command table found in any lens database, a CREATE TEMP VIEW "<table>" unions that table across every database that has it (a database that lacks the table simply contributes no arm). Columns are reconciled from each database's schema: the view's column list is the union of the arms' columns, and an arm fills a column it lacks with NULL AS "<col>", so script or knit-version drift does not break the union.

Two framework tables are handled specially rather than unioned raw. A synthesized platforms view holds one row per lens database, read from that database's metadata (the platform name as id, the fingerprint keys as columns), with a plain UNION so identical platforms collapse. The __provenance__ view unions every database's real edges and adds a synthesized executed edge from each database's platform to every node-table row on it, so a query can go (p:platform)-[:executed]->(n).

The returned text is meant to be prepended, in a single _knit_sqlite3 invocation, to the query that reads the views, because ATTACH and TEMP VIEW are session-scoped. The source databases are only read; the views live in the temp schema and vanish with the session.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the preamble SQL.

• ... -- [in] The lens database paths (index 0 = current/main, rest ATTACHed).

int _knit_query_build_names()

Build the name<->table map the transpiler needs (its --names SPEC) from the live registration state, returned through a caller-named variable. Every registered table contributes one table=command entry (the command being the demangled name knit stores in the provenance *_name columns); the transpiler resolves a node label through this map to the table it JOINs and the *_name value its edges carry (see the design's name<->table section). The map is rebuilt on every invocation and never persisted, so it can never go stale. Entries are sorted for a stable, traceable SPEC. Empty when no table is registered.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the newline-separated SPEC.

int _knit_query_build_schema()

Emit the flat text schema of the lens's union, for the pure transpiler to read on stdin. One line per table, a tab separating the table name from a comma-separated column list; the transpiler treats a table with an id column as a node table and __provenance__ as the edge table. Nothing is written to disk: the transpiler reads this text on stdin and never opens a database.

Every command table lists id first and its union columns after (a column already named id is not repeated); the platforms table (id plus the fingerprint columns) is emitted when any lens database has metadata; and the __provenance__ edge table is always emitted. Column types are not part of the flat form -- the transpiler needs only names and the id marker.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the flat schema text.

• ... -- [in] The lens database paths.

int _knit_query_catalog()

Body of 'query catalog': list the experiment database's tables and columns itself (from sqlite_master + PRAGMA table_info via _knit_sqlite3), annotated with command-name aliases and column types. With no --ref it lists every graph table and its columns; with a TABLE or TABLE.COLUMN reference in --ref it shows that table or validates the column, returning non-zero on an unknown reference. No external engine is used; the query itself is not recorded.

Parameters:

• ... -- [in] The command invocation arguments (an optional --ref TABLE[.COLUMN]).

Returns:

Non-zero on an unknown table/column reference.

int _knit_query_catalog_columns()

int _knit_query_catalog_graph_tables()

Print the names of every graph table in the database, sorted by name. A table is enumerated from sqlite_master (user tables only) and kept when _knit_query_catalog_is_graph_table accepts it.

int _knit_query_catalog_has_column()

Return success when the table has a column of the given name.

Parameters:

• table -- [in] The table name.

• want -- [in] The column name to look for.

int _knit_query_catalog_is_graph_table()

Return success when the named table exists and participates in the graph: it is the edge table __provenance__, or it has an id column (the uuid7 key that ties node rows to edges). This mirrors the node/edge classification the transpiler applies, so the catalog lists exactly the queryable entities.

Parameters:

• table -- [in] The table name to test.

int _knit_query_catalog_print_table()

Print one table's listing in the raw catalog format the annotator consumes: a table <name> line followed by one column <name> line per column.

Parameters:

• table -- [in] The table name to print.

int _knit_query_catalog_produce()

Produce the raw (un-annotated) catalog listing on standard output. With no reference, list every graph table and its columns, sorted by name. With a TABLE reference, list that one table. With a TABLE.COLUMN reference (split at the last dot), print the reference when the column exists. An unknown table or column is reported to stderr and returns non-zero, so the command exits with a failure status just as the engine's --catalog mode did.

Parameters:

• ref -- [in] Empty to list all; else TABLE or TABLE.COLUMN.

Returns:

Non-zero on an unknown table/column reference.

int _knit_query_cleanup_tmps()

Remove the temporary directories a lens created (bundle extraction). Empty entries are ignored. Called by a query body after the query has run.

Parameters:

• __knit_tmps -- [in] Name of the array of temporary directories to remove.

int _knit_query_column_types()

Fill a caller-named associative array with the column name -> SQL storage type map of a table, read from the live schema via sqlite's pragma_table_info. The type is what SQLite records in the schema (e.g. INTEGER, TEXT, REAL), which is the universal source: it also covers framework tables (jobs, runs, provenance) whose columns are not user-declared. Factored out so the catalog annotator can enrich each column line and so it is stubbable in unit tests. The array is cleared first; an unknown table simply yields no entries.

Parameters:

• __knit_ret -- [out] Name of the associative array to fill (name -> type).

• table -- [in] The table name to introspect.

int _knit_query_discover()

Register the query subcommands (catalog / graph / sql). Called lazily by the framework the first time a query subcommand is resolved, listed in "--help", or walked by "describe". The query_format enum used by --format is registered eagerly in src/ai.sh, so it is available when this runs.

Parameters:

• parent -- [in] The parent command's display name (unused).

int _knit_query_exec_over_lens()

Run one SQL statement over the query lens assembled from the already-resolved lens databases. The lens preamble (ATTACH + TEMP VIEWs) and the statement run in a single _knit_sqlite3 session, because the views are session-scoped; the statement refers to the lens views by their bare names. Both query sql and query graph route their --extra path through here (graph after transpiling its Cypher to SQL), so lens assembly and execution live in one place. Before running the query, a same-name/different-fingerprint platform mismatch is surfaced as a warning. The caller resolves the databases and cleans up any temporary directories afterward. The output flags are sqlite3 dot-commands, so a caller shapes the result the same way an ordinary single-database query does.

Parameters:

• __knit_dbs -- [in] Name of the array of lens database paths (main first).

• sql -- [in] The SQL to run over the lens.

• ... -- [in] sqlite3 output flags (mode/header/separator dot-commands).

Returns:

The exit status of sqlite3.

int _knit_query_graph()

The query_format enum (shared by 'ai query', 'query graph', and 'query sql') is defined in src/ai.sh, which loads before this file, so its type resolves when the format:query_format parameters below are declared.

Body of 'query graph': transpile the --exec Cypher statement to SQL with the pure transpiler (knit-cypher-to-sql), augmented with the live name<->table map (so a node label may be written as either its table name or its command name), then run that SQL over the lens and shape the output. With --explain the transpiled SQL is printed instead of run; with --ast the transpiler prints the parse tree (no schema, map, or output flags needed). --explain and --ast are mutually exclusive. Anything after a trailing -- is forwarded to the transpiler verbatim. Its exit status is propagated.

The query always spans a read-only lens over the current database plus any --extra sources (a lens over one database when there is no --extra). The transpiler is pure: it reads the lens's flat schema (whose union mirrors the lens, including the platforms node) on stdin and prints SQL; knit runs that SQL over the lens itself, so (p:platform) works with or without --extra and aggregation/ORDER BY/DISTINCT are correct across every platform. The names map gains a platforms=platform entry so (p:platform) resolves to the lens's synthesized platforms view. --explain prints that transpiled SQL. (--ast is the one path that does not build a lens: it needs no schema at all.)

Parameters:

• ... -- [in] The command invocation arguments, plus optional transpiler args after --.

Returns:

The exit status of the transpiler (--ast) or sqlite3 (the lens query).

int _knit_query_lens_schema()

Run one introspection pass over the lens and return two things the lens builders share: the ATTACH statements for the extra databases (the first database is the session's own main schema, so only the rest are attached, read-only as p1, p2, ...), and the raw schema|table|column rows for every table in every lens database. metadata and provenance are included in the rows so a caller can classify them as it needs. Rows are ordered by table, then schema, then column id, so a table's own-schema columns come before columns a drifted database adds. The file:...?mode=ro URI is the only way ATTACH opens a database read-only; paths from the resolver are plain filesystem paths, so only the SQL-literal quote needs escaping.

Parameters:

• __knit_ret1 -- [out] Name of the array to fill with the ATTACH statements.

• __knit_ret2 -- [out] Name of the variable to fill with the introspection rows.

• ... -- [in] The lens database paths (index 0 = current/main, rest ATTACHed).

int _knit_query_read_output_opts()

Read the OUTPUT-OPTS shared by knit query graph and knit query sql out of a command invocation into three caller-named variables. Factored here so both query commands parse --format/--header/--separator identically; both then translate the values into sqlite3 dot-commands (the lens runs every query through sqlite3). The format defaults to list and the header defaults OFF (query output is most often piped elsewhere, where a header is noise); the separator defaults to empty (the output mode's own default).

Parameters:

• __knit_ret1 -- [out] Name of the variable to hold the format value.

• __knit_ret2 -- [out] Name of the variable to hold the header flag ("true"/"false").

• __knit_ret3 -- [out] Name of the variable to hold the separator value.

• ... -- [in] The command invocation arguments to read the options from.

int _knit_query_resolve_extra()

Resolve the --extra source list into the ordered list of database files that form the query lens, together with the temporary directories the caller must remove after the query. Each comma-separated <src> is one of:

• a directory -> its <dir>/.knit/knit.db is used;

• a database file -> used as-is;

• a bundle (a .tar.gz or .tgz from knit bundle) -> extracted to a fresh temporary directory whose .knit/knit.db is used; the temporary directory is added to the cleanup list. The current experiment database (_KNIT_DATABASE) is always the first element, so an empty --extra yields exactly [current] and a single-database query is unchanged. Extra sources follow in the order given. A source that is neither a directory, a database file, nor a bundle, or one that resolves to a database that is not readable, is fatal. Empty fields (from a trailing comma) are skipped.

Parameters:

• __knit_ret1 -- [out] Name of the array to fill with database paths (current first).

• __knit_ret2 -- [out] Name of the array to fill with temporary dirs to remove.

• spec -- [in] The raw --extra value (comma-separated; empty for none).

int _knit_query_run_normalized()

Run a read query through _knit_sqlite3 with the given output flags and normalize the result's line endings. sqlite3's csv mode terminates rows with CRLF (RFC 4180) while every other output mode uses a bare LF; a stray trailing CR breaks a consumer that reads a value back as a typed result (e.g. a report feeding a "real"/"integer" knit_output), so the CRLF row terminators are rewritten to LF. Non-csv output has no CR, so the substitution is a no-op there. The CR is matched as a literal byte (ANSI-C quoting) so the pattern is portable across seds, and PIPESTATUS preserves sqlite3's own exit status across the filter. Shared by query sql/query graph (through _knit_query_exec_over_lens) and by ai query's standalone (no-lens) path.

Parameters:

• ... -- [in] _knit_sqlite3 arguments (output flags followed by the SQL).

Returns:

The exit status of sqlite3.

int _knit_query_sql()

Body of 'query sql': run the --exec SQL statement on knit's own read path (_knit_sqlite3) with the shared output options shaping the result. The statement is rejected unless it is read-only (leading SELECT/WITH/EXPLAIN/PRAGMA and no write keyword, via the shared guard) so a query can never mutate the provenance database. Output opts are read with the shared reader and translated to sqlite3 .mode/.headers/.separator dot-commands by the same helper ai query uses, so SQL and Cypher results present identically. sqlite3's exit status is propagated.

The statement always runs over a read-only lens spanning the current database plus any --extra sources (a lens over one database when there is no --extra), so the synthesized platforms view and its executed edges are queryable with or without --extra. It refers to tables by their bare names, which resolve to the lens's union views; the real metadata table is not shadowed, so it is still queryable directly.

Parameters:

• ... -- [in] The command invocation arguments.

Returns:

The exit status of sqlite3, or fatal on a non-read-only statement.

int _knit_query_table_alias()

Return, through a caller-named variable, the command name a table is registered under when it differs from the table name, or the empty string otherwise. A command that overrides its table with knit_with_table (e.g. submit -> jobs, submit:montecarlo -> montecarlo) has a distinct command-name alias; a command whose table is its own name (setups, downloads, plain commands) has none. Read from the live registration state (_KNIT_DB_REGISTERED_TABLES), so it can never go stale.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the alias (empty if none).

• table -- [in] The table name to look up.

int _knit_query_warn_fingerprint_mismatch()

Warn when two lens databases claim the same platform name with different fingerprints. Such a pair survives the platforms view's plain UNION as two rows with the same id, so a duplicated id is exactly the mismatch. Each mismatched platform is reported once; nothing is dropped, so the query still spans every database. An unnamed platform (empty id) is not a same-name claim and is ignored. The project name is deliberately not checked (a platform may be named per machine). Best effort: a failed check query is silent, never fatal.

Parameters:

• preamble -- [in] The lens preamble (which creates the platforms view).

remove.sh

Functions

int _knit_remove_append_failed_ids()

Append the failed row ids found in the given tables to a caller-named array. A table without the reserved "__exit_status__" column is skipped; from the rest only the rows whose status is a non-zero integer are selected. NULL (outcome not recorded, e.g. an eager row) and the empty migration back-fill (unknown) are excluded, so only a genuine non-zero exit is selected. The array is appended to, never reset. Shared by _knit_remove_failed_ids (all tables) and _knit_remove_failed_ids_of_kind (one kind's tables).

Parameters:

• __knit_ret -- [out] Name of the array to append the failed row ids to.

• ... -- [in] The table names to scan.

int _knit_remove_append_ids()

Run a read-only SELECT and append each non-empty result line to a caller-named array. Errors (e.g. a table that was registered but never created) are silenced and simply yield no rows, so a probe against a not-yet-materialized table is a no-op rather than a failure. The array is appended to, never reset, so several calls can accumulate matches (used when a selector spans more than one table).

Parameters:

• __knit_ret -- [out] Name of the array to append the selected ids to.

• sql -- [in] The SELECT statement to run.

int _knit_remove_artifact()

Body of 'remove artifact': delegate to the shared dispatch (resolve, close, refuse, report, confirm, and delete) for the artifact kind. Naming an artifact on its own is refused (its producer is kept); it is meaningful only with --from-root, which erases the whole lineage.

Parameters:

• ... -- [in] The command invocation arguments.

int _knit_remove_build_report()

Assemble the itemized removal report from the erase set and its maps, filling four caller-named arrays that _knit_remove_print_report lays out. All database reads and formatting happen here, so the printer is pure layout:

• the data-row lines, one per erase-set id in order, each "<kind> <label>

  <id>" with a short annotation. A setup or resource shows "<type> (<name>)"; a job submission shows its state; a job body row is marked as such; a run shows its launched app; an app or plain command shows its command name; an artifact shows its path.

• the provenance-edge lines: every provenance edge with an endpoint in the set (exactly the edges the deletion removes, including a kept provider's used_by edge into an erased consumer).

• the on-disk directories and artifact entries the removal deletes: when keep_mode is not "files", each job directory and each setup/resource instance directory, and -- when keep_mode is "none" -- each artifact entry under the artifact root. Under "files" this list is empty: the removal deletes nothing on disk.

• the "left on disk" lines: the plain non-artifact outputs (always, attributed to the owning command); when keep_mode is "artifacts", the kept artifact entries; and when keep_mode is "files", every job/setup/resource directory and artifact entry too, each tagged by kind (nothing is removed).

Parameters:

• __knit_ret1 -- [out] Name of the array to fill with data-row lines.

• __knit_ret2 -- [out] Name of the array to fill with provenance-edge lines.

• __knit_ret3 -- [out] Name of the array to fill with removed dir/entry paths.

• __knit_ret4 -- [out] Name of the array to fill with "left on disk" lines.

• __knit_tables -- [in] Name of the assoc array mapping id -> table.

• __knit_kinds -- [in] Name of the assoc array mapping id -> kind.

• __knit_apaths -- [in] Name of the assoc array mapping artifact id -> path.

• __knit_plain -- [in] Name of the assoc array mapping plain output path -> command.

• keep_mode -- [in] Removal policy: "none" removes everything, "artifacts" keeps artifact entries, "files" keeps the whole tree.

• ... -- [in] The erase-set ids, in order.

int _knit_remove_check_refusal()

Enforce the callee/artifact refusal for the default (downward) closure. For each originally selected id, walk the incoming "call"/"produced" edges BACKWARD from it and refuse the whole operation on reaching a caller or producer that is kept (not in the erase set) AND owns a data row -- a persistent entity whose recorded provenance would otherwise be left referring to a row that is gone. Erasing a callee while such a caller stays (or an artifact while its producer stays) is rejected.

The walk passes THROUGH table-less dispatcher frames: the "setup", "fetch", and "submit" dispatchers each record a "call" edge into the body row they create but own no table of their own, so such a frame is not itself a kept caller and the walk continues up from it. A branch that reaches the root (an empty source) or an ancestor already in the erase set first is not a refusal. This is what lets "remove setup"/"remove resource" work by default: the only "call" edge into a freshly built setup or resource body comes from its dispatcher, above which sits the root. It still refuses when a user's own table-backed command (e.g. one that itself calls "knit setup" to build several setups) is the kept caller, because that command opted into provenance by declaring a table -- the walk climbs past the dispatcher to it and points the user at removing THAT command or passing --from-root. A table-less helper command, by contrast, is walked through like a dispatcher, so setups it created can be removed independently.

A "call" refusal points the user at removing the caller (with the caller's own remove subcommand) or passing --from-root; a "produced" refusal (a bare remove:artifact) points at --from-root, which pulls the producer in and erases the whole lineage.

Parameters:

• __knit_selected -- [in] Name of the array of originally selected ids.

• __knit_erase -- [in] Name of the array holding the full erase set.

Returns:

Fatal on the first refusal; otherwise 0.

int _knit_remove_check_terminal_jobs()

Enforce that every job in the erase set has finished before anything is deleted. For each erase-set id whose table is the jobs table (read from the id -> table map _knit_remove_map_ids produced), the job's "state" column is read; a state other than "completed" or "killed" (that is, "submitted", "running", or "prepared") refuses the whole operation and points the user at "job cancel <id>" to stop or tear down the job first. There is no --force override: a live job must be cancelled before it can be erased. The refusal fires whether the job was named directly or pulled into the set by a cascade (a setup or resource it used being erased), because both reach it through the same erase-set membership.

Parameters:

• __knit_tables -- [in] Name of the assoc array mapping id -> table.

• ... -- [in] The erase-set ids.

Returns:

Fatal on the first non-terminal job; otherwise 0.

int _knit_remove_closure_downward()

Compute the default downward erase set from a set of starting ids, returned through a caller-named array. Starting from each id, the closure follows every outgoing provenance edge (source -> target) of the three edge types to a fixed point: a "call" edge (a caller owns its callees), a "produced" edge (a producer owns its artifacts), and a "used_by" edge from a provider (a setup or resource owns its consumers). Because the walk only follows edges where the current id is the SOURCE, a "used_by" edge is never followed backward into a provider (which would be the target), so erasing a consumer leaves its setup/resource intact. An associative visited set guards against revisiting an id (and against any accidental cycle from bad data), so each reachable id appears once.

Parameters:

• __knit_ret -- [out] Name of the array to fill with the erase-set ids.

• ... -- [in] The starting ids.

int _knit_remove_closure_from_root()

Compute the whole-lineage erase set that --from-root selects, returned through a caller-named array: the connected component of the starting ids in the subgraph made of "call" and "produced" edges only, traversed in BOTH directions. From each id the walk adds every neighbour across a call/produced edge, backward (target -> source: the caller that invoked it, or the producer that made it) and forward (source -> target: everything it called or produced). Iterated to a fixed point, the two directions climb to the root of the tree and back down over the whole tree, so pointing at any row (a callee, a run, an app, or an artifact) names the same lineage. "used_by" edges are NEVER queried, in either direction, so a setup or resource used by the tree is left intact (it belongs to its own tree). An associative visited set guards against revisiting an id, so each id appears once. There is no refusal check in this mode: pulling in the kept caller/producer is exactly what --from-root is for.

Parameters:

• __knit_ret -- [out] Name of the array to fill with the erase-set ids.

• ... -- [in] The starting ids.

int _knit_remove_command()

Body of 'remove command': delegate to the shared dispatch (resolve, close, refuse, report, confirm, and delete) for the command kind (also covers wrappers).

Parameters:

• ... -- [in] The command invocation arguments.

int _knit_remove_confirm()

Print the removal report and decide whether the erase may proceed. The report is always printed, so a run always leaves a record of what it did (or would do); the header tense matches the mode. With yes == "true" the past-tense header "Erased:" is printed and the function returns success without prompting. Otherwise the future-tense header "The following will be permanently erased:" is printed and the user is prompted: one line is read (read -r), and only "y", "Y", or "yes" proceeds; anything else (a bare Enter is the default No) prints "Aborted." and returns failure, which the caller treats as a clean decline (exit 0), not an error. When stdin is NOT an interactive terminal and yes was not given, the prompt cannot be answered, so rather than block on a read that never returns the function is a fatal refusal that tells the user to pass --yes for non-interactive use. The four report arrays are passed BY NAME and forwarded verbatim to _knit_remove_print_report.

Parameters:

• yes -- [in] "true" to skip the prompt (the report is still printed).

• rows_name -- [in] Name of the array of data-row lines.

• edges_name -- [in] Name of the array of provenance-edge lines.

• removed_name -- [in] Name of the array of removed dir/entry paths.

• left_name -- [in] Name of the array of "left on disk" lines.

Returns:

0 to proceed; 1 on an interactive decline. Fatal on a non-terminal stdin without --yes.

int _knit_remove_declare_failed_flag()

Declare the --failed filter flag. It is separate from _knit_remove_declare_flags because it is offered only by the kinds that carry an exit status (setup, resource, job, run, command) and NOT by "remove artifact" (an artifact is produced, not invoked, so it has no exit status). Call it between knit_register and knit_done.

int _knit_remove_declare_flags()

Declare the flags shared by every remove subcommand. Call it between knit_register and knit_done.

int _knit_remove_declare_selectors()

The remove command group erases recorded entities (a setup, a resource, a job, a run, an app invocation, a plain command invocation, or an artifact) together with their provenance edges and, transitively, everything downstream of them. It reads and writes the experiment database, so every subcommand is a post-bootstrap builtin: it is NOT marked knit_usable_before_bootstrap (the central runtime guard refuses it until the experiment is bootstrapped, which is also what makes the --when selector constraints legal), it declares no table with knit_with_table (remove must never record a row it is meant to be deleting), and it runs knit_without_provenance so it emits no call edge of its own.

This file currently provides the command surface, selection resolution, both closure modes, and the erase-set mapping: registration, the shared selector/flag declarations, the resolvers that turn a selector into the set of starting row ids, the fixed-point downward closure over the provenance graph, the callee/artifact refusal check that rejects erasing a callee (or artifact) whose caller (or producer) is kept, the whole-lineage closure that --from-root selects (the connected component over call/produced edges, traversed both directions, with no refusal check), the mapping that resolves every erase-set id to its (table, kind) and reads each artifact's on-disk path and type before the row is gone, the collector that finds the plain (non-artifact) file/directory outputs remove leaves on disk, the non-terminal-job refusal that rejects erasing a job that has not finished, the report builder and printer that render the itemized plan (the data rows, the provenance edges, the on-disk directories and artifact entries to remove, and what is left on disk), and bodies that validate the exactly-one-selector contract, resolve the selection, compute whichever closure the flags request, run the refusal check in the default mode, map the erase set, refuse a non-terminal job, and either print the report and stop under --dry-run, print the report and delete the rows and files under --yes, or print the resulting erase set. The deletion transaction removes, in one atomic BEGIN...COMMIT, every provenance edge touching the erase set and then the data rows table by table. After a successful commit the filesystem phase removes, best effort, each erased job/setup/resource instance directory (setups and resources only when their on-disk marker still names the erased row) and -- unless --keep-artifacts -- each artifact entry under the artifact root, and then reports the files it deliberately left on disk (a command's plain outputs, and kept artifact entries) and exits non-zero listing anything a removal could not clear. --keep-files is stronger: it erases only the database rows and edges and makes no filesystem changes at all (every directory, artifact, and output stays on disk). Unless --yes or --dry-run is given the report is printed and the user is prompted before anything is deleted (a non-interactive stdin without --yes is a fatal refusal), so a removal never happens without either explicit confirmation or --yes.

Declare a remove subcommand's selector parameters, each as an optional string with a --when constraint that enforces mutual exclusion: a selector applies only when every other selector of the same subcommand is empty, so providing two at once is fatal. This makes the exactly-one contract's exclusion half declarative; the presence half (rejecting zero selectors) is a body check (_knit_remove_require_one_selector). Call it between knit_register and knit_done, so the knit_with_optional calls attach to the command being registered.

Parameters:

• kind -- [in] The entity kind, used in the parameter descriptions.

• ... -- [in] The selector names to declare (a subset of id, name, type, group, path).

int _knit_remove_delete_rows()

Delete the erase set from the database in one transaction: first every provenance edge that touches the set, then the data rows table by table. The whole thing is a single ".bail on" BEGIN...COMMIT fed to _knit_sqlite3_write, so it runs under the advisory write lock and either the whole erase set goes or nothing does (a failing statement stops the CLI and rolls the open transaction back). Edges are deleted with "source_id IN (...) OR target_id IN (...)" over every erase-set id, which also clears a KEPT provider's used_by edge into an erased consumer (the edge is caught by its erased target even though its source stays). Data rows are grouped by table from the id -> table map so each table is cleared with one DELETE. Ids are escaped into the IN-lists by _knit_remove_id_in_list, never interpolated raw. Filesystem side effects (job, setup, and resource directories and artifact entries) are handled separately after a successful commit.

Parameters:

• __knit_tables -- [in] Name of the assoc array mapping id -> table.

• ... -- [in] The erase-set ids.

int _knit_remove_discover()

Register the remove subcommands (setup / resource / job / run / command / artifact). Called lazily by the framework the first time a remove subcommand is resolved, listed in "--help", or walked by "describe".

Parameters:

• parent -- [in] The parent command's display name (unused).

int _knit_remove_dispatch()

Shared body for every remove subcommand: enforce the exactly-one-selector contract, resolve the selection to its starting ids, compute the erase set, map each id to its table and kind, and refuse the operation if it would erase a non-terminal job. Which closure is computed depends on --from-root: without it, the default downward closure is taken and the callee/artifact refusal check runs (fatal before anything is printed or deleted); with it, the whole-lineage connected-component closure is taken and the refusal check is skipped by design. Under --dry-run the itemized report is built and printed and the command stops (no prompt, no deletion). Otherwise the report is printed and, unless --yes was given, the user is prompted to confirm (a non-interactive stdin without --yes is a fatal refusal; an interactive decline stops with exit 0). Once confirmed (or with --yes), the erase set is deleted from the database (edges and rows, in one transaction), the framework-managed on-disk state is removed best-effort (job/setup/resource directories and, unless --keep-artifacts, artifact entries; --keep-files removes nothing on disk), and the files left on disk are reported; the command exits non-zero if any attempted removal could not be cleared. --dry-run wins if both flags are given. The selector names are given as leading arguments up to a literal "--", after which come the command invocation arguments.

The --failed filter, when the subcommand offers it, narrows the starting set: with a selector it keeps only the failed rows the selector chose (an intersection); on its own it starts from every failed row of the kind. A selector that matches nothing is still fatal (it is resolved before the filter); a non-empty selection with no failures, or --failed alone with no failures, is an info message and exit 0, never a deletion.

Parameters:

• kind -- [in] The entity kind of the subcommand.

• ... -- [in] The selector names, then "--", then the invocation arguments.

int _knit_remove_erase_selection()

The shared erase flow, given a set of starting ids and the invocation arguments. Used by every selector-based subcommand (through _knit_remove_dispatch) and by "remove --failed". It computes the erase set (downward by default, whole-lineage under --from-root, with the callee/artifact refusal check in the default mode), maps each id to its table and kind, refuses a non-terminal job, builds the itemized report, and — unless --dry-run, or a declined prompt without --yes — deletes the rows and edges in one transaction and clears the framework-managed files. Returns non-zero if any attempted filesystem removal could not be cleared.

Parameters:

• ids_name -- [in] Name of the array holding the starting ids.

• ... -- [in] The command invocation arguments (flags read via knit_get_parameter).

int _knit_remove_failed_ids()

Collect the row ids of every failed invocation across the database: every table that carries the reserved "__exit_status__" column, selecting the rows whose status is a non-zero integer. The ids are returned through the caller-named array; an experiment with no failures yields an empty array. Bootstrap-gated: with no database there is nothing to scan.

Parameters:

• __knit_ret -- [out] Name of the array to fill with the failed row ids.

int _knit_remove_failed_ids_body_mapped()

The failed launcher/submission ids for a body-table kind (job, run). The numeric exit status of a job or a run lives in its per-body table, not in the jobs/runs launcher table (which is carved out of exit-status recording), so this finds the failed rows in the body tables of the given body kind ("job" bodies for a job, "app" bodies for a run) and maps each up to the launcher/submission that called it, via the call edge (source invoked target). The mapping is the same one _knit_remove_resolve_by_type uses to turn a body table into its submissions. A job/run whose body returned zero (a tolerated failure inside it) is therefore not itself failed. Bootstrap-gated.

Parameters:

• __knit_ret -- [out] Name of the array to fill with launcher/submission ids.

• body_kind -- [in] The owning-command kind of the body tables ("job"/"app").

int _knit_remove_failed_ids_of_kind()

Collect the failed row ids of ONE kind: the failed rows found in the tables that hold rows of <kind> (via _knit_remove_tables_of_kind), so it is the per-kind restriction of _knit_remove_failed_ids. It works directly for the kinds whose own tables carry the exit status (setup, resource, command). The body-table kinds (job, run) whose exit status lives in a per-body table read through a different path (see the dispatch). Bootstrap-gated.

Parameters:

• __knit_ret -- [out] Name of the array to fill with the failed row ids.

• kind -- [in] The entity kind to scan.

int _knit_remove_filesystem()

Remove the framework-managed on-disk state of the erase set, best-effort, after the deletion transaction has committed. For every erase-set id, by kind:

• job: its job directory <job-root>/<id> is removed;

• setup: its instance directory <setup-root>/<name> is removed ONLY when the directory's .setup.id marker names this row -- so erasing an older historical build never removes a newer kept build's live directory (both share the instance name; only one owns the directory);

• resource: its instance directory <resource-root>/<name> and its sibling sidecar markers (.<name>.resource.{type,source,id}) are removed under the same owns-the-dir guard (the .<name>.resource.id sidecar names the row);

• artifact: when keep_mode is "none", its on-disk entry <artifact-root>/<path> is removed (symlink-safe, containment-guarded, empty parents pruned -- see _knit_remove_rm_artifact); "artifacts" keeps it. When keep_mode is "files" this function touches nothing on disk and returns at once: every directory and artifact entry is kept. A command's plain (non-artifact) file/directory outputs are NEVER removed here; the caller lists the surviving ones with _knit_remove_report_left. Each entry whose removal was ATTEMPTED but did not clear is logged and its path is appended to a caller-named failures array, so the caller can print a final "remove by

  hand" list and exit non-zero; a deliberately kept file (a plain output, or an artifact under a keep_mode that spares it) is not a failure. The setup/resource instance names come from the map _knit_remove_instance_names read before deletion, and the artifact paths from the map _knit_remove_map_ids read before deletion.

Parameters:

• __knit_ret -- [out] Name of the array to append unremovable paths to.

• __knit_kinds -- [in] Name of the assoc array mapping id -> kind.

• __knit_apaths -- [in] Name of the assoc array mapping artifact id -> path.

• __knit_names -- [in] Name of the assoc array mapping id -> instance name.

• keep_mode -- [in] Removal policy: "none" removes directories and artifact entries, "artifacts" keeps artifact entries, "files" touches nothing on disk.

• ... -- [in] The erase-set ids.

int _knit_remove_id_in_list()

Build a SQL IN-list -- a comma-separated sequence of single-quoted, escaped ids such as 'a','b','c' -- from a set of ids, returned through a caller-named variable (empty when no non-empty id is given). Every id is escaped with _knit_sql_escape, never interpolated raw. The report builder uses it to select the provenance edges touching the erase set; the deletion phase reuses it.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the IN-list text.

• ... -- [in] The ids.

int _knit_remove_id_table()

Find which table a row id lives in, returned through a caller-named variable, by probing every registered table for the id. Row ids are globally unique across tables, so the first hit is authoritative. A table that was registered but never created is probed harmlessly (the error is silenced). The empty string means the id was not found in any table.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the table name (empty if none).

• id -- [in] The row id to locate.

int _knit_remove_instance_names()

Read the instance name of every setup and resource row in the erase set into a caller-named associative array (id -> name), from the "name" column each such row records. It must run BEFORE the rows are deleted, because the filesystem phase needs the names to locate the instance directories (setups/<name>, resources/<name>) after the rows are gone, exactly as the artifact path is read before deletion. Ids of other kinds are skipped (a job directory is named by its id, an artifact by its path), as is any row whose name column is empty.

Parameters:

• __knit_ret -- [out] Name of the assoc array to fill (id -> instance name).

• __knit_kinds -- [in] Name of the assoc array mapping id -> kind.

• __knit_tables -- [in] Name of the assoc array mapping id -> table.

• ... -- [in] The erase-set ids.

int _knit_remove_intersect()

Fill a caller-named array with the intersection of two id sets: the ids that appear in both. Order follows the first set; duplicates and empty ids are ignored. Used to combine a selector's rows with the failed rows of the kind.

Parameters:

• __knit_ret -- [out] Name of the array to fill with the intersection.

• __knit_a -- [in] Name of the first id array.

• __knit_b -- [in] Name of the second id array.

int _knit_remove_job()

Body of 'remove job': delegate to the shared dispatch (resolve, close, refuse, report, confirm, and delete) for the job kind. The setup and resource the job used are not followed (their used_by edges point INTO the job).

Parameters:

• ... -- [in] The command invocation arguments.

int _knit_remove_map_ids()

Map every erase-set id to the table it lives in and the entity kind of that table, returned through two caller-named associative arrays (id -> table and id -> kind). For each id whose kind is "artifact" it additionally reads that row's on-disk path and type into two more caller-named associative arrays (id -> path and id -> type), because the filesystem phase needs them after the row itself has been deleted. An id whose table cannot be located (its owning command is not registered in the current script, so its table is absent from the registry _knit_remove_id_table probes) is skipped: it cannot be classified or grouped for deletion here. Row ids are globally unique across tables, so the id -> table map doubles as the group-by-table index the delete phase needs.

Parameters:

• __knit_ret1 -- [out] Name of the assoc array to fill (id -> table).

• __knit_ret2 -- [out] Name of the assoc array to fill (id -> kind).

• __knit_ret3 -- [out] Name of the assoc array to fill (artifact id -> path).

• __knit_ret4 -- [out] Name of the assoc array to fill (artifact id -> type).

• ... -- [in] The erase-set ids.

int _knit_remove_plain_outputs()

Collect the plain (non-artifact) file/directory OUTPUTS of every erased row into a caller-named associative array (recorded path -> owning command name), reading the id -> table map knit_remove_map_ids produced. A plain output is a file/directory output a command declared that is NOT an artifact: for each row, the owning command's _KNIT_CMD<cmd>fileparams entries whose marker direction is "output" and whose name is not in _KNIT_CMD<cmd>_artifacts; the recorded path is that row's <name> column. Input file/directory parameters are ignored (they are the run's inputs, not its outputs), and so are artifacts (their on-disk entries are handled by the filesystem phase). The framework tables (jobs/runs/artifacts) carry no user-declared file outputs and are skipped, as is any row whose owning command is no longer registered in the current script -- its file-parameter markers are absent, so its columns cannot be classified. remove never deletes these paths; the caller lists them under "Left on disk".

Parameters:

• __knit_ret -- [out] Name of the assoc array to fill (path -> command).

• __knit_tables -- [in] Name of the assoc array mapping id -> table.

• ... -- [in] The erase-set ids.

int _knit_remove_print_report()

Lay out the itemized report the erase set built. The header line is a parameter so the tense matches the mode ("The following will be permanently erased:" before a prompt or under --dry-run; "Erased:" after the fact under --yes). The data-row section is always printed (the erase set is never empty); the provenance-edge, removed, and "left on disk" sections are printed only when they have entries. Each section header carries its count.

Parameters:

• header -- [in] The header line to print above the report.

• __knit_rows -- [in] Name of the array of data-row lines.

• __knit_edges -- [in] Name of the array of provenance-edge lines.

• __knit_removed -- [in] Name of the array of removed dir/entry paths.

• __knit_left -- [in] Name of the array of "left on disk" lines.

int _knit_remove_registry_tables_of_kind()

Fill a caller-named array with every NON-framework table in the live registry whose owning command's kind matches. This is the registry-scan half of _knit_remove_tables_of_kind (which short-circuits the framework kinds to their single table); it is exposed on its own so the --failed path can enumerate the per-body tables that carry the exit status: the per-job body tables (kind "job", the jobs framework table excluded) and the per-app body tables (kind "app").

Parameters:

• __knit_ret -- [out] Name of the array to fill with table names.

• kind -- [in] The owning-command kind to match.

int _knit_remove_report_left()

Print the "NOT removed" report after an actual removal: the files and directories remove deliberately left in place that STILL EXIST. In order:

• when keep_mode is "files", every job/setup/resource directory the removal left on disk, tagged by kind as "(<kind> directory, --keep-files)".

• every plain (non-artifact) file/directory output remove never deletes, attributed to its owning command as "(output of <command>)". A plain output that lived inside a removed job directory is already gone and is not listed.

• when keep_mode is "artifacts" or "files", every kept artifact entry <artifact-root>/<path>, marked "(artifact, <flag>)" where <flag> is the flag that kept it. Only entries that still exist are listed, so this is a truthful record of what survived. When nothing survived, nothing is printed. This report is informational and does not itself set the exit code; a removal that FAILED (as opposed to a deliberately kept file) is handled separately by the caller from the failures _knit_remove_filesystem collects.

Parameters:

• __knit_kinds -- [in] Name of the assoc array mapping id -> kind.

• __knit_apaths -- [in] Name of the assoc array mapping artifact id -> path.

• __knit_plain -- [in] Name of the assoc array mapping plain output path -> command.

• __knit_names -- [in] Name of the assoc array mapping setup/resource id -> name.

• keep_mode -- [in] Removal policy: "artifacts" kept artifact entries; "files" kept every directory and artifact on disk.

• ... -- [in] The erase-set ids.

int _knit_remove_require_one_selector()

Enforce the presence half of the exactly-one-selector contract: at least one of the named selectors must be provided. On a subcommand that offers the --failed filter, --failed also satisfies presence on its own (it selects every failed row of the kind), so the all-empty case is refused only when --failed is absent too. The mutual-exclusion half (at most one selector) is enforced declaratively by the --when constraints the selectors carry, and --failed is orthogonal to it (a plain flag with no --when), so a selector together with --failed is allowed. The selector names are given as leading arguments up to a literal "--", after which come the command invocation arguments.

Parameters:

• ... -- [in] The selector names, then "--", then the invocation arguments.

Returns:

Fatal if neither a selector nor --failed was provided; otherwise 0.

int _knit_remove_resolve_by_group()

Resolve a --group selector (remove:job) to a starting id set: every job whose "group" column equals the given name (a whole prepare batch). The reserved "group" identifier is quoted. No match is fatal.

Parameters:

• __knit_ret -- [out] Name of the array to fill with the starting ids.

• group -- [in] The group name to resolve.

int _knit_remove_resolve_by_id()

Resolve a --id selector to a one-element starting id set, verifying the id exists and belongs to the expected kind. A missing id is fatal; an id that exists but in a table of a different kind is fatal with a hint to use the right subcommand (e.g. remove:setup --id given a job id).

A run is one unit: its launch row (kind "run") and the app row it called (kind "app") are both valid targets of "remove run --id", mirroring how a job's submission and body rows are both kind "job". So an app id is accepted for the run kind. The "app" kind has no subcommand of its own (it is erased through "remove run"), so a wrong-kind hint that would otherwise name "remove app" names "remove run" instead.

Parameters:

• __knit_ret -- [out] Name of the array to fill with the starting id.

• kind -- [in] The expected entity kind.

• id -- [in] The row id to resolve.

int _knit_remove_resolve_by_name()

Resolve a --name selector (the instance name given at creation) to a starting id set. For setup/resource the name is scanned across every table of that kind via the "name" column, so it resolves from the database even after the instance directory is gone. For a job the name is the "jobs.name" alias. Only the kinds with a genuine instance name use --name; run and command have no instance name and select by --type instead (the launched app / the command's own table), resolved in _knit_remove_resolve_by_type. No match is fatal.

Parameters:

• __knit_ret -- [out] Name of the array to fill with the starting ids.

• kind -- [in] The entity kind.

• name -- [in] The instance/command name to resolve.

int _knit_remove_resolve_by_path()

Resolve a --path selector (remove:artifact) to a starting id set. The artifacts "path" column is UNIQUE, so at most one row matches. No match is fatal.

Parameters:

• __knit_ret -- [out] Name of the array to fill with the starting id.

• path -- [in] The artifacts-relative path to resolve.

int _knit_remove_resolve_by_type()

Resolve a --type selector (every instance of a type) to a starting id set. For setup/resource the type is the per-command table (setup:<type> / resource:<type>), so this selects every row in it. For a job the type is the job-body table; the starting ids are the job submissions (jobs rows) whose body rows live in that table, reached by the "call" edge from the submission to its body. For a run the type is the launched app: the runs rows whose "app" column matches (their app body rows follow through the run -> run:<app> call edge in the closure). For a plain command the type is the command's own table, selected whole. No match is fatal.

Parameters:

• __knit_ret -- [out] Name of the array to fill with the starting ids.

• kind -- [in] The entity kind (setup, resource, job, run, or command).

• type -- [in] The type to resolve.

int _knit_remove_resolve_selection()

Resolve a remove subcommand's selection into its starting id set, returned through a caller-named array. Reads whichever selector was provided (the exactly-one contract is already enforced by the --when constraints and the body presence check) and dispatches to the per-selector resolver for the kind. Every resolver is fatal on zero matches; one or many matches all become starting ids.

Parameters:

• __knit_ret -- [out] Name of the array to fill with the starting ids.

• kind -- [in] The entity kind of the subcommand.

• ... -- [in] The command invocation arguments (the selectors are read here).

int _knit_remove_resource()

Body of 'remove resource': delegate to the shared dispatch (resolve, close, refuse, report, confirm, and delete) for the resource kind.

Parameters:

• ... -- [in] The command invocation arguments.

int _knit_remove_rm_artifact()

Remove one on-disk artifact entry at <root>/<rel> best-effort, with a containment guard and empty-parent pruning. An absent entry is a success. The entry itself is never resolved, so a --link-from artifact (a symlink under the artifact root) is removed as the link and a target outside the root is left untouched; only the entry's PARENT is resolved, and the removal is refused (silently, as a no-op) unless that parent stays inside the artifact root -- a cheap defense, since the recorded path is framework-written and always relative. After a successful removal, now-empty parent directories are pruned up to but not including the artifact root, so the root does not accumulate empty subtrees.

Parameters:

• root -- [in] The resolved artifact root.

• rel -- [in] The artifacts-relative path recorded for the entry.

Returns:

0 if the entry is gone (or was absent, or outside the root); 1 if it remains.

int _knit_remove_rmtree()

Remove a single filesystem entry (a file, directory, or symlink) best-effort. An absent entry is a success (nothing to do). A symlink is removed as the link, never followed to its target (rm -rf never deletes a symlink's target). The return code reports the OUTCOME, not rm's exit status: 0 when the entry is gone afterward (or was never there), 1 when it still exists, so the caller can list a path it could not clear.

Parameters:

• path -- [in] The entry to remove.

Returns:

0 if the entry is gone (or was absent); 1 if it remains.

int _knit_remove_row_value()

Read a single column of a single row, identified by id, into a caller-named variable (empty when the row, the column, or the table is absent -- the error is silenced). The report builder uses it to fetch a display field for one erase-set row: a setup or resource instance name, a job name or state, or a run's launched app.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the column value.

• table -- [in] The table to read from.

• id -- [in] The row id.

• column -- [in] The column name to read.

int _knit_remove_run()

Body of 'remove run': delegate to the shared dispatch (resolve, close, refuse, report, confirm, and delete) for the run kind. A run is one unit: the launch row (runs) and the app row it called (reached through the run -> run:<app> call edge in the downward closure), mirroring how 'remove job' erases the submission and its body. Selecting a run whose enclosing job is kept is refused (the job's call edge would dangle).

Parameters:

• ... -- [in] The command invocation arguments.

int _knit_remove_setup()

int _knit_remove_table_kind()

Resolve a table name to the remove entity kind of the rows it holds, returned through a caller-named variable. The framework tables map directly (jobs -> job, runs -> run, artifacts -> artifact); any other table maps through its owning command's KNIT_CMD<cmd>_type marker (setup / resource / app, or command for a plain command). A wrapper table is reported as "command" because remove:command covers wrappers. An unregistered table (no owning command) yields the empty string.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the kind (empty if unknown).

• table -- [in] The table name to classify.

int _knit_remove_tables_of_kind()

Fill a caller-named array with every table that holds rows of a given entity kind. The framework kinds resolve to a single table each (job -> jobs, run -> runs, artifact -> artifacts); the other kinds (setup, resource, app, command) are gathered from the live table registry, skipping the framework tables (which carry their own kinds) and keeping only those whose owning command's kind matches. This is what lets a setup/resource --name scan every table of that kind.

Parameters:

• __knit_ret -- [out] Name of the array to fill with table names.

• kind -- [in] The entity kind to gather tables for.

int _knit_remove_toplevel()

Body of the top-level "remove" command. With --failed it erases every failed invocation (a non-zero recorded exit status) and, under --from-root, the whole lineage that contains them — so a job is erased because a run inside it failed. Without --failed there is nothing to do (a subcommand such as "remove job" dispatches before this body ever runs): point the user at the subcommands and the flag.

Parameters:

• ... -- [in] The command invocation arguments.

resource.sh

Functions

int _knit_command_is_resource()

Test whether a command is a resource type, i.e. it was registered with knit_register_resource. Used by the resource declaration directives to reject use on any other kind of command. Reads the command kind from the KNIT_CMD<cmd>_type field (see knit_register).

Parameters:

• cmd -- [in] Command (mangled name) to test.

Returns:

0 if the command is a resource type, 1 otherwise.

int _knit_fetch()

Body of the builtin fetch dispatcher command. It materializes a named resource instance at <resource-root>/<name> by dispatching to the requested resource type, mirroring knit setup:

./exp.sh fetch --name <name> [--ignore-checksum] -- <resource-type> [args...]

The instance name is validated as a single path component and the type must be a registered resource type. Fetching is idempotent by name and serialized on a per-name lock under .knit: if the instance already exists with a matching source identity the fetch is a no-op that just reprints the path; if it exists with a different source it is fatal (remove it to re-fetch); otherwise the download body runs, the instance is recorded, and its type / source identity / row id are written to sidecar markers beside it. The resolved instance path is printed to stdout (all logging is on stderr). On a failed download the partial instance is removed and no row survives. When the type declares knit_with_checksum and the pin still applies (the source defaults are unchanged), it is exported as KNIT_RESOURCE_EXPECTED_CHECKSUM for the body to verify; --ignore-checksum (exported as KNIT_IGNORE_CHECKSUM) bypasses that verification.

int _knit_fetch_git()

Git backend: clone <url> into <dest>, check out <ref>, and print the resolved commit SHA (git rev-parse HEAD) to stdout so a mutable ref is pinned to the commit actually obtained. The clone and checkout write their progress to stderr (redirected here so stdout carries only the SHA), and the resulting tree is made read-only. Returns non-zero (leaving cleanup to the caller) if git is missing or any step fails.

Parameters:

• dest -- [in] Directory to create and clone into (must not already exist).

• url -- [in] Repository URL to clone.

• ref -- [in] Ref (branch, tag, or commit) to check out.

int _knit_fetch_local()

Local backend: materialize <dest> from a local source <src>. By default <dest> is a symlink to <src> (so a large already-staged dataset is not duplicated), left writable through its target. With <copy> = "true" the source is copied into a self-contained snapshot which is then made read-only. The source path is resolved to an absolute path first, so a symlink instance does not depend on the caller's working directory. When <expected> is a non-empty knit_with_checksum pin the instance's sha256 (of the file, or a recursive digest of a directory) is checked against it and a mismatch fails the fetch. Returns non-zero (leaving cleanup to the caller) if the source is missing or a step fails.

Parameters:

• dest -- [in] Path to create for the instance (must not already exist).

• src -- [in] Local source path (file or directory) to link or copy.

• copy -- [in] "true" to copy a read-only snapshot, "false" to symlink the source.

• expected -- [in] Expected sha256 to verify, or "" to skip verification.

int _knit_fetch_url()

URL backend: download <url> with curl into <dest>, optionally unpacking it. The artifact is saved under its URL basename; when <uncompress> is "true" it is extracted with tar -xf (auto-detecting gzip/bzip2/xz) and the archive is then removed, so the instance holds the unpacked tree. curl uses -f so an HTTP error fails the fetch. When <expected> is a non-empty knit_with_checksum pin the downloaded archive's sha256 is checked against it (before it is unpacked or removed) and a mismatch fails the fetch. The resulting instance is made read-only. Returns non-zero (leaving cleanup to the caller) on any failed step.

Parameters:

• dest -- [in] Directory to create and download into (must not already exist).

• url -- [in] URL of the artifact to download.

• uncompress -- [in] "true" to unpack the downloaded archive, "false" to keep it.

• expected -- [in] Expected archive sha256 to verify, or "" to skip verification.

int _knit_resource_check_method()

Done callback installed by knit_register_resource. Fatal if the resource type completed registration without declaring a download method, so a type must declare exactly one of knit_with_git / knit_with_url / knit_with_local.

Parameters:

• cmd -- [in] The resource type's command (mangled name).

int _knit_resource_check_sha()

Compare a computed digest against an expected knit_with_checksum pin, case- insensitively (sha256 hex is case-independent). On a mismatch it prints a knit_error naming what failed and returns 1, so the caller aborts the fetch and removes the partial instance; on a match it returns 0.

Parameters:

• expected -- [in] The expected value (the knit_with_checksum pin).

• actual -- [in] The computed value.

• what -- [in] Short label for the artifact (for the error message).

int _knit_resource_cleanup_dir()

Remove a partial or failed instance at <dir>, restoring write permission first so a read-only tree (see _knit_resource_make_readonly) can be deleted. A symlink instance is unlinked without touching its target. A no-op when <dir> is empty or does not exist, so it is safe to call unconditionally on any failure path.

Parameters:

• dir -- [in] Path to the instance to remove.

int _knit_resource_defaults_used()

Test whether a fetch still uses the resource type's declared source defaults, so a knit_with_checksum pin still applies. The pin is authored against the default source; if the user overrides the source the artifact would legitimately hash differently, so the pin no longer holds. For each source-defining parameter of the backend (git: url + ref; url: url; local: path) it compares the effective value against the declared default; it returns 0 when every one matches (defaults in use), 1 otherwise. The uncompress/copy flags are not compared: they do not change the archive/file content the pin covers.

Parameters:

• cmd -- [in] The resource type's command (mangled name).

• method -- [in] The backend method ("git", "url", or "local").

• ... -- [in] The fetch's expanded arguments (read via knit_get_parameter).

int _knit_resource_dep_after_cb()

After-callback installed by knit_with_resource on the consuming command, one per declared resource parameter. It records a "used_by" edge from the fetched resource instance the command depends on to the command itself. It runs as an after-callback (not the before-callback that validates the instance) because the consumer's frame is on the executing stacks only from push time onward, so the consumer's resolved row id — the edge target — is available here but not in the before-callback (this mirrors _knit_setup_dep_after_cb).

The declared parameter name and required type are bound at registration time and carried as the first two arguments; the trailing arguments are the command's own runtime arguments, scanned for the instance name. A missing name is silently skipped: the before-callback already fataled on it, so this is only reached with a valid instance.

Parameters:

• param -- [in] Normalized name of the resource parameter to read.

• type -- [in] Resource type of the named instance (unused; kept for symmetry with the before-callback's bound arguments).

int _knit_resource_dep_before_cb()

Before-callback installed by knit_with_resource on the consuming command, one per declared resource parameter. Before the command body runs it validates that the resource named by the parameter has been fetched and is of the required type, using only the on-disk sidecar markers (no database read):

• the parameter value (the instance name) missing → fatal;

• <resource-root>/<name> absent → fatal, printing the knit fetch command to run;

• the .<name>.resource.type sidecar marker absent or not equal to the declared type → fatal type-mismatch. The instance name is also validated as a single path component. Returning normally lets the command proceed; a fatal aborts it before the body runs.

The declared parameter name and required type are bound at registration time and carried as the first two arguments; the trailing arguments are the command's own runtime arguments, scanned for the parameter value.

Parameters:

• param -- [in] Normalized name of the resource parameter to read.

• type -- [in] Resource type the named instance must have been fetched as.

int _knit_resource_fetch_body()

Shared body registered for every resource type (the function every fetch:<type> command runs). It reads the type's declared download method from the per-command marker (KNIT_CMD<cmd>_fetch_method) and dispatches to the matching backend, materializing the instance at KNIT_RESOURCE_PREFIX (the target path the knit fetch dispatcher exports; it must not already exist, since each backend creates it). The git backend's resolved commit SHA is recorded as the "commit" output. On any backend failure the partial instance is removed and the body returns non-zero so the dispatcher aborts without recording a row.

When the type declares knit_with_checksum and the pin still applies (the source defaults are unchanged), the dispatcher exports it as KNIT_RESOURCE_EXPECTED_- CHECKSUM; the body verifies it — the git commit SHA directly, and the url/local artifacts through their backends — unless the fetch bypassed it with --ignore-checksum (KNIT_IGNORE_CHECKSUM). A mismatch fails the fetch.

int _knit_resource_make_readonly()

Make a fetched instance tree read-only so consumers cannot mutate a shared input: chmod -R a-w clears the write bit for everyone while preserving the existing read and execute bits. Applied by the git and url backends and by the local backend's --copy path; a symlinked local instance is deliberately left alone (its target stays owned and writable by whoever staged it).

Parameters:

• dir -- [in] Path to the instance tree (or file) to make read-only.

int _knit_resource_param_type()

Store the resource type a parameter was declared with in the caller-named variable, or the empty string when the parameter is an ordinary parameter (not declared through knit_with_resource). Reads the per-parameter marker (KNIT_CMD<cmd>resource) that knit_with_resource records, so describe and --help can annotate a resource parameter with its type from the registration tables alone (no database read). The parameter name must be normalized, as it is stored in the parameter sets.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the resource type (empty if none).

• cmd -- [in] Mangled command name.

• param -- [in] Normalized parameter name.

int _knit_resource_record_used_by_edge()

Record a "used_by" provenance edge from a fetched resource instance to a consuming invocation. The edge's source is the instance (its row id read from the .<name>.resource.id sidecar, its node name "fetch:<type>" -- the owning command that recorded the row, so the source_name matches the instance's own "call"/"executed" edges and the query transpiler's label resolution -- with the type read from the .<name>.resource.type sidecar); its target is the consumer. Delegates the gated write to _knit_record_used_by_edge, so it records nothing when recording is disabled, on a suppressed rank, before bootstrap, when the target does not participate in the graph, or when the sidecar has no id (e.g. an instance fetched before provenance shipped).

Parameters:

• name -- [in] Resource instance name (as passed to knit fetch --name).

• target_cmd -- [in] Mangled command name of the consumer (the edge target).

• target_id -- [in] Resolved row id of the consumer (the edge target).

int _knit_resource_require_registration()

Fatal unless a resource type is currently being registered (i.e. this is called between knit_register_resource and knit_done). Shared by the download decorators and knit_with_checksum, which are valid only on a resource type.

Parameters:

• directive -- [in] Name of the calling directive (for the error message).

int _knit_resource_root()

Store the resolved resource root — the directory under which resource instances live — in the caller-named variable. Reads the verbatim resource_path from the metadata table (falling back to "resources" when unset, for robustness) and resolves it against the experiment root via _knit_resolve_experiment_path. Mirrors _knit_setup_root / _knit_job_root.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the resolved resource root.

int _knit_resource_set_method()

Record the download method chosen by a download decorator in the current resource type's per-command marker (KNIT_CMD<cmd>_fetch_method), enforcing the "exactly one download decorator per resource type" rule: a second call (whether the same decorator twice or two different backends) is fatal. Mirrors the per-command marker pattern used elsewhere for at-most-once declarations.

Parameters:

• method -- [in] The backend name ("git", "url", or "local").

int _knit_resource_source_identity()

Build the source-identity string for a fetch and store it in the caller-named variable. The identity is the tuple that identifies where an instance came from, recorded on its row and compared on a same-name re-fetch to distinguish an idempotent skip (identical source) from a conflicting re-fetch (same name, a different source). It is derived from the backend method and the resolved parameters, per backend:

• git: the repository url and the requested ref

• url: the url and the uncompress flag

• local: the source path and the copy flag The commit a git ref resolves to is deliberately not part of the identity: a mutable ref (e.g. main) still identifies the same source between fetches.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the identity string.

• method -- [in] The backend method ("git", "url", or "local").

• ... -- [in] The fetch's arguments (read via knit_get_parameter).

int _knit_sha256()

Compute the sha256 of a file or directory and store the 64-hex digest (with no trailing filename and no algorithm prefix) in the caller-named variable. A regular file is hashed directly. A directory is hashed recursively into one digest: every regular file under it is hashed together with its relative path (sha256sum prints "<hash> <path>"), the lines are sorted for a stable order independent of readdir order, and the aggregate is hashed again — so both the tree structure and every file's content contribute. A symlink is followed (so a local symlink instance hashes its target). Returns non-zero (with a knit_error) when sha256sum is unavailable or a step fails.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the digest.

• path -- [in] Path to the file or directory to hash.

Variables

AssociativeArray _KNIT_RESOURCES

Associative array mapping registered resource type names to 1. Populated by knit_register_resource and consulted by the knit fetch dispatcher (to resolve a requested type) and by knit_with_resource (to validate a declared dependency's type). Declared global (-g) so it survives being sourced from within a function (as the bats tests do). Mirrors _KNIT_SETUPS.

rocrate.sh

Functions

int _knit_bundle_rocrate_generate()

Write the RO-Crate manifest (ro-crate-metadata.json) for the experiment. The manifest is JSON-LD in the RO-Crate 1.1 / Process Run Crate vocabulary: a metadata descriptor, a root Dataset that conforms to the profile and lists the packed files (hasPart) and the recorded actions (mentions), a File or Dataset data entity per packed path, a SoftwareApplication per distinct command, one CreateAction per provenance node, and the PropertyValue entities for the rows' columns. The action graph comes from the provenance edges: a "call" edge links the caller's result to the callee action; a "used_by" edge links the consumer's object to the setup or resource action; a "produced" edge links the producer's result to the artifact. Row columns become PropertyValues, split into the action's object (inputs) and result (outputs); "native_cmd" becomes the action's description and "state" its actionStatus.

The manifest describes exactly the given packed paths, so the same generator serves "knit bundle --ro-crate" (the files travel with it) and "knit export

ro-crate" (the manifest alone, describing the files in place).

Parameters:

• output -- [in] The path to write, or "-" for standard output.

• root -- [in] The absolute experiment root.

• ... -- [in] The packed relative paths the manifest must describe.

int _knit_export_discover()

int _knit_export_rocrate()

Body of "knit export ro-crate": write the RO-Crate manifest alone, with no archive and no copied files. It describes the experiment's on-disk files by their current relative paths, using the same default contents "knit bundle" would pack, so it is the cheap way to inspect or regenerate the manifest in place. The command is read-only: it declares no table and takes knit_without_provenance.

Parameters:

• ... -- [in] The command invocation arguments.

int _knit_rocrate_edges_json()

Print the provenance edges, as a JSON array of objects, one per edge, read straight from the provenance table with sqlite's JSON output. The bootstrap subtree is filtered out (an edge naming "bootstrap" at either end), so the crate describes the science, not the plumbing. The empty array is printed when the table is absent or holds no non-bootstrap edge.

int _knit_rocrate_encoding_format()

Print a best-guess IANA media type for a packed file, chosen from its base name, or nothing when no guess fits. The RO-Crate manifest records the type as a File entity's encodingFormat, so a reader knows how to open the file without Knit. The guesses cover the file kinds a bundle carries: the provenance database, the experiment and job scripts, the JSON lock files, the Spack YAML manifests, and the plain-text logs and markers.

Parameters:

• name -- [in] The file's base name.

int _knit_rocrate_files_json()

Print, as a JSON array of objects, one entry per packed file, describing it for the RO-Crate data entities. Each entry holds the packed relative path, the @id it takes in the crate (a trailing "/" for a directory), the base name, a directory flag, a guessed encodingFormat, whether it is the experiment script, and — for a file under a job directory — the job's id (so an action can list its own logs as results). The manifest file itself is never described as a data entity, so it is skipped here.

Parameters:

• root -- [in] The absolute experiment root.

• script_rel -- [in] The experiment script's path relative to the root.

• job_rel -- [in] The job root's path relative to the root ("jobs" by default).

• ... -- [in] The packed relative paths.

int _knit_rocrate_outputs_json()

Print, as a JSON array of strings, the normalized output column names of a command. The RO-Crate mapping puts a command's inputs (params and flags) in an action's "object" and its outputs in the action's "result", so the generator needs to know which of a data row's columns are outputs. The names come from the command's registration output set and are normalized (hyphens to underscores) to match the database column names. An unknown command, or one with no outputs, yields the empty array.

Parameters:

• mangled -- [in] The mangled command name.

int _knit_rocrate_rows_json()

Print the recorded data rows, as a JSON array of objects, one per row, drawn from every per-command table that is both registered in this run and present in the database. Each object carries the row id, its table and owning command, the command's output column names, and the row's full column map ("cols"), so the generator can turn a row into an action's PropertyValues and split them into inputs and outputs. A table with no rows contributes nothing.

int _knit_rocrate_table_exists()

Test whether a table exists in the database. Used before a "SELECT *" read so a per-command table that was registered but never written (no invocation yet) does not draw an error.

Parameters:

• table -- [in] The table name.

Returns:

0 if the table exists, 1 otherwise.

sched.sh

Functions

int _knit_jobid_read()

Read a job's recorded launcher id from <jobdir>/.job.id and resolve both the scheduler backend that minted it and the backend-specific id. The file is written tagged as "<backend>:<payload>" (see _knit_job_submit), so the backend travels with the id and every reader routes by the id's true namespace rather than the ambient backend of wherever the command runs. For a pilot sub-job the payload is itself "<parent-uuid>:<sub-job-id>"; this returns it whole and the pilot layer splits it.

A file with no recognised "<backend>:" prefix is a legacy raw id (written before tagging): the whole content is the id and the backend falls back to the ambient _knit_sched_backend, preserving the old behaviour. The prefix counts as a tag only when it names a known backend, so a legacy id that happens to carry a colon is not misparsed.

The caller is responsible for checking that <jobdir>/.job.id exists; this does not.

Parameters:

• __knit_ret1 -- [out] Name of the variable to hold the backend name.

• __knit_ret2 -- [out] Name of the variable to hold the backend-specific id.

• jobdir -- [in] Job directory holding the .job.id file.

int _knit_sched_backend()

Resolve which scheduler backend a submission targets. Inside a pilot job (KNIT_PILOT set) the backend is "pilot": knit submit is retargeted to the in-allocation knit-pilotd daemon rather than the machine scheduler. Otherwise it is the machine's real backend (see _knit_sched_real_backend). Returns one of "pilot", "local", "none", "slurm", "pbs", "flux".

Parameters:

• __knit_ret -- [out] Name of the variable to hold the resolved backend name.

int _knit_sched_cancel()

Dispatch to the configured backend's cancel function, which asks the scheduler to terminate a running job. Each backend uses its native primitive (local: kill, slurm: scancel, pbs: qdel). Cancelling a job that is already gone is not an error. Knit's terminal state (killed) is recorded by the caller.

Parameters:

• backend -- [in] Scheduler backend name ("local", "none", "slurm", "pbs", "flux").

• jobid -- [in] Backend job id (scheduler id, or a PID for local/none backends).

int _knit_sched_directives()

Dispatch to the configured backend's directive generator, printing the batch scheduler directive lines (e.g. "#SBATCH ..." / "#PBS ...") for the resolved options. The local and none backends print nothing.

Parameters:

• backend -- [in] Scheduler backend name ("local", "none", "slurm", "pbs", "flux").

• arr_name -- [in] Name of the resolved-options associative array.

• jobdir -- [in] Job directory (used by backends for --output/--error paths).

int _knit_sched_hostfile()

Print, one per line, the raw host entries a job is running on, as the current backend's scheduler reports them. Unlike the other dispatchers this resolves the backend itself (via _knit_sched_backend): it is meant to be called at runtime from inside a job body, where no backend has been resolved by a caller.

The output is the "raw" host list: hostnames may be repeated (once per slot, e.g. PBS's $PBS_NODEFILE) or carry trailing ":N" info. Callers that want a deduplicated, cleaned list post-process it (see knit_job_hostnames).

int _knit_sched_pick_queue()

Select a queue automatically for the "--queue auto" strategy, choosing the first queue declared in the machine profile that would accept the job as the user specified it. Queues are read from the profile's .scheduler.queues object in declaration order (jq preserves object key order), and the first one that violates none of its declared constraints is returned by nameref.

The constraints checked are the node count (min_nodes/max_nodes) and, only when the user gave a concrete walltime, the walltime bounds (min_walltime/ max_walltime, compared in seconds). A "none" walltime (no wall-clock limit) does not filter on walltime, the same as an unspecified one. A constraint the profile omits — or a profile walltime that does not parse — is not an obstacle (permissive). The walltime argument is the raw user request, so this must run before walltime is defaulted (the default itself is queue-derived, which is why auto resolves the queue first).

Outcomes:

• no queues declared in the profile -> empty selection, return 0 (the caller then submits with no queue directive, i.e. the scheduler's own default);

• a queue accepts the job -> that queue's name, return 0;

• queues are declared but none accept it -> knit_fatal with a per-queue rejection diagnostic (this runs at build time, before any submission).

Parameters:

• __knit_ret -- [out] Name of the variable to hold the selected queue name.

• nodes -- [in] Requested node count.

• walltime -- [in] Requested walltime (HH:MM:SS), "none" for no limit, or empty when unspecified. Only a concrete HH:MM:SS filters.

int _knit_sched_profile_field()

Return a field from the bootstrapped experiment's machine profile, or the empty string when no profile is configured. The value comes from the profile JSON frozen at bootstrap (profile_json metadata), read via knit_get_profile_field. The profile argument is retained as a gate so callers can request a field unconditionally.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the field value.

• profile -- [in] Profile label (may be empty; empty means "no profile").

• jq_path -- [in] jq path expression, e.g. '.scheduler.default_queue'.

int _knit_sched_real_backend()

Resolve the machine's real scheduler backend, ignoring any active pilot: bootstrap metadata (scheduler), else live detection. Detection's "<unknown>" (no batch scheduler present) means the workstation case and maps to the local background-process backend. An explicit "none" in metadata is a real, deliberate backend (a user-owned cluster driven without a scheduler) and flows through untouched. Returns one of "local", "none", "slurm", "pbs", "flux".

Parameters:

• __knit_ret -- [out] Name of the variable to hold the resolved backend name.

int _knit_sched_resolve()

Resolve the submission options for a job into a caller-provided associative array. For each option the value is resolved by precedence:

explicit CLI argument -> bootstrap metadata -> machine profile -> hard-coded

The array is keyed by canonical option name (job-name, account, project, queue, nodes, cpus-per-node, walltime, gpus-per-node, wait) plus "extra-args" for the site-mandatory scheduler arguments captured at bootstrap, and "walltime-defaulted" ("true"/"false") recording whether the walltime was chosen by knit rather than requested (so the submit path can warn). The per-node core count (cpus-per-node) is derived rather than requested (knit allocates whole nodes). Job stdout/stderr are fixed to <job-dir>/.stdout and <job-dir>/.stderr by the backend, so they are not resolved here.

Parameters:

• out_array -- [out] Name of an associative array to populate (passed by name).

• ... -- [in] The submission CLI arguments (everything before "--").

int _knit_sched_status()

Dispatch to the configured backend's status query, which asks the scheduler for the true current state of a submitted job and returns a normalized verdict. Used by job reconcile to correct job rows that are stuck at running/submitting because a hard SIGKILL prevented the compute-side kill trap from recording the terminal state (see src/job_cli.sh _knit_job_reconcile_apply).

The verdict is one of: running (job is still active), completed (job exited cleanly), killed (job was terminated by a signal or timeout, or the scheduler has no record of it), failed (job exited with a non-signal non-zero exit code). Callers treat "unknown" as "killed" per the agreed policy; backends that cannot determine the outcome return "unknown" directly.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the verdict string.

• backend -- [in] Scheduler backend name ("local", "none", "slurm", "pbs", "flux").

• jobid -- [in] Backend job id (scheduler id, or a PID for local/none backends).

int _knit_sched_submit()

Dispatch to the configured backend's submission function, which submits the already-written batch script and prints the resulting scheduler job id (or, for the local/none backend, the process id) to stdout.

Parameters:

• backend -- [in] Scheduler backend name ("local", "none", "slurm", "pbs", "flux").

• arr_name -- [in] Name of the resolved-options associative array.

• script -- [in] Path to the batch script to submit.

• jobdir -- [in] Job directory (holds .stdout/.stderr for local/none backends).

int _knit_sched_submit_cmdline()

Dispatch to the configured backend's submission-command builder, which fills a caller-provided array (passed by name) with the argv the backend runs to submit the batch script (e.g. "sbatch <script>", "qsub <script>", "bash <script>"). This is built separately from _knit_sched_submit so the resolved submission command can be recorded in the jobs table and logged before it is issued.

Parameters:

• backend -- [in] Scheduler backend name ("local", "none", "slurm", "pbs", "flux").

• arr_name -- [in] Name of the resolved-options associative array.

• script -- [in] Path to the batch script to submit.

• argv_name -- [in] Name of the array to fill with the submission argv.

int _knit_sched_wait()

Dispatch to the configured backend's blocking wait, which returns once the scheduler no longer considers the job active. Each backend blocks using the native mechanism its scheduler actually provides (see the per-backend functions); knit's own terminal state (completed/killed) is read from the jobs table afterwards, so this only has to unblock when the job stops running.

Parameters:

• backend -- [in] Scheduler backend name ("local", "none", "slurm", "pbs", "flux").

• jobid -- [in] Backend job id (scheduler id, or a PID for local/none backends).

int _knit_sched_write_jobscript()

Write the batch script that a scheduler runs on the compute node. The script carries the backend's directives, exports the job/setup prefixes, cd's into the job directory, and re-enters the experiment script to run the job body via exp.sh submit <job-name> <args>. Arguments are q-quoted so they survive the round-trip through the batch script unchanged.

Parameters:

• script_path -- [in] Path of the batch script to create.

• backend -- [in] Scheduler backend name.

• arr_name -- [in] Name of the resolved-options associative array.

• setup_path -- [in] Setup directory (exported as KNIT_SETUP_PREFIX; empty for a setup-less job, in which case KNIT_SETUP_PREFIX is not exported).

• jobdir -- [in] Job directory (exported as KNIT_JOB_PREFIX; the cd target).

• source_id -- [in] Submitting invocation's row id (exported as KNIT_SOURCE_ID so the compute-side job body records a call edge back to the submitter; empty to omit the export).

• source_command -- [in] Submitting invocation's command name (exported as KNIT_SOURCE_COMMAND; emitted only when source_id is set).

• job_name -- [in] Registered job name to run.

• ... -- [in] Arguments to pass to the job.

int _knit_uuidv7()

Generate a version-7 UUID (RFC 9562) and print it to stdout.

A uuidv7 encodes a 48-bit big-endian Unix millisecond timestamp in its leading bits, so lexically sorting uuidv7 strings orders them by creation time. This is why job directories are named with one: they sort chronologically.

Layout (hex digits, formatted 8-4-4-4-12):

• digits 1-12 : 48-bit millisecond timestamp

• digit 13 : version nibble, always "7"

• digits 14-16 : random

• digit 17 : variant nibble, one of 8, 9, a, b

• digits 18-32 : random

Randomness comes from /dev/urandom when available, falling back to the bash ${RANDOM} generator; the timestamp prefix guarantees ordering and near uniqueness even in the fallback case. The millisecond clock falls back to whole-second precision when date lacks nanosecond (N) support.

Variables

String _KNIT_SCHED_KILL_WARNING_SEC

Seconds before the walltime limit at which a scheduler is asked to warn the job (Slurm --signal). The warning lets the job record itself as "killed" before it is hard-killed (see _knit_job_killed_trap).

String _KNIT_SCHED_POLL_INTERVAL

Seconds between polls when a backend has to wait for a job by polling its scheduler (Slurm squeue, PBS qstat — neither offers a reliable blocking "wait for completion" primitive). Overridable, chiefly so tests can drive the poll loops quickly.

sched_flux.sh

Functions

int _knit_sched_flux_cancel()

Cancel a Flux job with flux cancel. Flux sends SIGTERM then SIGKILL after a grace period, which lets the batch shell's pre-termination handler record the job "killed" (see _knit_job_killed_trap). Cancelling an already-finished or unknown job may print a diagnostic; that is not treated as a knit-level failure since the job is gone.

Parameters:

• jobid -- [in] Flux job id (from the job's .job.id).

int _knit_sched_flux_directives()

Emit the "# flux:" directive lines for a resolved job. A Flux batch directive is a comment whose body starts with "flux:", the analogue of "#SBATCH"/"#PBS". Knit allocates whole nodes exclusively, so --exclusive is always set with --nodes. Job stdout/stderr are fixed to <jobdir>/.stdout and <jobdir>/.stderr. The per-node core count is not emitted: the allocation is whole-node exclusive, so per-node task sizing belongs to the launcher (flux run), not the allocation. Optional fields (bank, queue, gpus) are emitted only when set.

Parameters:

• arr_name -- [in] Name of the resolved-options associative array.

• jobdir -- [in] Job directory (for the --output/--error paths).

int _knit_sched_flux_hostfile()

Print the host list for a Flux job, one hostname per line. flux hostlist expands a job's hostlist; the source "local" reads the enclosing job from $FLUX_JOB_ID, and "-ed '\n'" expands every host with a newline delimiter. This is a per-node list. If the call fails (e.g. not running inside a Flux job), warn and fall back to this machine's hostname, exactly as the other backends do.

int _knit_sched_flux_parse_jobid()

Reduce flux batch output to the bare job id by keeping the first line and trimming surrounding whitespace. The id is a single opaque FLUID token (e.g. "f2QzoR8xF"), so no further splitting is done.

Parameters:

• raw -- [in] Raw stdout captured from flux batch.

int _knit_sched_flux_status()

Ask Flux for the true state of a submitted job and return a normalized verdict (one of: running, completed, killed, failed, unknown). Two flux-jobs queries are issued: one for the job state and, for an INACTIVE job, one for the result.

State / result mapping: state DEPEND PRIORITY SCHED RUN CLEANUP -> running state INACTIVE + result COMPLETED -> completed state INACTIVE + result CANCELED or TIMEOUT -> killed state INACTIVE + result FAILED -> failed not found / empty state -> unknown

Parameters:

• __knit_ret -- [out] Name of the variable to hold the verdict string.

• jobid -- [in] Flux job id (from the job's .job.id).

int _knit_sched_flux_submit()

Submit an already-written batch script with flux batch and print the resulting job id. Flux has no sbatch --wait equivalent at submit time, so when the resolved "wait" flag is "true" the function blocks after submission with flux job status <id> (a real blocking wait; see _knit_sched_flux_wait). The --output/--error redirection is carried by the script's directives, so the job directory is not needed here.

Parameters:

• arr_name -- [in] Name of the resolved-options associative array.

• script -- [in] Path to the batch script to submit.

• jobdir -- [in] Job directory (unused; redirection is set via directives).

int _knit_sched_flux_submit_cmdline()

Build the flux batch submission command into a caller-provided array, passed by name: "flux batch <script>". Flux has no submit-time blocking flag (see _knit_sched_flux_submit for how the wait option is honoured), so the argv does not depend on the resolved "wait" option.

Parameters:

• argv_name -- [out] Name of the array to fill with the submission argv.

• arr_name -- [in] Name of the resolved-options associative array (unused).

• script -- [in] Path to the batch script to submit.

int _knit_sched_flux_wait()

Block until the Flux job completes. flux job status <id> blocks until the job reaches a terminal state and exits with the job's largest task exit code. It works for any job (it does not need the "waitable" flag), so unlike the Slurm/PBS backends this is a single blocking call, not a poll loop. The job's knit terminal state is read from the DB by the caller afterwards.

Parameters:

• jobid -- [in] Flux job id (from the job's .job.id).

int _knit_sched_flux_walltime_fsd()

Convert an HH:MM:SS walltime to a Flux Standard Duration (FSD) in whole seconds with a trailing "s", e.g. "01:00:00" -> "3600s". Flux --time-limit reads a bare number as minutes, so the explicit "s" suffix avoids that ambiguity. The conversion uses the shared _knit_walltime_to_seconds parser; a value that is not HH:MM:SS is printed verbatim so a site that already writes a valid FSD (e.g. "30m") is passed through unchanged.

Parameters:

• walltime -- [in] Walltime string, normally "HH:MM:SS".

sched_local.sh

Functions

int _knit_sched_local_cancel()

Cancel a locally-launched job by sending SIGTERM to its process. SIGTERM (not SIGKILL) is used deliberately: the running job installs a handler for it (see _knit_job_killed_trap) so it can record itself "killed" before exiting. A pid that is not a positive integer, or a process that is already gone, is treated as nothing to do.

Parameters:

• pid -- [in] Process id recorded in the job's .job.id by the local backend.

int _knit_sched_local_directives()

Emit the batch directives for the local backend. The local backend runs the job as a background process with no scheduler, so there are no directives and this prints nothing. It exists so every backend honours the same contract.

Parameters:

• arr_name -- [in] Name of the resolved-options associative array (unused).

• jobdir -- [in] Job directory (unused).

int _knit_sched_local_hostfile()

Print the host list for the local backend. A local run has no scheduler and no node allocation, so it runs on a single host: print this machine's hostname.

int _knit_sched_local_status()

Return the status of a locally-launched job process as a normalized verdict (one of: running, killed, unknown). The launcher id is a PID recorded in the job's .job.id. A non-integer id returns unknown. kill -0 (a liveness probe) is used to check whether the process is still alive: success means running; failure means the process is gone and is reported as killed (there is no exit record for a bare background process, so completed and failed cannot be distinguished).

Parameters:

• __knit_ret -- [out] Name of the variable to hold the verdict string.

• pid -- [in] Process id recorded in the job's .job.id by the local backend.

int _knit_sched_local_submit()

Submit a batch script on a machine with no scheduler by running it as a detached background process via _knit_submit_local. Job stdout/stderr are redirected to <jobdir>/.stdout and <jobdir>/.stderr, and the resolved walltime (if any) caps the run; a walltime of "none" imposes no cap. When the resolved "wait" flag is "true", block until the process finishes. Prints the process id to stdout.

Parameters:

• arr_name -- [in] Name of the resolved-options associative array.

• script -- [in] Path to the batch script to run.

• jobdir -- [in] Job directory holding .stdout/.stderr.

int _knit_sched_local_submit_cmdline()

Build the local backend's submission command into a caller-provided array, passed by name: "bash <script>". The local backend runs this in the background via _knit_submit_local, which adds stdout/stderr redirection and an optional walltime cap; those are knit-managed conveniences rather than part of the job command, so the recorded/traced command is the bare "bash <script>".

Parameters:

• argv_name -- [out] Name of the array to fill with the submission argv.

• arr_name -- [in] Name of the resolved-options associative array (unused).

• script -- [in] Path to the batch script to run.

int _knit_sched_local_wait()

Block until a locally-launched job process exits. The job wait command runs in a different process than the submit that launched the job, so the pid is not a child of this shell and the wait builtin cannot be used; instead poll kill -0 (a liveness probe that sends no signal) every _KNIT_SCHED_POLL_INTERVAL seconds until the process is gone. A pid that is not a positive integer, or is already gone, returns immediately.

Parameters:

• pid -- [in] Process id recorded in the job's .job.id by the local backend.

sched_none.sh

Functions

int _knit_sched_none_cancel()

Cancel a none-backend job by signalling its process, identically to the local backend. Delegates to _knit_sched_local_cancel.

Parameters:

• pid -- [in] Process id recorded in the job's .job.id.

int _knit_sched_none_directives()

The "none" backend is for a user who owns a multi-node cluster but does not submit through a batch scheduler (SSH-reachable nodes, a hand-maintained MPI hostfile, etc.). A job launches exactly as the local backend does — as a background process on the submitting host — so the lifecycle functions (directives, submit, cancel, wait) delegate to their local counterparts. The only difference is host reporting: the allocation is the node list configured at bootstrap via --default-nodefile (metadata default_nodefile), not just the submitting host's name. That is what _knit_sched_none_hostfile provides.

Emit the batch directives for the none backend. Like the local backend, the job runs as a background process with no scheduler, so there are no directives. Delegates to _knit_sched_local_directives.

Parameters:

• arr_name -- [in] Name of the resolved-options associative array.

• jobdir -- [in] Job directory.

int _knit_sched_none_hostfile()

Print the host list for a none-backend job: the contents of the node file configured at bootstrap (metadata default_nodefile), one host per line. This is the "raw" host list per the dispatcher contract — entries may repeat or carry a trailing ":N" slot count; callers such as knit_job_hostnames deduplicate and clean it. Blank lines are dropped so an empty entry never reaches the normaliser. When no readable node file is configured (unset, or the path does not exist / is not readable), warn and fall back to this machine's hostname.

int _knit_sched_none_status()

Return the status of a none-backend job process. Identical to the local backend (the job is a background process on this host). Delegates to _knit_sched_local_status.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the verdict string.

• pid -- [in] Process id recorded in the job's .job.id.

int _knit_sched_none_submit()

Submit a batch script for the none backend by running it as a detached background process on the submitting host, identically to the local backend. Delegates to _knit_sched_local_submit, which handles stdout/stderr redirection, walltime, and optional blocking, and prints the process id.

Parameters:

• arr_name -- [in] Name of the resolved-options associative array.

• script -- [in] Path to the batch script to run.

• jobdir -- [in] Job directory holding .stdout/.stderr.

int _knit_sched_none_submit_cmdline()

Build the none backend's submission command, identically to the local backend (the job runs as a background "bash <script>" on this host). Delegates to _knit_sched_local_submit_cmdline.

Parameters:

• argv_name -- [in] Name of the array to fill with the submission argv.

• arr_name -- [in] Name of the resolved-options associative array.

• script -- [in] Path to the batch script to run.

int _knit_sched_none_wait()

Block until a none-backend job process exits. Identical to the local backend (the job is a background process on this host). Delegates to _knit_sched_local_wait.

Parameters:

• pid -- [in] Process id recorded in the job's .job.id.

sched_pbs.sh

Functions

int _knit_sched_pbs_cancel()

Cancel a PBS job with qdel. qdel sends SIGTERM then SIGKILL, letting the batch shell's pre-termination handler record the job "killed" (see _knit_job_killed_trap). A qdel of an already-finished/unknown job may print a diagnostic; that is not treated as a knit-level failure since the job is gone.

Parameters:

• jobid -- [in] PBS job id (from the job's .job.id).

int _knit_sched_pbs_directives()

Emit the #PBS directive lines for a resolved job. Knit allocates whole nodes exclusively: the resource request is one chunk per node (-l select=<nodes>) with -l place=excl. When the per-node core count is known it pins ncpus and mpiprocs on the chunk (mpiprocs is what populates $PBS_NODEFILE, i.e. the launchable slots); otherwise the chunk takes the site defaults. Job stdout/stderr are fixed to <jobdir>/.stdout and <jobdir>/.stderr. Optional fields (account, project, queue, gpus) are emitted only when set.

Parameters:

• arr_name -- [in] Name of the resolved-options associative array.

• jobdir -- [in] Job directory (for the -o/-e paths).

int _knit_sched_pbs_hostfile()

Print the host list for a PBS job. PBS writes the allocated slots to the file named by $PBS_NODEFILE, one hostname per line, each repeated once per launchable slot (mpiprocs) on that node. Print its contents verbatim. If $PBS_NODEFILE is unset or unreadable (e.g. not running inside a PBS job), warn and fall back to this machine's hostname.

int _knit_sched_pbs_parse_jobid()

Reduce a qsub job id to its bare sequence number by keeping the first output line and stripping the server suffix, e.g. "98765.pbsserver" -> "98765".

Parameters:

• raw -- [in] Raw stdout captured from qsub.

int _knit_sched_pbs_status()

Ask PBS for the true state of a submitted job and return a normalized verdict (one of: running, completed, killed, failed, unknown). Uses qstat -x -f to query finished-job history (requires job_history_enable on the server, which the test cluster enables). Parses job_state and Exit_status with the same awk -F'=' style used by _knit_sched_pbs_wait.

State mapping from job_state / Exit_status: R Q H W T S B E -> running (active or queued) F with Exit_status absent -> killed (history truncated or OOM) F with Exit_status = 0 -> completed F with Exit_status >= 128 (signal exit) -> killed F with other non-zero Exit_status -> failed no record at all -> unknown

Parameters:

• __knit_ret -- [out] Name of the variable to hold the verdict string.

• jobid -- [in] PBS job id (from the job's .job.id).

int _knit_sched_pbs_submit()

Submit an already-written batch script with qsub and print the resulting job id. When the resolved "wait" flag is "true", qsub is run with "-W block=true" so it blocks until the job completes (its exit status becomes qsub's). The -o/-e redirection is carried by the script's directives, so the job directory is not needed here.

Parameters:

• arr_name -- [in] Name of the resolved-options associative array.

• script -- [in] Path to the batch script to submit.

• jobdir -- [in] Job directory (unused; redirection is set via directives).

int _knit_sched_pbs_submit_cmdline()

Build the qsub submission command into a caller-provided array, passed by name: "qsub [-W block=true] <script>". The "-W block=true" arguments are added when the resolved "wait" option is "true" (see _knit_sched_pbs_submit for its effect).

Parameters:

• argv_name -- [out] Name of the array to fill with the submission argv.

• arr_name -- [in] Name of the resolved-options associative array.

• script -- [in] Path to the batch script to submit.

int _knit_sched_pbs_wait()

Block until PBS no longer runs the job. OpenPBS ships no qwait, so poll qstat every _KNIT_SCHED_POLL_INTERVAL seconds. -x also reports finished jobs from history when it is enabled. A job whose job_state is "E" (exiting, its Exit_status is already set) or "F" (finished) is terminal. "E" is treated as terminal because a job can linger in it after its script has stopped running, and waiting for "F" would then block far longer than the job runs.

An empty result (no job_state line) is ambiguous: the job is either not yet registered or already gone. qsub can return before qstat exposes a fresh job, so an inline submit --wait polls almost immediately after submission and must not mistake that registration lag for completion. The wait therefore only treats "gone" as finished once the job has been seen at least once; a job never seen is polled up to _KNIT_SCHED_PBS_REGISTER_GRACE_POLLS times (covering the registration window) before it is concluded unknown/aged-out and the wait returns. The job's knit terminal state is read from the DB by the caller.

Parameters:

• jobid -- [in] PBS job id (from the job's .job.id).

Variables

String _KNIT_SCHED_PBS_REGISTER_GRACE_POLLS

How many consecutive "job not visible in qstat" polls _knit_sched_pbs_wait tolerates before it concludes a never-seen job is unknown or aged out of history (and returns). This guards the just-submitted race: qsub can return a job id before qstat exposes the job, so an inline submit --wait may poll before the job registers; without a grace window an empty first result would be mistaken for completion. Overridable, chiefly so tests can drive it quickly.

sched_slurm.sh

Functions

int _knit_sched_slurm_cancel()

Cancel a Slurm job with scancel. scancel sends SIGTERM (then SIGKILL after a grace period), which lets the batch shell's pre-termination handler record the job "killed" (see _knit_job_killed_trap). scancel exits 0 even for a job that has already finished, so no special-casing is needed here.

Parameters:

• jobid -- [in] Slurm job id (from the job's .job.id).

int _knit_sched_slurm_directives()

Emit the #SBATCH directive lines for a resolved job. Knit allocates whole nodes exclusively, so --exclusive is always set and --ntasks-per-node is pinned to the derived per-node core count (when known) so every core is a launchable slot. Job stdout/stderr are fixed to <jobdir>/.stdout and <jobdir>/.stderr. Optional fields (account, project/wckey, partition, gpus) are emitted only when set.

Parameters:

• arr_name -- [in] Name of the resolved-options associative array.

• jobdir -- [in] Job directory (for the --output/--error paths).

int _knit_sched_slurm_hostfile()

Print the host list for a Slurm job, one hostname per line. Slurm exposes the allocation as a compressed nodelist ($SLURM_JOB_NODELIST, e.g. "node[01-03]") rather than a hostfile, so expand it with scontrol show hostnames, which prints one hostname per allocated node. This is a per-node list (not per-task): Slurm has no native per-slot hostfile, so a raw request yields the same per-node entries. If the nodelist is unset or the expansion fails (e.g. not running inside a Slurm job), warn and fall back to this machine's hostname.

int _knit_sched_slurm_parse_jobid()

Extract the numeric job id from sbatch's output line "Submitted batch job <N>". Falls back to the last whitespace-separated token.

Parameters:

• raw -- [in] Raw stdout captured from sbatch.

int _knit_sched_slurm_status()

Ask Slurm for the true state of a submitted job and return a normalized verdict (one of: running, completed, killed, failed, unknown). sacct is used rather than squeue because sacct retains accounting records after the job finishes, so the result is available after the job exits. When sacct has no record (the accounting database is absent or the job predates it), squeue is tried as a fallback: a non-empty squeue result means the job is still queued/running; an empty result means the job is gone but we cannot determine its outcome (unknown).

State mapping from sacct's job state field (first whitespace-separated token, because Slurm emits "CANCELLED by <uid>" for user cancellations): RUNNING PENDING REQUEUED RESIZING SUSPENDED CONFIGURING COMPLETING -> running COMPLETED -> completed TIMEOUT OUT_OF_MEMORY NODE_FAIL CANCELLED PREEMPTED BOOT_FAIL DEADLINE -> killed FAILED -> failed

Parameters:

• __knit_ret -- [out] Name of the variable to hold the verdict string.

• jobid -- [in] Slurm job id (from the job's .job.id).

int _knit_sched_slurm_submit()

Submit an already-written batch script with sbatch and print the resulting job id. When the resolved "wait" flag is "true", sbatch is run with --wait so it blocks until the job completes (its exit status becomes sbatch's). The --output/--error redirection is carried by the script's directives, so the job directory is not needed here.

Parameters:

• arr_name -- [in] Name of the resolved-options associative array.

• script -- [in] Path to the batch script to submit.

• jobdir -- [in] Job directory (unused; redirection is set via directives).

int _knit_sched_slurm_submit_cmdline()

Build the sbatch submission command into a caller-provided array, passed by name: "sbatch [--wait] <script>". The --wait flag is added when the resolved "wait" option is "true" (see _knit_sched_slurm_submit for its effect).

Parameters:

• argv_name -- [out] Name of the array to fill with the submission argv.

• arr_name -- [in] Name of the resolved-options associative array.

• script -- [in] Path to the batch script to submit.

int _knit_sched_slurm_wait()

Block until Slurm no longer lists the job as active. Slurm has no reliable blocking "wait for completion" primitive after submission: scontrol wait_job returns as soon as the job is allocated (not when it finishes), and sbatch --wait only applies at submit time. So poll squeue for the job id every _KNIT_SCHED_POLL_INTERVAL seconds until it produces no rows, which is true once the job has completed, failed, or been cancelled (a running or completing CG job still lists). The job's knit terminal state is read from the DB by the caller afterwards.

Parameters:

• jobid -- [in] Slurm job id (from the job's .job.id).

search.sh

Functions

int _knit_search_is_descriptor()

Cheap shape check: return 0 if a token has the form of a domain descriptor (a non-empty "{...}", "[...]", "@glob:...", or "@file:..."), 1 otherwise. This is the fork-free gate used at declaration time to reject a gross typo without a full parse; it does not check the internal well-formedness of a range or the type x kind rules (those are the parser's and validator's job, on demand).

Parameters:

• token -- [in] The token to check.

Returns:

0 if the token looks like a descriptor, 1 otherwise.

int _knit_search_job_terminal_status()

Read a job trial's terminal status from the jobs table, by row id. Returns 0 and sets the output to "completed" (the job finished cleanly) or "failed" (the job ended in any other terminal state, "failed" or "killed"); returns 1 and clears the output while the job is not yet terminal (no row, "submitted", or "running"), so the loop keeps polling. The state column is written compute-side by the job before/after callbacks.

Parameters:

• __knit_ret -- [out] The terminal status ("completed" or "failed").

• uuid -- [in] The job's row id (its UUID).

Returns:

0 if the job reached a terminal state, 1 otherwise.

int _knit_search_loop_app()

The app-mode search loop: launch each trial's blocking "knit run" in the background and detect completion by reaping the next finished child with "wait -n". Keeps up to <parallel> runs in flight and launches at most <budget> in total (empty budget means the proposer's ask exhaustion is the only limit). Each background child writes its recorded row id to a temporary file and exits with the run's own status, so the loop learns both after the reap.

Parameters:

• parallel -- [in] Maximum runs in flight at once (>= 1).

• budget -- [in] Maximum trials to launch (empty for unbounded).

• ask_fn -- [in] Proposer ask callback (ask_fn <tag-out> <argv-out>; returns 1 when exhausted).

• observe_fn -- [in] Proposer observe callback (see _knit_search_loop_finish).

• collect_fn -- [in] Optional collector callback (empty to skip).

Returns:

0 on a completed campaign, 1 when the proposer aborted it.

int _knit_search_loop_finish()

Handle one finished trial: read its objective through the optional collector, then hand the outcome to the proposer's observe callback. The collector runs only when a collector function was given (the proposer wants the objective); the search core owns no reporting surface. The observe callback decides, by its return value, whether the campaign continues (0) or aborts (non-zero), so the failure policy stays proposer-specific (sweep "stop" aborts; sweep "continue" and BO "ignore"/"penalize" continue).

Parameters:

• observe_fn -- [in] The proposer's observe callback (observe_fn <tag> <uuid> <status> <objective>).

• collect_fn -- [in] The optional collector callback (collect_fn <out> <uuid>), or empty to skip the objective read.

• tag -- [in] The proposer's opaque key for this trial (from ask).

• uuid -- [in] The trial child's recorded row id.

• status -- [in] The trial's terminal status ("completed" or "failed").

Returns:

0 to continue the campaign, non-zero to abort it (observe's verdict).

int _knit_search_loop_job()

The job-mode search loop: launch each trial as an ordinary submission and detect completion by polling the jobs table state column. Keeps up to <parallel> trials in flight and launches at most <budget> in total (empty budget means the proposer's ask exhaustion is the only limit). Between polls, when no trial finished, it waits <poll> seconds (through the overridable "sleep", so a test can drive it deterministically).

Parameters:

• parallel -- [in] Maximum trials in flight at once (>= 1).

• budget -- [in] Maximum trials to launch (empty for unbounded).

• poll -- [in] Seconds to wait between polls when nothing finished.

• ask_fn -- [in] Proposer ask callback (ask_fn <tag-out> <argv-out>; returns 1 when exhausted).

• observe_fn -- [in] Proposer observe callback (see _knit_search_loop_finish).

• collect_fn -- [in] Optional collector callback (empty to skip).

Returns:

0 on a completed campaign, 1 when the proposer aborted it.

int _knit_search_parse_literal()

Parse one domain descriptor token into its structured form. The same grammar is used for a declared "--domain" value and for a sweep call-site token, so this is the single parser for both. The caller MUST declare the first output as an associative array (declare -A) and the second as an ordinary array.

Keys that do not apply to the parsed kind are set to the empty string. A "@glob:" pattern is stored verbatim (its expansion to a list happens later, at use time); a "@file:" path is read now, one value per line. A malformed or unrecognised token is fatal.

Example:

local -A desc; local values
_knit_search_parse_literal desc values '[1..10:2]'   # desc[kind]=range, lo=1, hi=10, step=2
_knit_search_parse_literal desc values '{a,b,c}'     # desc[kind]=list, values=(a b c)

Parameters:

• __knit_ret1 -- [out] Associative array to receive the descriptor scalars: keys kind (list|range|glob), lo, hi, step, glob.

• __knit_ret2 -- [out] Indexed array to receive the list values (empty unless kind=list).

• token -- [in] The descriptor token to parse.

int _knit_search_validate_descriptor()

Validate a parsed descriptor against a parameter's declared type, applying the type x kind rules. Fatals on any violation (the same hard-stop behaviour as the other declaration-time checks):

• a range needs an integer or real parameter; each bound and the step must be a valid value of that type;

• a glob needs a string parameter;

• a list is allowed for any type, and every value must be valid for it (an enum list thus restricts the enum to the named subset; a boolean list must be true/false);

• the --log scale hint applies only to a real range.

Parameters:

• __knit_desc -- [in] Name of the associative array holding the descriptor scalars (as produced by _knit_search_parse_literal).

• __knit_values -- [in] Name of the indexed array holding the list values.

• type -- [in] The parameter's declared type (name or alias).

• log -- [in] Whether the --log scale hint was given ("true" to enable).

Returns:

0 if the descriptor is valid (otherwise fatal).

set.sh

Functions

int _knit_set_add()

Add one or more elements to a set. Each element that is not already present is appended to the set's companion order array, so iteration reflects the order in which elements were first added. Duplicates are ignored (neither re-added nor re-ordered). The order array is created lazily, so this works on sets declared directly with declare -gA as well as those created via _knit_set_new().

Example:

_knit_set_add MY_SET "Shane" "Matthieu" "Rob"

Parameters:

• set_name -- [in] Name of the set in which to add the element.

• ...items -- [in] Elements to add to the set.

int _knit_set_array()

Return a set's elements, in insertion (declaration) order, as an indexed array written through a nameref output parameter. This is the fork-free alternative to while read … < <(_knit_set_iter …).

Example:

local -a members
_knit_set_array members MY_SET

Parameters:

• out -- [out] Name of the array variable to populate (nameref output).

• set_name -- [in] Name of the set to read.

int _knit_set_exists()

Checks if a set with the given name is defined, i.e. the variable is defined and it is an associative array.

Example:

_knit_set_exists MY_SET

Parameters:

• set_name -- [in] Name of the set.

int _knit_set_find()

Check if an element exists in a set.

Example:

_knit_set_find MY_SET "Phil"

Parameters:

• set_name -- [in] Name of the set in which to search.

• item -- [in] Item to find.

Returns:

0 if the element is found, 1 otherwise.

int _knit_set_iter()

Print each element of a set on its own line, in insertion (declaration) order.

Example:

_knit_set_iter MY_SET | while read -r key; do
    echo "Key: $key"
done

Parameters:

• set_name -- [in] Name of the set to iterate over.

int _knit_set_new()

Create a new empty set (associative array).

A set remembers the order in which elements were first added: alongside the backing associative array NAME, a companion indexed array NAME__order records insertion order so that _knit_set_iter() and _knit_set_array() yield elements in declaration order rather than the arbitrary hash order of the associative array's keys.

Example:

_knit_set_new MY_SET

Parameters:

• set_name -- [in] Name of the set to create.

int _knit_set_remove()

Remove one or more elements from a set, keeping the companion order array consistent.

Example:

_knit_set_remove MY_SET "Shane" "Matthieu"

Parameters:

• set_name -- [in] Name of the set from which to remove the elements.

• ...items -- [in] Elements to remove from the set.

setup.sh

Functions

int _knit_default_setup()

Body of the builtin "default" setup. It builds nothing: the setup exists only to carry the platform activation (inlined into .activate.sh by _knit_setup_default_after_cb) to jobs that declare no setup of their own.

int _knit_default_setup_path()

Print the directory where bootstrap instantiates the builtin "default" setup: <setup-root>/default, i.e. the reserved "default" instance under the experiment's setup root (see _knit_setup_root). It travels with the experiment (relative setup roots resolve against the experiment root) and survives the compute-side cd into a job directory. This is the path knit_with_setup "default" and the implicit-default job adoption resolve to when no --setup is given.

int _knit_experiment_root()

Store the experiment root — the directory that contains .knit — in the caller-named variable. Derived from _KNIT_PREFIX (already absolute) by stripping the trailing "/.knit" component, so it stays correct even after a compute-side cd into a job directory. Uses parameter expansion (no fork).

Parameters:

• __knit_ret -- [out] Name of the variable to hold the experiment root.

int _knit_has_user_setup()

Return 0 when at least one user setup instance exists directly under the given setup root, non-zero otherwise. A setup instance is a subdirectory of the root; the builtin "default" instance is not a user setup and is ignored, so a root that holds only "default" (or nothing) reports "no user setup". A missing root also reports "no user setup".

Parameters:

• root -- [in] Absolute setup root to scan (see _knit_setup_root).

int _knit_highlight_if_no_user_setup()

Highlight predicate (see knit_highlight_if) for the builtin "setup" command: return 0 ("highlight") when the experiment is bootstrapped but no setup other than the builtin "default" has been instantiated yet, non-zero otherwise. This bolds "setup" in the root "--help" once bootstrap is done — pointing at the natural next step — and leaves it plain as soon as the user has built a setup. Before bootstrap it never highlights (the setup root does not exist yet, and the command is filtered from "--help" anyway).

Parameters:

• cmd -- [in] The demangled command name (unused; the predicate is state-only).

int _knit_job_root()

Store the resolved job root — the directory under which jobs live — in the caller-named variable. Reads the verbatim job_path from the metadata table (falling back to "jobs" when unset, for robustness) and resolves it against the experiment root via _knit_resolve_experiment_path.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the resolved job root.

int _knit_resolve_experiment_path()

Resolve a stored path value into an absolute directory and store it in the caller-named variable. An absolute value (starting with "/") is returned unchanged; a relative value is resolved against the experiment root, so an experiment with relative roots stays portable across machines and clones.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the resolved path.

• stored -- [in] The stored path value (as typed at bootstrap).

int _knit_setup()

Entry point for the setup CLI command. Materializes a setup instance named --name under the experiment's setup root (<setup-root>/<name>), exports KNIT_SETUP_PREFIX to that directory, invokes the named setup subcommand inside it, and saves the resulting environment to $KNIT_SETUP_PREFIX/.activate.sh. On success the setup type is also recorded in $KNIT_SETUP_PREFIX/.setup.type so knit submit can validate a job's knit_with_setup requirement, and the setup body's row id in $KNIT_SETUP_PREFIX/.setup.id so a consumer can record a "used_by" edge to it. Removes the directory and fatals on failure.

The instance name must be a single path component (validated) and is idempotent by name: re-running with the same name rebuilds the same directory. The name "default" is reserved for the builtin default setup type.

Usage:

./exp.sh setup --name <name> -- <setup-type> [args...]

int _knit_setup_after_cb()

After-callback installed on every setup subcommand by knit_register_setup. Inlines the platform activation (${KNIT_PREFIX}/platform.sh) at the top of $KNIT_SETUP_PREFIX/.activate.sh, then writes the activation lines the setup body declared through the knit_setup_env* / knit_setup_activate_line functions, in call order. A job depending on this setup sources that file to reproduce the build environment. Inlining the platform (rather than sourcing) keeps .activate.sh self-contained and re-initializes the module system for consumers, above the declared lines.

The declared lines are composable: an append/prepend extends the job's own variable rather than replacing it, so nothing snapshots the login-node state. A setup that declares nothing produces a header-only .activate.sh. The Spack and launcher after-callbacks (registered after this one) append their blocks last, so activation order is header -> declared lines -> spack -> launcher.

int _knit_setup_before_cb()

Before-callback installed on every setup subcommand by knit_register_setup. Verifies that KNIT_SETUP_PREFIX is set, ensuring the setup was invoked through knit setup rather than called directly, then sources the platform environment so the setup body builds against the platform's modules and env. Also clears the declared-activation-line array so each setup invocation starts with an empty set of lines.

int _knit_setup_check_type()

Check that a setup directory was built by the required setup type. knit setup records the type in a .setup.type marker file inside the setup directory. Fatals if the marker is missing or its recorded type differs from the required one; returns normally on a match. Shared by knit submit (jobs) and by the generic setup-dependency before-callback used for every other command that declares knit_with_setup.

Parameters:

• setup_path -- [in] Path to the setup directory to check.

• required -- [in] Setup type the directory must have been built by.

int _knit_setup_default_after_cb()

After-callback for the builtin "default" setup. Unlike the generic setup after-callback it does NOT dump the environment: the default setup builds nothing, so re-exporting the bootstrap shell's environment would freeze unrelated login-node state into every job. It writes only the platform activation header, so .activate.sh carries the platform modules/environment (and is effectively empty when there is no profile).

int _knit_setup_dep_after_cb()

After-callback installed by knit_with_setup on a plain command or an app, alongside _knit_setup_dep_before_cb. It records a "used_by" edge from the setup the command depends on to the command itself. It runs as an after-callback (not the before-callback that sources the environment) because the consumer's frame is on the executing stacks only from push time onward, so the consumer's resolved row id — the edge target — is available here but not in the before-callback.

It resolves the same setup directory the before-callback used (via _knit_setup_dep_resolve_path), so the two always agree.

Parameters:

• required -- [in] Setup type the dependency must have been built by. Remaining arguments are the command's own runtime arguments, scanned for --setup.

int _knit_setup_dep_before_cb()

Before-callback installed by knit_with_setup on a plain command or an app. It resolves the setup directory the command depends on, checks its type, and sources its .activate.sh so the command body runs in the setup's environment. Jobs do not use this callback: their setup is resolved at submit time (_knit_submit) and re-sourced on the compute node (_knit_job_before_cb). Setups and wrappers cannot declare knit_with_setup at all (knit_with_setup rejects them).

The setup directory is resolved by _knit_setup_dep_resolve_path (the --setup option, else the ambient KNIT_SETUP_PREFIX, else the builtin default path when the required type is "default").

Parameters:

• required -- [in] Setup type the dependency must have been built by. Remaining arguments are the command's own runtime arguments, scanned for --setup.

int _knit_setup_dep_resolve_path()

Resolve the setup directory a knit_with_setup command depends on, by precedence: the command's --setup option (a setup instance name, resolved to <setup-root>/<name> via _knit_setup_name_to_path), else an already-set KNIT_SETUP_PREFIX (an absolute path, e.g. an app running inside a job inherits the job's setup), else — only when the required type is the builtin "default" — the auto-instantiated default setup path. Prints the resolved path (empty when nothing resolves). Shared by the before- and after-callbacks so they always agree on which directory is referenced.

Parameters:

• required -- [in] Setup type the dependency must have been built by. Remaining arguments are the command's own runtime arguments, scanned for --setup.

int _knit_setup_name_to_path()

Resolve a setup instance name into its absolute directory under the experiment's setup root: <setup-root>/<name> (see _knit_setup_root). The name is validated as a single path component first (fatal otherwise). This is how a --setup value is resolved for jobs, plain commands, and apps: users refer to a setup by the name they gave knit setup --name, not by its on-disk path.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the resolved setup directory.

• name -- [in] The setup instance name (as passed to --setup).

int _knit_setup_provides_launcher_after_cb()

After-callback installed by knit_provides_launcher, registered after the generic _knit_setup_after_cb (and any Spack re-activation block) so it appends to an already-written .activate.sh. It runs in the setup's own process, right after the body built and PATH-prepended its MPI, so the launcher is present on PATH by construction. It clears the detection cache, detects the launcher once against the active PATH, and freezes the concrete result by appending export KNIT_PROVIDED_LAUNCHER=<impl> to .activate.sh — the contract read by the launcher precedence (_knit_launch_backend). The value is also recorded as provenance via the mpi_launcher output.

If detection finds no launcher ("<unknown>") the setup declared it may provide one but built none reachable on PATH. Whether that is fatal depends on the machine: knit_provides_launcher means only "I may supply a launcher where the

machine has none", and the machine's own launcher (the profile's

launcher) wins over the setup's contract in _knit_launch_backend. So:

• When the machine has a concrete launcher (e.g. PALS on a Cray, whose mpiexec comes from a launcher module, not from the MPI the setup builds against), the setup's contract is subordinate and unused; a failed detection is expected and harmless. Skip the contract, record <unknown> provenance, and continue — a portable setup that unconditionally declares knit_provides_launcher is correct here (the profile wins).

• When the machine has no launcher (a laptop; launcher is <unknown>, "none", or unset), the setup's contract is the only possible source, so a failed detection is fatal — an early, clear failure rather than a silent degrade to the "none" backend at run time.

Trailing arguments are the setup's runtime arguments and are ignored.

int _knit_setup_record_used_by_edge()

Record a "used_by" provenance edge from the setup built at <setup_path> to a consuming invocation. The edge's source is the setup (its row id read from the setup directory's .setup.id marker, its name "setup:<type>" from .setup.type); its target is the consumer. A "used_by" edge has no duration, so both timestamps are NULL. Shared by knit submit (jobs) and by the generic setup-dependency after-callback (plain commands and apps).

It reads the setup directory's .setup.id / .setup.type markers and delegates the gated write to _knit_record_used_by_edge (which records nothing when recording is disabled, on a suppressed rank, before bootstrap, or when the target does not participate in the graph). A setup directory with no .setup.id (e.g. built before provenance shipped) yields no edge.

Parameters:

• setup_path -- [in] Path to the setup directory the consumer references.

• target_cmd -- [in] Mangled command name of the consumer (the edge target).

• target_id -- [in] Resolved row id of the consumer (the edge target).

int _knit_setup_require_body()

Guard shared by the declarative activation functions (knit_setup_env_* / knit_setup_activate_line): fatal unless the caller runs from inside a setup body. A setup body is the case where the currently executing command has type "setup"; this excludes a job body (which also sets KNIT_SETUP_PREFIX, because it sources the setup's environment) and any use outside a registered command.

Parameters:

• fn -- [in] Name of the calling function, used in the error message.

int _knit_setup_root()

Store the resolved setup root — the directory under which setup instances live — in the caller-named variable. Reads the verbatim setup_path from the metadata table (falling back to "setups" when unset, for robustness) and resolves it against the experiment root via _knit_resolve_experiment_path.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the resolved setup root.

int _knit_setup_source_platform()

Source the materialized platform fragment (${_KNIT_PREFIX}/platform.sh) into the current shell so a setup builds with the platform's modules and environment (e.g. mpicc/cmake from modules) already active. A guarded no-op when the file is absent (no profile, or the profile declared neither modules nor environment). Shared by the generic setup before-callback and the Spack-env before-callback.

int _knit_setup_validate_var()

Validate that a variable name given to a declarative activation function is a valid shell identifier (^[A-Za-z_][A-Za-z0-9_]*$). Fatal otherwise; returns normally on a match.

Parameters:

• var -- [in] The variable name to validate.

int _knit_setup_write_activate_header()

Print the header of a setup's .activate.sh to stdout: the shebang, a generated marker, and — when the materialized platform fragment (${_KNIT_PREFIX}/platform.sh) exists — its contents inlined. Inlining (rather than sourcing) keeps .activate.sh self-contained and re-initializes the module system for consumers. When there is no profile the platform fragment is absent and only the shebang/marker are emitted. Shared by the generic setup after-callback (which appends the setup's declared activation lines) and the builtin "default" setup's after-callback (which emits nothing more).

int _knit_stdin_is_terminal()

Return success if standard input is an interactive terminal. Factored out (and shared by the environment-manager stdin-manifest guard, prepare, and remove) so tests can stub it: a real terminal is unavailable under bats, so the no-manifest guard cannot be exercised without this seam.

int _knit_validate_instance_name()

Validate a setup or job instance name. A name must be a single path component: it matches ^[A-Za-z0-9._-]+$ (letters, digits, dot, underscore, hyphen; no slashes). Fatals with guidance on anything else; returns normally on a match.

Parameters:

• name -- [in] The instance name to validate.

Variables

AssociativeArray _KNIT_SETUPS

Associative array mapping registered setup names to 1. Used to validate that a setup name passed to knit setup is known. Declared global (-g) so the builtin "default" entry, populated at load time, survives even when knit.sh is sourced from within a function (as the bats tests do).

Array _KNIT_SETUP_ACTIVATE_LINES

Per-invocation array of activation lines a setup body declared through the knit_setup_env_* / knit_setup_activate_line functions. Reset to empty at the start of each setup's execution (in _knit_setup_before_cb) and read by the generic setup after-callback, which writes the lines into $KNIT_SETUP_PREFIX/.activate.sh in call order. Declared global (-g) so the body functions and the after-callback share the same array.

shorthand.sh

Functions

int _knit_shorthand_find_function()

Discover the name a name-extracting registration shorthand (@command, @job, @app, @setup, @wrapper) decorates. The shorthand captures its own call site and passes it here; the name is read from the source rather than taken as an argument.

The source file is read and the lines after "line" are scanned. Blank lines, comments, and intervening decorator lines (@with_*, @usable_if, ...) are skipped. The scan stops at the first of:

• an "@empty" marker line -> the discovered body name is "knit_empty";

• a function definition -> its name is returned. All common styles are recognised: "name() {", "name () ...", "function name ...", one-liners, and names containing "@" (bash allows them).

The discovery is textual, so the function need not be defined yet when the shorthand runs.

Fatal when the source is not a readable file (piped/eval'd script), and when neither a function definition nor an "@empty" marker is found before the next "@done"/"knit_done" or the end of the file.

Parameters:

• The -- [out] name of the variable that receives the discovered name.

• file -- [in] The source file to read (BASH_SOURCE of the call site).

• line -- [in] The 1-based line number of the shorthand call (BASH_LINENO).

int _knit_shorthand_generate()

Define the "@" shorthand functions, honouring KNIT_WITHOUT_SHORTHAND. Runs once when knit.sh is sourced. For each non-opted-out pass-through token it defines "@<token>() { <target> "$"; }"; for each non-opted-out extracting

token it defines a wrapper that discovers the decorated function's name and

injects it at argument position 2. Any KNIT_WITHOUT_SHORTHAND token that is not

a real shorthand produces a warning and is otherwise ignored; the special

value "all" suppresses every shorthand.

int _knit_shorthand_opted_out()

Return success when a shorthand token is suppressed by KNIT_WITHOUT_SHORTHAND, either because the token is listed or because the special value "all" is present. Surrounding whitespace around each listed token is trimmed, so "job, app" behaves like "job,app". Used by knit_register_shorthand to honour the same opt-out the source-time generator honours.

Parameters:

• token -- [in] The shorthand token (the part after "@").

Variables

AssociativeArray _KNIT_SHORTHAND_EXTRACTOR

Curated map from a name-extracting shorthand token to the knit_register* function it forwards to. These discover the decorated function's name from the source and inject it at argument position 2. The wrappers are generated in a later step; the map is declared here so opt-out validation already recognises these tokens as real shorthands.

AssociativeArray _KNIT_SHORTHAND_PASSTHROUGH

The "@" shorthand layer. This file defines a terse, decorator-style twin for every declaration/decoration function in the public API: "knit_x" gains an "@x" twin (with the register family remapped: knit_register -> @command, knit_register_<x> -> <x>). The shorthand is additive — the knit_* functions are unchanged and remain the canonical, stable API — and it is enabled by default. A user opts out by setting KNIT_WITHOUT_SHORTHAND before sourcing knit (a comma-separated list of tokens, or "all").

The curated set of shorthands lives in the two maps below. It cannot be derived mechanically from the knit_* prefix, because that prefix also covers runtime helpers (knit_get_parameter, the loggers, ...) which get no shorthand.

Pass-through shorthands forward their arguments verbatim to the target (@x() { knit_x "$@"; }). The extracting shorthands (the register family) are wired up in a later step: they discover the decorated function's name from the source rather than taking it as an argument.

Curated map from a pass-through shorthand token (the part after "@") to the knit_* function it forwards to verbatim. These take no function-name argument and are not name-extracting: the generated wrapper simply forwards "$@".

skills.sh

Functions

int _knit_skills_discover()

int _knit_skills_download()

Download the knit repository at a ref as a tarball and extract it into the destination directory, so <dest>/agent/ holds the skills and commands. Uses curl and tar (as the sqlite/jq/spack provisioning does) so knit needs no git, and --strip-components=1 drops the archive's single "knit-<ref>/" top-level directory without having to resolve the ref to a commit first. GitHub serves any ref at archive/<ref>.tar.gz.

Parameters:

• dest -- [in] Destination directory (already created by the caller).

• ref -- [in] Git ref (branch, tag, or commit SHA) to download.

int _knit_skills_install()

Implementation of 'knit skills install'. Downloads the latest agent/ tree from the knit repository and installs the skills and commands into ".agents/", the canonical cross-harness location. The copy merges into any existing skills/commands directory (so other, non-knit skills there are preserved) and overwrites knit's own, which makes a re-install idempotent.

With --claude, Claude Code is pointed at that same install by symlinking knit's skills and commands into ".claude" one item at a time (see _knit_skills_link_claude), so there is one real copy on disk, not a per-harness fork, and a project's own ".claude" items are left in place.

It also drops the agent/AGENTS.md pointer at the project root so an agent orients itself even before a skill is loaded. An existing project AGENTS.md is never overwritten (it may hold the user's own notes); the pointer is written only when no AGENTS.md is present.

int _knit_skills_link_claude()

Point Claude Code at the canonical ".agents" install by symlinking knit's items into ".claude/skills" and ".claude/commands", one entry at a time. The two containers are kept as real directories (created if missing, and an existing one — even a symlink — is left as is), so a project's own skills/commands and its ".claude/settings.json" live alongside the links rather than being replaced.

Each immediate entry of ".agents/skills" (a skill directory) and ".agents/commands" (a command file) is symlinked into the matching container. The target is absolute (${PWD}/.agents/...) so the link is valid wherever the container physically lives — in particular when the container is itself a symlink, where a relative target would resolve from the wrong place. An entry name already present is skipped with a warning rather than overwritten, so a user's own item is never clobbered; an entry that is already the exact link this would create is skipped silently, which makes a re-install idempotent.

Variables

String _KNIT_SKILLS_DEFAULT_REF

Git ref the install resolves against when no --ref is given. This is the default branch, not the running knit version tag, because the agent/ tree on the default branch carries the most up-to-date skills (mirrors the profile store convention).

String _KNIT_SKILLS_REPO

GitHub repository (org/repo) that serves the agent skills and commands tree under agent/. The install downloads this repository's archive.

sqlite.sh

Variables

String _KNIT_DATABASE

Path to the Knit database file.

String _KNIT_SQLITE_EXE

Path to the SQLite executable.

String _KNIT_SQLITE_SOURCE_NAME

Name of the SQLite source archive.

String _KNIT_SQLITE_SOURCE_URL

URL to download the SQLite source archive.

str.sh

Functions

int _knit_str_hyphens_to_underscores()

Convert hyphens to underscores, storing the result in the caller-named variable.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the result.

• input -- [in] String to convert.

int _knit_str_render_cmd()

Render an argument vector, passed by array name, as a single shell-safe string: each element is q-quoted and the elements are joined by single spaces. An empty array renders to the empty string. Used to capture the resolved launcher and scheduler commands for the "native_cmd" columns of the runs and jobs tables and for the trace log emitted before each command is issued.

Parameters:

• argv_name -- [in] Name of the array holding the command and its arguments.

int _knit_str_underscores_to_hyphens()

Convert underscores to hyphens, storing the result in the caller-named variable.

Parameters:

• __knit_ret -- [out] Name of the variable to hold the result.

• input -- [in] String to convert.

sweep.sh

Functions

int _knit_sweep()

Entry point for the sweep CLI command. The sweep's own options are parsed and validated by the framework before the body runs (the control options are declared with "--no-record" so they validate but add no table column). This body splits the downstream off the first "--" and validates that a downstream command is present.

The scan (M2), declared-domain resolution (M3), odometer and line reconstruction (M4), and execution (M5) are added in later milestones; for now the body stops after the split so the command registers, parses, and records a campaign row with the fixed column set.

Usage:

./exp.sh sweep [SWEEP ARGS] -- <downstream knit command, with sweep-syntax values>

int _knit_sweep_ask()

The sweep's "ask" callback for knit_search_loop: yield the next cartesian product combination as a reconstructed downstream command line, then advance the odometer cursor. Like every search proposer (see the fake proposer in tests/test_search_loop.sh), it reads its state from the enclosing dispatcher frame through dynamic scope -- here _knit_sweep's locals: the plan arrays (template/axes/flat/offsets/counts), the product size (total), and the cursor it advances. Returns 1 when the product is exhausted.

Parameters:

• __knit_ret_tag -- [out] The opaque trial tag (the combination index).

• __knit_ret_argv -- [out] The reconstructed downstream argv.

int _knit_sweep_dimensions_json()

Build the campaign's "dimensions" JSON object: one key per axis (its command-qualified "<segment>.<option>" name) mapping to the ordered array of that axis's values, e.g. {"submit.nodes":[1,2,4],"montecarlo.samples":[1000,100000]}. The qualified key keeps two same-named axes in different segments distinct. Numeric-looking values are emitted as JSON numbers, others as strings. This is the single JSON column recorded on the sweeps row (so a 1-node command and a 1024-node job produce rows of the same shape).

Parameters:

• __knit_ret -- [out] Name of the variable to receive the JSON string.

• __knit_axes -- [in] Name of the axis-descriptor array.

• __knit_flat -- [in] Name of the flattened axis-values array.

• __knit_off -- [in] Name of the per-axis offset array.

• __knit_cnt -- [in] Name of the per-axis count array.

int _knit_sweep_downstream()

Split a sweep invocation's arguments at the first "--" and return, by nameref, the downstream token list: everything after that first "--". The downstream is the real knit command the sweep wraps (e.g. "submit ... -- <job> ..." or a bare "<command> ..."); any inner "--" it carries (a submit/prepare/run line has one) is preserved, because only the first "--" separates the sweep's own options from the downstream.

Parameters:

• __knit_ret -- [out] Name of the array to receive the downstream tokens.

• ... -- [in] The sweep invocation arguments.

int _knit_sweep_enumerate()

Enumerate one literal axis descriptor token into its concrete list of values. The token is parsed through the shared search-core parser, then the parsed descriptor is enumerated by kind: a list (or

int _knit_sweep_enumerate_domain()

Enumerate a parameter's declared search domain into its concrete value list, applying the enumerability gate the proposer owns. A real range (kind "range", type "real") is bounds-only (a Bayesian-optimization proposer samples it; a sweep cannot enumerate it), so asking to sweep one is a fatal with the fix message. Every other declared domain (a list, a glob, an enum list, or an integer range) enumerates like a call-site literal of the same shape.

Parameters:

• __knit_ret -- [out] Name of the array to receive the value list.

• mangled -- [in] The command the parameter belongs to (mangled name).

• param -- [in] The parameter name (normalized).

int _knit_sweep_expand_glob()

Expand a "@glob:<pattern>" axis into its list of matching paths, at scan time. Pathname expansion is done with nullglob on (so a non-matching pattern yields an empty list, not the literal pattern), and a glob that matches nothing is a fatal: a zero-value axis would collapse the whole product to zero runs, which is almost always a mistake (a typo or the wrong working directory).

Parameters:

• __knit_ret -- [out] Name of the array to receive the matching paths.

• pattern -- [in] The glob pattern (already stripped of its "@glob:" prefix).

int _knit_sweep_observe()

The sweep's "observe" callback for knit_search_loop: apply the --on-failure policy to one finished trial. The sweep reads no objective (it is the no-model proposer), so it uses only the terminal status: under "stop" a failed child aborts the campaign (return non-zero); under "continue" the whole product runs regardless. The policy is read from _knit_sweep's frame (dynamic scope).

Parameters:

• $1 -- [in] The trial tag (unused).

• $2 -- [in] The child row id (unused).

• $3 -- [in] The trial's terminal status ("completed" or "failed").

• $4 -- [in] The objective (unused; the sweep has no collector).

int _knit_sweep_plan()

Build the odometer plan from the resolved per-axis value lists. Each axis's newline-joined value list is split into the shared "flat" array; its start offset and count let the reconstructor read any axis value by (axis, index) without an array of arrays (which bash lacks). The total is the product of the per-axis counts (1 when there are no axes: the single literal combination).

Parameters:

• __knit_ret1 -- [out] Name of the array to receive the per-axis value counts.

• __knit_ret2 -- [out] Name of the array to receive all axis values, flattened.

• __knit_ret3 -- [out] Name of the array to receive each axis's start offset.

• __knit_ret4 -- [out] Name of the variable to receive the product total.

• __knit_vals -- [in] Name of the per-axis value-list array (newline joined).

int _knit_sweep_range_grid()

Enumerate a range descriptor into its concrete grid of values. Two cases:

• no step ("[lo..hi]"): an integer grid with step 1, ascending or descending (lo may be greater than hi). The bounds must be integers; a bare real interval is bounds-only (a BO samples it, a sweep cannot enumerate it) and is a fatal with the fix message.

• an explicit step ("[lo..hi:step]"): a linear grid from lo toward hi by the step magnitude (integer or real; real arithmetic is done with awk so a fractional step lands exactly). Descending is allowed (lo greater than hi).

Parameters:

• __knit_ret -- [out] Name of the array to receive the grid values.

• lo -- [in] The low (first) bound.

• hi -- [in] The high (last) bound.

• step -- [in] The step, or empty for the integer step-1 grid.

• token -- [in] The original token, for error messages.

int _knit_sweep_reconstruct()

Rebuild one combination's downstream command line from the template: copy the template, then overwrite each axis's template slot with that axis's value for the given combination index. The index is decoded as a mixed-radix odometer in which the last axis varies fastest (so the product enumerates in a stable, row-major order).

Parameters:

• __knit_ret -- [out] Name of the array to receive the reconstructed line.

• __knit_tmpl -- [in] Name of the reconstruction template array.

• __knit_axes -- [in] Name of the axis-descriptor array.

• __knit_flat -- [in] Name of the flattened axis-values array.

• __knit_off -- [in] Name of the per-axis offset array.

• __knit_cnt -- [in] Name of the per-axis count array.

• idx -- [in] The combination index (0 <= idx < total).

int _knit_sweep_resolve_domains()

Resolve the declared-domain axes (@domain / @default / --fill), splicing the results into the M2 scan. It runs only when it must: --fill domain, or a @domain/@default marker at the call site. With --fill default and no marker it returns at once, having read nothing.

Two passes:

• markers: each call-site @domain/@default axis is resolved against the command of the segment it sits in (knit_search_segment_command), so the same option name in two segments (e.g. "submit --nodes @domain -- job

  --nodes @domain") resolves against two different commands. A @domain enumerates the declared domain (via _knit_sweep_enumerate_domain) and fills the marker axis in place; a @default pins the marker to the resolved default and drops it as an axis.

• --fill domain: for each parameter of the downstream TARGET left unspecified on the line that has a declared domain, append a new "--param <value>" axis. (A dispatcher-segment parameter is varied only when marked explicitly, not by --fill.)

An explicit @domain on a parameter with no declared domain is a fatal; a @domain / @default naming a parameter its segment does not have is a fatal.

Parameters:

• __knit_ret1 -- [inout] Name of the reconstruction template array.

• __knit_ret2 -- [inout] Name of the axis-descriptor array.

• __knit_ret3 -- [inout] Name of the per-axis value-list array.

• __knit_seen -- [in] Name of the seen map (option -> classification).

• fill -- [in] The --fill policy ("default" or "domain").

• ... -- [in] The downstream tokens (used to resolve each axis's segment to its own command).

int _knit_sweep_scan()

Thin wrapper around knit_search_scan that adds literal-axis enumeration. Delegates classification to knit_search_scan, then for each axis whose kind is "literal" calls _knit_sweep_enumerate on the raw token and stores the newline-joined value list in __knit_ret3. The output contract is unchanged from before the refactor: __knit_ret3[i] is a newline-joined value list for a "literal" axis and empty for a "domain"/"default" marker.

Parameters:

• __knit_ret1 -- [out] Name of the array to receive the reconstruction template (a copy of the downstream tokens).

• __knit_ret2 -- [out] Name of the array to receive the axis descriptors.

• __knit_ret3 -- [out] Name of the array to receive the per-axis value lists (newline-joined; empty for "domain"/"default").

• __knit_ret4 -- [out] Name of the associative array to receive the seen map.

• passthrough_csv -- [in] Comma-separated option names whose values are never treated as sweep-syntax (may be empty).

• ... -- [in] The downstream tokens.

int _knit_sweep_segment_has_param()

Return success when command $1 declares a parameter named $2 (normalized).

Parameters:

• cmd -- [in] The mangled command name.

• want -- [in] The normalized parameter name to look for.

Variables

String _KNIT_SWEEPS_TABLE

Name of the table recording every sweep campaign. The row id is the sweep's own UUID. Each child point (a submitted/run/command invocation, or a prepared row) records an automatic "call" edge back to this row through the provenance graph, so a campaign's children are found by walking the edges, not by a stored column. The column set is fixed and kind-independent: id, the recorded "group" label, and the "target"/"dispatcher"/"dimensions"/"count" outputs. The control options (--on-failure/--max-inflight/--passthrough/--fill/--dry-run) are declared with "--no-record" so they are parsed and validated but add no column.

types.sh

Functions

int _knit_type_check_date()

Validate that a string is a date in YYYY-MM-DD format with valid ranges.

Parameters:

• value -- [in] String to validate.

Returns:

0 if valid, 1 otherwise.

int _knit_type_check_time()

Validate that a string is a time in hh:mm:ss format with valid ranges.

Parameters:

• value -- [in] String to validate.

Returns:

0 if valid, 1 otherwise.

int _knit_type_is_checksummable()

Return 0 if a type name (or alias) refers to a target whose content Knit checksums — the "file" and "directory" types (and the "dir" alias). Every other type, including "path" and "filename", is not checksummable. This is the single predicate the declaration and recording paths consult to decide whether a parameter or output gains a companion checksum column.

Parameters:

• type_name -- [in] Type name or alias to test.

Returns:

0 if checksummable, 1 otherwise.

int _knit_type_resolve_alias()

Resolve a type name or alias to its canonical type name. If the name is already a canonical built-in type or an enum, it is returned as-is. If it is an alias, the corresponding canonical name is printed.

Example:

local t; _knit_type_resolve_alias t "int"      # t == "integer"
local t; _knit_type_resolve_alias t "integer"  # t == "integer"
local t; _knit_type_resolve_alias t "color"    # t == "color" (if enum defined)

Parameters:

• __knit_ret -- [out] Name of the variable to hold the resolved type name.

• type_name -- [in] Type name or alias to resolve.

Returns:

0 if resolved successfully, 1 if the name is unknown.

int _knit_type_to_sqlite()

Map a Knit type name (or alias) to its corresponding SQLite type affinity. Returns INTEGER for integer, REAL for real, and TEXT for all other types (including boolean, string, path, file, directory, filename, date, time, datetime, uuid, and user-defined enums).

Example:

local t; _knit_type_to_sqlite t "integer"  # t == INTEGER
local t; _knit_type_to_sqlite t "real"     # t == REAL
local t; _knit_type_to_sqlite t "uuid"     # t == TEXT
local t; _knit_type_to_sqlite t "int"      # t == INTEGER (alias resolved)

Parameters:

• __knit_ret -- [out] Name of the variable to hold the SQLite type affinity.

• type_name -- [in] Knit type name or alias.

Returns:

0 on success, 1 if the type is unknown.

Variables

AssociativeArray _KNIT_BUILTIN_ENUMS

Set of enum type names that have been marked as framework builtins (via _knit_is_builtin). Used to distinguish knit's own enums from user-defined ones.

AssociativeArray _KNIT_BUILTIN_TYPES

Set of built-in canonical type names.

String _KNIT_CURRENT_ENUM

Name of the enum whose definition is currently open for extension, i.e. set by knit_extend on an enum and cleared by the matching knit_done. Unset when no enum-edit context is active. This is the enum counterpart of _KNIT_CURRENT_COMMAND; knit_enum_value appends to this enum.

AssociativeArray _KNIT_ENUMS

Set of user-defined enum type names.

String _KNIT_LAST_ENUM

Name of the most recently defined enum (set by knit_enum). Consulted by _knit_is_builtin when called outside a command registration, so a builtin enum can be marked immediately after its definition.

AssociativeArray _KNIT_TYPE_ALIASES

Associative array mapping type alias names to their canonical type names.