Jobs

Jobs#

Submitting jobs to the scheduler and tracking their lifecycle.

Register and submit a job#

Register a command as a job and submit it to the scheduler with knit submit.

APIs: knit_register_job, submit, knit_with_setup

Register a job with knit_register_job instead of knit_register. A job is a command that runs asynchronously — knit hands it to the batch scheduler (Slurm, PBS, …) rather than running it in the foreground — and, on a cluster, on other machines. The declaration block is otherwise identical to a plain command: bind it to a software environment with knit_with_setup, declare its parameters, and define the body:

@job "julia" "Render a Julia-set fractal as a submitted job."
@with_setup "juliaenv"
@with_optional "width:integer"    "800"    "Image width in pixels."
@with_optional "height:integer"   "600"    "Image height in pixels."
@with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
@with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
@with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    # A submitted job already runs with its working directory set to its own job
    # directory (exported as KNIT_JOB_PREFIX), so a bare relative output PNG
    # would land there too. We build an absolute path anyway: it is explicit
    # about where the image belongs and stays correct even if the body (or a
    # program it launches) changes directory first.
    local png="${KNIT_JOB_PREFIX%/}/fractal.png"
    julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
        "${png}" "${colormap}"
}
@done

knit_register_job "julia" _julia "Render a Julia-set fractal as a submitted job."
knit_with_setup "juliaenv"
knit_with_optional "width:integer"    "800"    "Image width in pixels."
knit_with_optional "height:integer"   "600"    "Image height in pixels."
knit_with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
knit_with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
knit_with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    # A submitted job already runs with its working directory set to its own job
    # directory (exported as KNIT_JOB_PREFIX), so a bare relative output PNG
    # would land there too. We build an absolute path anyway: it is explicit
    # about where the image belongs and stays correct even if the body (or a
    # program it launches) changes directory first.
    local png="${KNIT_JOB_PREFIX%/}/fractal.png"
    julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
        "${png}" "${colormap}"
}
knit_done

Jobs are subcommands of the builtin submit dispatcher, so you launch one by naming it after --. Scheduler options (nodes, walltime, queue, …) go before the --; the job’s own parameters go after it:

$ ./exp.sh submit --nodes 2 --walltime 01:00:00 -- julia --width 1920 --height 1080
018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a

submit prints the new job’s UUID on stdout and returns immediately (add --wait to block; see Wait for and inspect a job). Capture the UUID to track the job later:

$ id=$(./exp.sh submit --nodes 2 -- julia --width 1920)
$ ./exp.sh job status --id "$id"

If a job declares knit_with_setup, name the setup instance to run in with submit --setup <name>. See Depend on a setup for how a job is bound to its environment, and Provide the default setup for the environment a job adopts when it declares no setup.

List the job types you can submit#

Discover registered jobs with submit –help and submit one with the – dispatcher syntax.

APIs: submit

submit is a dispatcher, not an ordinary parent command: the jobs registered with knit_register_job are not nested subcommands of it, they are the tokens you pass after --. submit --help lists them under a Jobs heading:

$ ./exp.sh submit --help
Usage: ./exp.sh submit [OPTIONS] -- <job> [OPTIONS]
...
Jobs
----
  julia   Render a Julia-set fractal as a submitted job.

The usage line shows the shape to follow. A job is submitted as submit [scheduler options] -- <jobname> [job options] — the -- separates the scheduler options (which belong to submit) from the job’s own options:

$ ./exp.sh submit --nodes 2 -- julia --width 1920

This is the one place Knit’s invocation syntax departs from ordinary subcommand nesting. Writing ./exp.sh submit julia --width 1920 (with no --) does not submit the julia job — julia would be read as a stray argument to submit itself. The -- is required.

To see a specific job’s parameters, name it before --help (no -- needed for the help form): ./exp.sh submit julia --help.

Request resources for a job#

Ask the scheduler for nodes, walltime, and GPUs with submit –nodes/–walltime/–gpus-per-node.

APIs: submit

The resources a job needs are submit options, so they go before the --:

$ ./exp.sh submit --nodes 4 --walltime 02:30:00 --gpus-per-node 2 -- julia

  • --nodes (default 1) is the number of nodes. Knit allocates whole nodes exclusively: the per-node core count comes from the machine profile (or bootstrap detection), so there is no per-job CPU-count option — a job gets all the cores on each node it is given.

  • --walltime is the wall-clock limit as HH:MM:SS. If you omit it, knit fills one in, in this order: the project default set with bootstrap --default-walltime, then the selected queue’s default (or maximum) from the machine profile, and finally one hour. Pass --walltime none to run with no wall-clock limit — honored on the local, none, and pilot schedulers (which do not cap a job’s run), and rejected on batch schedulers (Slurm, PBS, Flux), where every job runs under a queue walltime cap.

  • --gpus-per-node (default 0) requests GPUs per node on a GPU partition.

These values become the scheduler directives in the generated batch script (which you can inspect with job show:script) and are recorded in the jobs table, so a run’s resource request travels with its records.

Name a job and give it a stable alias#

Distinguish submit –name (a knit alias for the job) from –job-name (the scheduler’s name).

APIs: submit

submit has two different “name” options, for two different audiences.

--name is a stable alias inside knit for this job instance. Knit symlinks <job-root>/<name> to the job’s <uuid> directory and records the name in the jobs table, so you can refer to a run by a memorable name instead of a UUID. It must be unique within the experiment:

$ ./exp.sh submit --name hi-res-run -- julia --width 3840

--job-name is the name the scheduler shows (#SBATCH --job-name / #PBS -N) — what appears in squeue / qstat. It defaults to the experiment script name:

$ ./exp.sh submit --job-name julia-nightly -- julia

The two are independent: --name organizes runs within knit and never reaches the scheduler, while --job-name only labels the job in the queue and is not a handle knit tracks. Set either, both, or neither.

Set the scheduler account, project, and queue#

Override the accounting and queue for one submission, and set project-wide defaults at bootstrap.

APIs: submit, bootstrap

Clusters charge jobs to an account/project and run them in a named queue (or partition). submit takes all three as options; they go before the --:

$ ./exp.sh submit --account m1234 --queue debug -- julia

Each one defaults so you rarely pass it per submission:

  • --account defaults to the __account__ metadata.

  • --project defaults to the __project__ metadata.

  • --queue defaults to the __default_queue__ metadata, then to the machine profile’s default queue.

Set the metadata defaults once, at bootstrap, so every submit inherits them:

$ ./exp.sh bootstrap --account m1234 --project julia-sets

After that, a bare submit uses those values, and the options above are only for overriding them on a single run. The resolved values are written into the batch script’s directives and recorded with the job.

Let knit pick the queue with --queue auto: it walks the machine profile’s declared queues in order and selects the first whose node and walltime bounds accept the job as you specified it.

$ ./exp.sh submit --queue auto --nodes 4 --walltime 02:00:00 -- julia

Selection uses only what you asked for — the node count, and the walltime only when you gave one — and resolves to a concrete queue that is recorded with the job (never the literal auto). If no declared queue fits, submit stops with a per-queue explanation instead of letting the scheduler reject the request. Make it the default for every submission by storing it at bootstrap, or by setting default_queue to "auto" in the machine profile:

$ ./exp.sh bootstrap --default-queue auto

Find the job’s hosts and node count#

Read a running job’s allocated hosts and node count from its body.

APIs: knit_job_hostnames, knit_job_nodecount

Inside a job body, knit_job_hostnames and knit_job_nodecount report the placement the scheduler gave the job — which nodes it got and how many:

@job "julia" "Render a Julia-set fractal as a submitted job."
@with_setup "juliaenv"
@with_optional "width:integer"    "800"    "Image width in pixels."
@with_optional "height:integer"   "600"    "Image height in pixels."
@with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
@with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
@with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    local png="${KNIT_JOB_PREFIX%/}/fractal.png"

    # Report where the scheduler placed this job.
    printf 'The job is running on hosts: %s\n' \
        "$(knit_job_hostnames --separator ', ')"

    # Was (Step 4): julia-fractal "${width}" ... "${png}" "${colormap}"
    # Now: launch the render app instead of running the binary here, with one rank
    # per allocated node. knit_job_nodecount is 1 on a laptop, so this runs
    # anywhere; on a cluster it scales with the nodes the job was given (submit
    # --nodes N). The values we used to pass positionally become named parameters.
    knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render \
        --width "${width}" --height "${height}" \
        --c-re "${c_re}" --c-im "${c_im}" --max-iter "${max_iter}" \
        --colormap "${colormap}" --output "${png}"
}
@done

knit_register_job "julia" _julia "Render a Julia-set fractal as a submitted job."
knit_with_setup "juliaenv"
knit_with_optional "width:integer"    "800"    "Image width in pixels."
knit_with_optional "height:integer"   "600"    "Image height in pixels."
knit_with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
knit_with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
knit_with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    local png="${KNIT_JOB_PREFIX%/}/fractal.png"

    # Report where the scheduler placed this job.
    printf 'The job is running on hosts: %s\n' \
        "$(knit_job_hostnames --separator ', ')"

    # Was (Step 4): julia-fractal "${width}" ... "${png}" "${colormap}"
    # Now: launch the render app instead of running the binary here, with one rank
    # per allocated node. knit_job_nodecount is 1 on a laptop, so this runs
    # anywhere; on a cluster it scales with the nodes the job was given (submit
    # --nodes N). The values we used to pass positionally become named parameters.
    knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render \
        --width "${width}" --height "${height}" \
        --c-re "${c_re}" --c-im "${c_im}" --max-iter "${max_iter}" \
        --colormap "${colormap}" --output "${png}"
}
knit_done

knit_job_hostnames prints the allocated hosts, one per line by default. Shape the output with:

  • --separator ', ' — join the hosts with a custom separator instead of newlines.

  • --json — emit a JSON array (handy for passing hosts to another tool).

  • --raw — keep the scheduler’s verbatim host list, including any repeats (e.g. one line per slot); the default deduplicates and keeps first-seen order.

  • --select <start>:<length> — take a slice of the (0-based) host list, e.g. --select 0:1 for just the first host.

knit_job_nodecount prints the number of distinct nodes (the deduplicated host count) — so a host offering several slots counts once. It pairs naturally with knit run --procs-per-node 1 to place exactly one rank per node.

Both work off a scheduler too: on a laptop knit_job_hostnames reports the local host and knit_job_nodecount is 1, so the same body runs anywhere. (The knit run line above launches an MPI app across the allocation; that is covered in the Apps category.)

Write results into the job’s directory#

Use KNIT_JOB_PREFIX, the job’s own working directory, to place output files.

APIs: KNIT_JOB_PREFIX

Every job runs in its own directory under the job root (<job-root>/<uuid>), exported to the body as KNIT_JOB_PREFIX and already set as the process’s current directory. Anything the job writes there travels with the run’s records, so this is where output files belong:

@job "julia" "Render a Julia-set fractal as a submitted job."
@with_setup "juliaenv"
@with_optional "width:integer"    "800"    "Image width in pixels."
@with_optional "height:integer"   "600"    "Image height in pixels."
@with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
@with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
@with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    # A submitted job already runs with its working directory set to its own job
    # directory (exported as KNIT_JOB_PREFIX), so a bare relative output PNG
    # would land there too. We build an absolute path anyway: it is explicit
    # about where the image belongs and stays correct even if the body (or a
    # program it launches) changes directory first.
    local png="${KNIT_JOB_PREFIX%/}/fractal.png"
    julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
        "${png}" "${colormap}"
}
@done

knit_register_job "julia" _julia "Render a Julia-set fractal as a submitted job."
knit_with_setup "juliaenv"
knit_with_optional "width:integer"    "800"    "Image width in pixels."
knit_with_optional "height:integer"   "600"    "Image height in pixels."
knit_with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
knit_with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
knit_with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    # A submitted job already runs with its working directory set to its own job
    # directory (exported as KNIT_JOB_PREFIX), so a bare relative output PNG
    # would land there too. We build an absolute path anyway: it is explicit
    # about where the image belongs and stays correct even if the body (or a
    # program it launches) changes directory first.
    local png="${KNIT_JOB_PREFIX%/}/fractal.png"
    julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
        "${png}" "${colormap}"
}
knit_done

Because KNIT_JOB_PREFIX is already the working directory, a bare relative path (fractal.png) lands there too. Building an absolute path from KNIT_JOB_PREFIX — as the body above does — is still worth it: it is explicit about where the file belongs and stays correct even if the body, or a program it launches, changes directory first.

The basename of KNIT_JOB_PREFIX is the job’s UUID — the same id submit printed and the key used by job show, job status, and the rest of the job commands — so a job can also derive its own id from it if it needs to. KNIT_JOB_PREFIX is set only while a job runs; it is unset on the login/submit side.

Wait for and inspect a job#

Block on a job, check its lifecycle state, and show its recorded parameters.

APIs: submit, job:wait, job:status, job:show

A submitted job moves through lifecycle states — submitted → running → completed, or killed if the scheduler terminates it. Knit records the state in the jobs table keyed by the job’s UUID.

To submit and block in one step, add --wait: submit then returns the job’s own exit code, so it composes in a script like any foreground command:

$ ./exp.sh submit --wait -- julia --width 1920 && echo "render ok"

To block on an already-submitted job, use job wait. It waits on the scheduler itself (not by polling the database), prints the terminal state, and exits non-zero if the job was killed:

$ id=$(./exp.sh submit -- julia --width 1920)
$ ./exp.sh job wait --id "$id"
completed

Check the current state without blocking with job status, and see everything recorded for the job — its submission options and its own parameters — with job show (add --json for machine-readable output):

$ ./exp.sh job status --id "$id"
running
$ ./exp.sh job show --id "$id"
$ ./exp.sh job show --id "$id" --json

Retrieve a job’s output#

Print a job’s captured stdout, stderr, and generated batch script by id.

APIs: job:show:stdout, job:show:stderr, job:show:script

A job’s standard output and error are always captured to files in its working directory, and the batch script knit generated to run it is kept alongside them. Retrieve any of the three by the job’s UUID — no need to know the paths:

$ ./exp.sh job show:stdout --id "$id"
$ ./exp.sh job show:stderr --id "$id"
$ ./exp.sh job show:script --id "$id"

show:stdout and show:stderr accept --follow to stream the file as it grows, like tail -f — handy for watching a still-running job:

$ ./exp.sh job show:stdout --id "$id" --follow

show:script prints the exact scheduler script knit submitted (its directives, the setup activation, and the submit line it re-invokes on the compute node), which is the first thing to read when a job behaves unexpectedly.

Cancel or resubmit a job#

Stop a running job, or replay a past submission as a fresh job.

APIs: job:cancel, job:resubmit

Stop a running job with job cancel. Knit asks the scheduler to terminate it; the job’s compute-side handler records the killed state as it goes down:

$ ./exp.sh job cancel --id "$id"

To run a job again with exactly the same inputs, use job resubmit. Knit reads the old job’s recorded submission options (setup, nodes, walltime, …) and its job parameters from the database and reconstructs an equivalent submit invocation — so a run is reproducible from its id alone, with nothing to remember or retype:

$ new_id=$(./exp.sh job resubmit --id "$id")

Resubmitting always mints a new job UUID; the original id is never reused, so the old run’s records and outputs stay intact. This makes job resubmit the building block for re-running a past experiment exactly as it was first run.

Prepare a job instead of submitting it#

Build and record a job without dispatching it, leaving it queued as a prepared row.

APIs: prepare, job:cancel

submit does two things at once: it builds a job (validate it, create its directory, generate the batch script, record the row) and it dispatches that job to the scheduler. prepare does only the first half. It mirrors submit argument-for-argument — over the same job registry, so you prepare the same jobs you submit — but stops before the scheduler, leaving a row in state prepared:

# A job you prepare now and release later. Nothing about it is special: `prepare`
# and `submit` share one job registry, so any job you can submit you can prepare.
@job "sim" "Run one simulation."
@without_setup
@with_optional "n:integer"      "10"   "Number of steps to run."
@with_optional "label:string"   "run"  "A label recorded with the run."
@with_table
@with_output   "result:integer" "0"    "Twice the step count (a stand-in result)."
_sim() {
    local n label
    n=$(knit_get_parameter "n" "$@")
    label=$(knit_get_parameter "label" "$@")
    knit_output "result" "$(( n * 2 ))"
    printf '%s: result=%s\n' "${label}" "$(( n * 2 ))"
}
@done

# A job you prepare now and release later. Nothing about it is special: `prepare`
# and `submit` share one job registry, so any job you can submit you can prepare.
knit_register_job "sim" _sim "Run one simulation."
knit_without_setup
knit_with_optional "n:integer"      "10"   "Number of steps to run."
knit_with_optional "label:string"   "run"  "A label recorded with the run."
knit_with_table
knit_with_output   "result:integer" "0"    "Twice the step count (a stand-in result)."
_sim() {
    local n label
    n=$(knit_get_parameter "n" "$@")
    label=$(knit_get_parameter "label" "$@")
    knit_output "result" "$(( n * 2 ))"
    printf '%s: result=%s\n' "${label}" "$(( n * 2 ))"
}
knit_done

Prepare the job the way you would submit it, but with prepare (note there is no --wait — nothing is dispatched yet). --group tags related runs with a free-form label you can filter on later:

$ ./exp.sh prepare --group sweep -- sim --n 5
018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a

The whole submission spec — nodes, walltime, queue, setup, and the job’s own arguments — is frozen at prepare time and baked into the batch script, so releasing the job never re-opens those options. A prepared job is an ordinary jobs row, so it needs no new listing command: job list --status prepared shows them (add --types to narrow by job type):

$ ./exp.sh job list --status prepared

A prepared job has contacted no scheduler, so job cancel simply removes it — its row, its directory, and any --name alias — with nothing to cancel remotely:

$ ./exp.sh job cancel --id 018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a

See Release prepared jobs for how to hand a prepared job to the scheduler, and Prepare many jobs from a plan for preparing a whole batch at once.

Run many jobs in one allocation (pilot jobs)#

Mark a job a pilot so its body can submit sub-jobs that pack onto its own allocation.

APIs: knit_with_pilot, submit, job:wait

A single knit submit buys one allocation from the machine scheduler. To run many smaller jobs inside that allocation — a parameter campaign, an ensemble, a many-task workload — mark a job a pilot with knit_with_pilot. When the pilot job starts, knit launches a small in-allocation scheduler (the knit-pilotd daemon, provisioned at bootstrap like the other bundled tools) over the job’s own nodes. From then on a knit submit in the pilot’s body goes to that daemon instead of to Slurm/PBS, so the sub-jobs pack onto the allocation and run concurrently — with no second trip through the batch queue.

The body fans out the sub-jobs, keeping the UUID each knit submit returns, and waits on the whole batch with knit job wait --id (which accepts a comma-separated list):

# @with_pilot marks this job a pilot. Inside its body, `knit submit` targets the
# in-allocation daemon (started over the job's own nodes) instead of the machine
# scheduler, so the sub-jobs pack onto the allocation and run concurrently.
@job "campaign" "Run many worker sub-jobs on one allocation."
@with_pilot
@with_optional "count:integer" "4" "Number of worker sub-jobs to fan out."
_campaign_job() {
    local count seed ids=""
    count=$(knit_get_parameter "count" "$@")

    # Each `knit submit` queues a sub-job with the pilot daemon and returns at
    # once with its UUID; --nodes 1 asks for a one-node slice. Collect the UUIDs.
    for (( seed = 1; seed <= count; seed++ )); do
        ids+="${ids:+,}$(knit submit --nodes 1 -- worker --seed "${seed}")"
    done

    # Wait for the whole batch: `knit job wait --id` takes a comma-separated list
    # and blocks until every sub-job is terminal (non-zero if any failed/killed).
    knit job wait --id "${ids}"
}
@done

# @with_pilot marks this job a pilot. Inside its body, `knit submit` targets the
# in-allocation daemon (started over the job's own nodes) instead of the machine
# scheduler, so the sub-jobs pack onto the allocation and run concurrently.
knit_register_job "campaign" _campaign_job "Run many worker sub-jobs on one allocation."
knit_with_pilot
knit_with_optional "count:integer" "4" "Number of worker sub-jobs to fan out."
_campaign_job() {
    local count seed ids=""
    count=$(knit_get_parameter "count" "$@")

    # Each `knit submit` queues a sub-job with the pilot daemon and returns at
    # once with its UUID; --nodes 1 asks for a one-node slice. Collect the UUIDs.
    for (( seed = 1; seed <= count; seed++ )); do
        ids+="${ids:+,}$(knit submit --nodes 1 -- worker --seed "${seed}")"
    done

    # Wait for the whole batch: `knit job wait --id` takes a comma-separated list
    # and blocks until every sub-job is terminal (non-zero if any failed/killed).
    knit job wait --id "${ids}"
}
knit_done

Each sub-job is an ordinary knit job — here a tiny worker — so it records its own row and a provenance call edge back to the pilot; a campaign is queryable as a normal parent/child job graph:

# A small, ordinary sub-job. Nothing marks it "for pilots only" — a pilot simply
# submits many of them. @without_setup keeps it self-contained (no --setup).
@job "worker" "A small sub-job."
@without_setup
@with_optional "seed:integer" "1" "This worker's PRNG seed."
_worker_job() {
    local seed
    seed=$(knit_get_parameter "seed" "$@")
    printf 'worker seed %s on %s\n' "${seed}" "$(hostname)"
}
@done

# A small, ordinary sub-job. Nothing marks it "for pilots only" — a pilot simply
# submits many of them. @without_setup keeps it self-contained (no --setup).
knit_register_job "worker" _worker_job "A small sub-job."
knit_without_setup
knit_with_optional "seed:integer" "1" "This worker's PRNG seed."
_worker_job() {
    local seed
    seed=$(knit_get_parameter "seed" "$@")
    printf 'worker seed %s on %s\n' "${seed}" "$(hostname)"
}
knit_done

Launch it like any job; give the pilot the nodes the sub-jobs will pack onto:

$ ./exp.sh submit --nodes 2 --wait -- campaign --count 4

Outside a pilot job, a knit submit from within a job body is rejected (it would nest batch submissions) and the error names knit_with_pilot. Two options tune a pilot: knit_with_pilot --exclude-launch-host drops the daemon’s own node from the pool (for machines where the job script shares the first compute node; a no-op otherwise), and knit_with_pilot --kill-on-exit kills any sub-jobs still outstanding when the body returns instead of the default, which drains them to completion first. Killing the pilot job itself tears the whole sub-tree down: every running sub-job is signalled and every queued one dropped, and all are recorded killed — a killed pilot never leaves a sub-job running or lost.

Release prepared jobs#

Hand prepared jobs to the scheduler one at a time, by id or oldest-first, with a drain loop.

APIs: submit:next, submit:prepared

Once jobs are prepared (see Prepare a job instead of submitting it), release them to the scheduler with two submit subcommands. submit next releases the oldest prepared job, optionally restricted by job type (--type) or group (--group); submit prepared --id releases one specific job. Both take --wait (block until the job finishes) but no scheduler options — the submission spec was frozen at prepare time:

$ ./exp.sh submit next --group sweep
018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a
$ ./exp.sh submit prepared --id 018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a

The claim is atomic — picking and marking a job is one conditional update under knit’s write lock — so several releasers may run concurrently (cron ticks, or a loop keeping N jobs in flight) without ever double-submitting the same row.

submit next returns non-zero when no prepared job matches, so a fill-the-queue loop can drain a group and stop on its own:

# Release every prepared job in the "sweep" group, oldest first.
while ./exp.sh submit next --group sweep --wait; do :; done

To drain a whole batch without writing that loop — with concurrency throttling, a release cap, and optional background execution — use submit drain (see Drain a prepared batch with submit drain).

Releasing advances the row prepared -> submitted -> running -> completed (or killed); the id never changes, so the job you prepared and the job that ran are the same recorded artifact.

Drain a prepared batch with submit drain#

Release a whole prepared batch in one command, throttled to N jobs at once.

APIs: submit:drain, knit_progress_bar

Once a batch is prepared (see Prepare a job instead of submitting it), you could release it by hand with a loop over submit next (see Release prepared jobs). submit drain wraps that loop as one command: it releases prepared jobs — optionally filtered by --type (job name) or --group — until the matching queue is empty.

How fast they go out is set by --max-inflight:

  • 1 (the default) releases one job, waits for it, then releases the next — serial;

  • N > 1 keeps at most N jobs alive in the scheduler at once, releasing the next whenever a slot frees — a queue filler that never floods the scheduler;

  • 0 releases every matching job back to back without waiting.

$ ./exp.sh submit drain --group sweep --max-inflight 4
Released 12 job(s): 12 completed, 0 failed.

--count N releases at most N jobs this run; --stop-on-failure stops releasing new jobs once one fails — a job whose body exits non-zero (state failed) or that is killed (cancel / OOM / walltime) — while in-flight jobs still finish, and drain exits non-zero when any released job failed. --dry-run lists what would be released, in order, without claiming anything, and --json-summary prints a machine-readable object to stdout:

$ ./exp.sh submit drain --group sweep --dry-run
018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a  sim  [sweep]
018f9c3a-8c3f-7d52-ae1b-2f3e4d5c6b7c  sim  [sweep]
$ ./exp.sh submit drain --group sweep --json-summary
{"released":2,"completed":2,"failed":0,"drained":true,"stopped":false,"dry_run":false}

On a terminal, add --show-progress for a live, full-width progress bar that fills as the batch drains — a framed bar split into failed, done, running, and remaining sections, with a count line beneath it:

$ ./exp.sh submit drain --group sweep --max-inflight 4 --show-progress
┌─ draining ──────────────────────────────────────────────────────────────┐
│████████████████████████████████████████████████▒▒▒▒                      │
└──────────────────────────────────────────────────────────────────────────┘
  █ failed   █ done   ▒ running      10 done  2 run  0 fail  0 left  (12)  83%

The bar is drawn to stderr only (so a piped --json-summary on stdout stays clean), appears only on a terminal, and is skipped entirely under --dry-run.

Like submit next, draining records nothing of its own — each release advances an existing jobs row — so the jobs you prepared and the jobs that ran are the same recorded rows.

Retry failed or killed jobs#

Re-queue terminal-failed jobs back to prepared so a later drain retries them.

APIs: job:reset, submit:drain

prepare + submit drain is a durable queue: the backlog lives in the database as prepared rows, so a drainer that dies leaves the un-started jobs still prepared and re-running submit drain resumes them for free (see Drain a prepared batch with submit drain). The gap is a job that was in flight when the drainer died: it is no longer prepared, so drain never picks it up again. The common cause is draining inside a pilot job killed at walltime — its running subjobs end up killed (their pre-termination trap fired) or failed. Those are terminal states, so the work is lost unless you put the job back in the queue.

job reset does that. It moves a killed / failed job back to prepared and clears the stale run (the backend id, the recorded hostnames), while keeping the job’s directory, batch script, recorded arguments, and id — so a later submit drain re-runs it from the same inputs, under the same provenance. It never touches running / submitting jobs (knit cannot know they are dead) and only redoes a completed job with --force:

$ ./exp.sh job reset --group sweep             # re-queue every killed/failed job in the group
Reset 2 job(s) to prepared.
$ ./exp.sh job reset --id 018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a
$ ./exp.sh job reset --group sweep --state failed   # only the failed ones

Each reset increments the job’s retries count, so a query can see how many attempts a job took.

The job stands in here for one killed at walltime: it fails on its first attempt and succeeds on the retry, because job reset preserves its directory (where it left a marker):

# A job that fails on its first attempt and succeeds on the retry. It records an
# ".attempted" marker in its own directory (KNIT_JOB_PREFIX), which `job reset`
# preserves, so the second run sees the marker and completes. A real job does not
# fail on purpose; this one stands in for one killed by a walltime limit.
@job "flaky" "Fail on the first attempt, succeed on the retry."
@without_setup
_flaky() {
    if [[ -f "${KNIT_JOB_PREFIX}/.attempted" ]]; then
        printf 'flaky: retry attempt, succeeding\n'
        return 0
    fi
    : > "${KNIT_JOB_PREFIX}/.attempted"
    printf 'flaky: first attempt, failing\n'
    return 1
}
@done

# A job that fails on its first attempt and succeeds on the retry. It records an
# ".attempted" marker in its own directory (KNIT_JOB_PREFIX), which `job reset`
# preserves, so the second run sees the marker and completes. A real job does not
# fail on purpose; this one stands in for one killed by a walltime limit.
knit_register_job "flaky" _flaky "Fail on the first attempt, succeed on the retry."
knit_without_setup
_flaky() {
    if [[ -f "${KNIT_JOB_PREFIX}/.attempted" ]]; then
        printf 'flaky: retry attempt, succeeding\n'
        return 0
    fi
    : > "${KNIT_JOB_PREFIX}/.attempted"
    printf 'flaky: first attempt, failing\n'
    return 1
}
knit_done

Retry inside the drain. submit drain --retry <states> folds the reset into the drain: once at the start of the pass it re-queues every matching job in those states (killed and/or failed, within the drain’s --group / --type scope), then drains the prepared backlog. This is the one command a re-submittable pilot runs:

# login node, once: prepare the whole batch
$ ./exp.sh prepare --group sweep -- sim --n 5

# inside the (re-submittable) pilot body: resume AND retry what a previous
# pilot left behind
$ ./exp.sh submit drain --group sweep --max-inflight 8 --retry killed,failed --max-retries 3

The reset is a one-shot snapshot taken at the start of the pass, so a job that fails again during the same pass is left terminal for the next invocation — one pass cannot spin in an immediate retry loop. --max-retries <n> caps the per-job retry count (via the retries column), so a poison job that always fails is not retried without bound; it is left terminal with a warning. Leave --retry off and submit drain behaves exactly as before, releasing only prepared jobs.

See Drain a prepared batch with submit drain for the drain itself and Prepare a job instead of submitting it for building the backlog.

Prepare many jobs from a plan#

Prepare a whole batch of jobs from a JSON plan, including a GitHub-Actions-style matrix.

APIs: prepare:from

To prepare a whole batch at once, describe it in a JSON plan and feed it to prepare from — from a file with --file, or on stdin when --file is omitted. Each entry is prepared as if you had run prepare -- <job> … by hand:

{
  "group": "sweep",
  "jobs": [
    { "job": "sim", "args": { "n": 5, "label": "baseline" } },

    { "matrix": {
        "job": "sim",
        "axes": {
          "args":  [ {"label": "a"}, {"label": "b"} ],
          "nodes": [ 1, 2 ]
        },
        "exclude": [ { "args": {"label": "b"}, "nodes": 2 } ],
        "include": [ { "args": {"label": "c"}, "nodes": 4 } ]
    } }
  ]
}

Plan schema. A plan is a JSON object with three top-level keys:

  • group (optional string) — a default group applied to every prepared job (a per-entry group, or a --group on the command line, overrides it).

  • defaults (optional object) — a field map merged under every entry, so an explicit field on the entry wins. Applies to concrete entries and to every matrix combination. The args object is deep-merged: an entry (or a matrix combination) overrides only the arguments it names and keeps the rest of defaults.args, so you can sweep one argument in a matrix while sharing the others through defaults.args. (An args given as a raw-token array takes full control and does not merge with defaults.args.)

  • jobs (required array) — the entries to prepare. Each element is either a concrete entry or a matrix block.

Entry fields. Within a concrete entry (and within each matrix combination):

  • job (required string) — the registered job name (the token after --).

  • args (optional) — the job’s own arguments. An object {"n": 5} becomes --n 5 (a boolean true is a bare flag, false is omitted); an array ["--n", "5"] is passed through as raw tokens verbatim.

  • extra (optional array) — raw tokens appended after args.

  • any other key — a submission option (nodes, walltime, setup, group, …), exactly as on the prepare command line. An unknown key is a fatal plan error naming the key, so a typo is never silently dropped.

Matrix expansion. A matrix block expands to one prepared job per combination: the cartesian product of its axes, minus every exclude combination, plus each include entry appended as a standalone combination. The block’s fixed fields (like job) are carried into every combination. Because a bare axis key is a submission option, an axis varies a submission field (like nodes) directly; to vary the job’s own arguments, use an args axis (see Varying job arguments below). The plan above expands to five prepared jobs: the baseline entry, three surviving product combinations (a/1, a/2, b/1 — b/2 is excluded), and the c/4 include.

Feed the plan on stdin (a here-doc, here-string, or pipe) or from a file:

$ ./exp.sh prepare from --file plan.json
$ ./exp.sh prepare from < plan.json

The whole plan is validated before any job is prepared, so a malformed plan leaves nothing half-prepared. Jobs are prepared in plan order (matrix combinations in product order, then includes), so submit next later releases them in that order — see Release prepared jobs.

Varying job arguments#

A submission field (nodes, queue, …) is a bare axis key, but a job’s own arguments live under args. The args axis takes either shape:

  • an object of sub-axes — "args": { "n": [1,2], "label": ["a","b"] } — where each key is an independent argument axis, so the matrix is their full product (times any submission axes);

  • an array of complete arg objects — "args": [ {"n":1}, {"n":8} ] — a single axis of pre-built tuples, for arguments that must vary together.

An exclude matches a submission field by value, but matches args as a subset: it need only name the sub-keys you want to pin, so it can drop combinations by a single argument. This plan takes the product of nodes, n, and label (2 x 2 x 2 = 8), then drops every label = b combination (an args subset), leaving four prepared jobs:

{
  "group": "grid",
  "jobs": [
    { "matrix": {
        "job": "sim",
        "axes": {
          "nodes": [ 1, 2 ],
          "args":  { "n": [ 1, 2 ], "label": [ "a", "b" ] }
        },
        "exclude": [ { "args": { "label": "b" } } ]
    } }
  ]
}

Recover jobs stuck at running#

Correct jobs that a hard SIGKILL left stuck at running/submitting to their true terminal state, then retry them.

APIs: job:reconcile, submit:drain

When a compute node crashes, runs out of memory, or a drainer process is hard-killed, the scheduler terminates the job with SIGKILL. The knit compute-side kill trap has no chance to run, so the job’s database row stays running indefinitely. job reset and submit drain --retry will not touch it, because both require a terminal state (killed / failed) before they re-queue a row.

job reconcile bridges the gap. It walks every running or submitting row in the database, reads the per-job .job.id file that the scheduler assigned at dispatch time, and asks the scheduler for the job’s true fate. The verdict drives a state correction:

  • running — the job is genuinely alive; the row is left unchanged.

  • completed / failed / killed — the row is updated to match.

  • unknown (no scheduler record) — the row is moved to killed (a job the scheduler cannot confirm active is not a success).

The write is conditional (WHERE state IN ('running','submitting')), so a kill trap that fires concurrently always wins a close race.

Run it by group to scope it to a single sweep, or without a selector to walk the entire database:

$ ./exp.sh job reconcile --group sweep
Reconciled 2 job(s); left 1 running.

$ ./exp.sh job reconcile --dry-run    # inspect without writing
$ ./exp.sh job reconcile --id "$id"   # one job by UUID

The job used as the example here always completes, so a stuck row is recovered as killed (the scheduler reports its PID gone with no exit record):

# An ordinary job that always completes. It only does arithmetic, so it runs
# fast on any backend and is suitable as the "victim" job whose state we
# manipulate to simulate a hard-kill scenario.
@job "sim" "Run one trivial simulation."
@without_setup
@with_optional "n:integer"      "1"  "Number of steps to run."
@with_table
@with_output   "result:integer" "0"  "Twice the step count (a stand-in result)."
_sim() {
    local n
    n=$(knit_get_parameter "n" "$@")
    knit_output "result" "$(( n * 2 ))"
    printf 'sim: n=%s result=%s\n' "${n}" "$(( n * 2 ))"
}
@done

# An ordinary job that always completes. It only does arithmetic, so it runs
# fast on any backend and is suitable as the "victim" job whose state we
# manipulate to simulate a hard-kill scenario.
knit_register_job "sim" _sim "Run one trivial simulation."
knit_without_setup
knit_with_optional "n:integer"      "1"  "Number of steps to run."
knit_with_table
knit_with_output   "result:integer" "0"  "Twice the step count (a stand-in result)."
_sim() {
    local n
    n=$(knit_get_parameter "n" "$@")
    knit_output "result" "$(( n * 2 ))"
    printf 'sim: n=%s result=%s\n' "${n}" "$(( n * 2 ))"
}
knit_done

Recover and retry in one drain pass. submit drain --reconcile folds the reconcile step into the drain loop: before releasing any prepared job it corrects every stuck row in the drain’s --group/--type scope, then --retry re-queues the newly-killed rows, and the drain releases them. This is the pattern a re-submittable pilot uses to survive a hard cluster kill:

# inside the (re-submittable) pilot body
$ ./exp.sh submit drain --group sweep --reconcile --retry killed --max-retries 3

Order inside the pass is always: reconcile → retry → drain. --reconcile is skipped under --dry-run. Combined with --retry, a single pilot submission can recover, re-queue, and complete an entire batch that was interrupted mid-flight by a hard scheduler kill.

Pilot sub-jobs. A sub-job submitted inside a pilot job runs under the in-allocation daemon, not the batch scheduler, so the scheduler cannot answer for it directly. job reconcile handles this by walking from the stuck sub-job to its parent pilot — which is an ordinary scheduler job — and asking the scheduler for the parent’s fate: if the pilot is gone, its orphaned sub-jobs died with the allocation and are marked killed; if the pilot is still running, the sub-jobs are left alone. So a job reconcile run from the login node after a hard-killed pilot correctly recovers the sub-jobs it left stuck, and running it against a live pilot never false-kills a sub-job that is still working.

See Retry failed or killed jobs for the retry mechanics and Run many jobs in one allocation (pilot jobs) for the pilot pattern.

Run a parameter sweep#

Run a command, job, or app once per point in a cartesian product of values with knit sweep.

APIs: sweep, submit:drain, knit_with_optional, knit_parameter_space, knit_command_parameters

knit sweep runs a downstream command once per point in the cartesian product of a set of values per parameter. It is the exhaustive, no-model proposer: you write the exact invocation you would run once, and mark any value you want to vary in sweep-syntax. The sweep is opaque to the downstream command — it scans the tokens after -- for sweep-syntax, treats each as an axis, passes everything else through verbatim, and re-emits the reconstructed line once per combination.

A value is a swept axis when it is {a,b,c} (a list), [lo..hi] / [lo..hi:step] (a range), @glob:<pattern> or @file:<path>; quote the braces and brackets so the shell does not expand them. Everything else passes through. The downstream command carries an ordinary job here:

# An ordinary job whose tunable parameter carries a search --domain, so the sweep
# can draw its values from the declaration (--fill domain / --n @domain) as well as
# from an explicit set on the command line (--n '{1,2,4}').
@job "sim" "Run one trivial simulation."
@without_setup
@with_optional "n:integer"      "1" "Number of steps." --domain '{1,2,4}'
@with_table
@with_output   "result:integer" "0" "Twice the step count (a stand-in result)."
_sim() {
    local n
    n=$(knit_get_parameter "n" "$@")
    knit_output "result" "$(( n * 2 ))"
    printf 'sim: n=%s result=%s\n' "${n}" "$(( n * 2 ))"
}
@done

# An ordinary job whose tunable parameter carries a search --domain, so the sweep
# can draw its values from the declaration (--fill domain / --n @domain) as well as
# from an explicit set on the command line (--n '{1,2,4}').
knit_register_job "sim" _sim "Run one trivial simulation."
knit_without_setup
knit_with_optional "n:integer"      "1" "Number of steps." --domain '{1,2,4}'
knit_with_table
knit_with_output   "result:integer" "0" "Twice the step count (a stand-in result)."
_sim() {
    local n
    n=$(knit_get_parameter "n" "$@")
    knit_output "result" "$(( n * 2 ))"
    printf 'sim: n=%s result=%s\n' "${n}" "$(( n * 2 ))"
}
knit_done

Ephemeral: run the product now. Write the real invocation after -- and mark the swept values. The dispatcher you use (submit, run, or a plain command) selects what each point does; the sweep launches them in-process, keeping at most --max-inflight in flight (default 1):

# a job per point (3 submissions), grouped as a campaign:
$ ./exp.sh sweep -- submit --group mc --nodes {1,2} -- sim --n '{1,2,4}'
# an app per point, inside one allocation:
$ ./exp.sh sweep -- run --procs {1,2,4} -- stencil --tile '{64,128}'
# a plain command per point:
$ ./exp.sh sweep -- sim --n '{1,2,4}'

A --group you put on the submit line tags the jobs the sweep produces, so the campaign’s jobs share a jobs.group you can later drain, inspect, or remove as a unit. The sweep does not forward its own --group onto that line: a --group on sweep labels only the sweeps (parent) row, and the two are kept independent so you can gather jobs from several submits — by hand or under one or more sweeps — under one job-level group. Independently of any group, every point records an automatic call edge back to the sweep’s row, so a query can always walk a campaign to its children. --on-failure stop aborts on the first failed child (continue, the default, runs the whole product). --dry-run prints each reconstructed command line and exits without recording or running — the quickest way to see exactly what a sweep would do.

Vary a declared domain. When a parameter declares a --domain (see Declare a parameter’s search domain), the sweep can draw its values from the declaration instead of the command line: --param @domain opts one parameter in, --fill domain opts in every unspecified parameter with an enumerable domain, and --param @default opts one back out. A real-interval range is bounds-only and cannot be enumerated, so sweeping one is a hard error with a fix message. This is the one point where the sweep introspects the target, through the same accessors a Bayesian-optimization proposer uses (knit_parameter_space / knit_command_parameters):

$ ./exp.sh sweep --fill domain -- submit -- sim          # sim --n over its {1,2,4} domain
$ ./exp.sh sweep -- submit -- sim --n @domain            # same, opting --n in explicitly

Durable: prepare now, drain later (resumable). Swap -- submit for -- prepare and the sweep writes the whole product as a durable to-do list — one prepared job per point, no scheduler contact — then a separate submit drain releases them. This is the form to use for a large or long campaign, or whenever you want to resume after an interruption: a drainer that dies leaves the un-started points still prepared, so re-running drain resumes exactly the remainder (see Drain a prepared batch with submit drain and Retry failed or killed jobs):

# 1. prepare the product once (durable, grouped -- --group on prepare labels
#    the prepared jobs so the drain can select them):
$ ./exp.sh sweep -- prepare --group mc -- sim --n '{1,2,4}'
# 2. drain it -- from the login node, or inside a pilot job:
$ ./exp.sh submit drain --group mc --max-inflight 8 --reconcile --retry killed,failed

Resumability is not a sweep feature at all — it is a property of the prepare/drain queue the sweep writes into. Draining inside a knit_with_pilot job packs the swept jobs onto the pilot’s nodes with no second queue wait; a pilot killed at walltime is resumed by re-submitting it, where --reconcile first corrects the sub-jobs the hard kill stranded at running and --retry re-queues them. The sweep’s own --max-inflight / --on-failure govern only the ephemeral forms; durable concurrency, recovery, and failure policy are drain’s.