Jobs

Jobs#

Submitting jobs to the scheduler and tracking their lifecycle.

Register and submit a job#

Register a command as a job and submit it to the scheduler with knit submit.

APIs: knit_register_job, submit, knit_with_setup

Register a job with knit_register_job instead of knit_register. A job is a command that runs asynchronously — knit hands it to the batch scheduler (Slurm, PBS, …) rather than running it in the foreground — and, on a cluster, on other machines. The declaration block is otherwise identical to a plain command: bind it to a software environment with knit_with_setup, declare its parameters, and define the body:

Shorthand

@job "julia" "Render a Julia-set fractal as a submitted job."
@with_setup "juliaenv"
@with_optional "width:integer"    "800"    "Image width in pixels."
@with_optional "height:integer"   "600"    "Image height in pixels."
@with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
@with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
@with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    # A submitted job already runs with its working directory set to its own job
    # directory (exported as KNIT_JOB_PREFIX), so a bare relative output PNG
    # would land there too. We build an absolute path anyway: it is explicit
    # about where the image belongs and stays correct even if the body (or a
    # program it launches) changes directory first.
    local png="${KNIT_JOB_PREFIX%/}/fractal.png"
    julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
        "${png}" "${colormap}"
}
@done

Long form

knit_register_job "julia" _julia "Render a Julia-set fractal as a submitted job."
knit_with_setup "juliaenv"
knit_with_optional "width:integer"    "800"    "Image width in pixels."
knit_with_optional "height:integer"   "600"    "Image height in pixels."
knit_with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
knit_with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
knit_with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    # A submitted job already runs with its working directory set to its own job
    # directory (exported as KNIT_JOB_PREFIX), so a bare relative output PNG
    # would land there too. We build an absolute path anyway: it is explicit
    # about where the image belongs and stays correct even if the body (or a
    # program it launches) changes directory first.
    local png="${KNIT_JOB_PREFIX%/}/fractal.png"
    julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
        "${png}" "${colormap}"
}
knit_done

Jobs are subcommands of the builtin submit dispatcher, so you launch one by naming it after --. Scheduler options (nodes, walltime, queue, …) go before the --; the job’s own parameters go after it:

$ ./exp.sh submit --nodes 2 --walltime 01:00:00 -- julia --width 1920 --height 1080
018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a

submit prints the new job’s UUID on stdout and returns immediately (add --wait to block; see Wait for and inspect a job). Capture the UUID to track the job later:

$ id=$(./exp.sh submit --nodes 2 -- julia --width 1920)
$ ./exp.sh job status --id "$id"

If a job declares knit_with_setup, name the setup instance to run in with submit --setup <name>. See Depend on a setup for how a job is bound to its environment, and Provide the default setup for the environment a job adopts when it declares no setup.

List the job types you can submit#

Discover registered jobs with submit –help and submit one with the – dispatcher syntax.

APIs: submit

submit is a dispatcher, not an ordinary parent command: the jobs registered with knit_register_job are not nested subcommands of it, they are the tokens you pass after --. submit --help lists them under a Jobs heading:

$ ./exp.sh submit --help
Usage: ./exp.sh submit [OPTIONS] -- <job> [OPTIONS]
...
Jobs
----
  julia   Render a Julia-set fractal as a submitted job.

The usage line shows the shape to follow. A job is submitted as submit [scheduler options] -- <jobname> [job options] — the -- separates the scheduler options (which belong to submit) from the job’s own options:

$ ./exp.sh submit --nodes 2 -- julia --width 1920

This is the one place Knit’s invocation syntax departs from ordinary subcommand nesting. Writing ./exp.sh submit julia --width 1920 (with no --) does not submit the julia job — julia would be read as a stray argument to submit itself. The -- is required.

To see a specific job’s parameters, name it before --help (no -- needed for the help form): ./exp.sh submit julia --help.

Request resources for a job#

Ask the scheduler for nodes, walltime, and GPUs with submit –nodes/–walltime/–gpus-per-node.

APIs: submit

The resources a job needs are submit options, so they go before the --:

$ ./exp.sh submit --nodes 4 --walltime 02:30:00 --gpus-per-node 2 -- julia

--nodes (default 1) is the number of nodes. Knit allocates whole nodes exclusively: the per-node core count comes from the machine profile (or bootstrap detection), so there is no per-job CPU-count option — a job gets all the cores on each node it is given.
--walltime is the wall-clock limit as HH:MM:SS. If you omit it, knit fills one in, in this order: the project default set with bootstrap --default-walltime, then the selected queue’s default (or maximum) from the machine profile, and finally one hour.
--gpus-per-node (default 0) requests GPUs per node on a GPU partition.

These values become the scheduler directives in the generated batch script (which you can inspect with job show:script) and are recorded in the jobs table, so a run’s resource request travels with its records.

Name a job and give it a stable alias#

Distinguish submit –name (a knit alias for the job) from –job-name (the scheduler’s name).

APIs: submit

submit has two different “name” options, for two different audiences.

--name is a stable alias inside knit for this job instance. Knit symlinks <job-root>/<name> to the job’s <uuid> directory and records the name in the jobs table, so you can refer to a run by a memorable name instead of a UUID. It must be unique within the experiment:

$ ./exp.sh submit --name hi-res-run -- julia --width 3840

--job-name is the name the scheduler shows (#SBATCH --job-name / #PBS -N) — what appears in squeue / qstat. It defaults to the experiment script name:

$ ./exp.sh submit --job-name julia-nightly -- julia

The two are independent: --name organizes runs within knit and never reaches the scheduler, while --job-name only labels the job in the queue and is not a handle knit tracks. Set either, both, or neither.

Set the scheduler account, project, and queue#

Override the accounting and queue for one submission, and set project-wide defaults at bootstrap.

APIs: submit, bootstrap

Clusters charge jobs to an account/project and run them in a named queue (or partition). submit takes all three as options; they go before the --:

$ ./exp.sh submit --account m1234 --queue debug -- julia

Each one defaults so you rarely pass it per submission:

--account defaults to the __account__ metadata.
--project defaults to the __project__ metadata.
--queue defaults to the __default_queue__ metadata, then to the machine profile’s default queue.

Set the metadata defaults once, at bootstrap, so every submit inherits them:

$ ./exp.sh bootstrap --account m1234 --project julia-sets

After that, a bare submit uses those values, and the options above are only for overriding them on a single run. The resolved values are written into the batch script’s directives and recorded with the job.

Find the job’s hosts and node count#

Read a running job’s allocated hosts and node count from its body.

APIs: knit_job_hostnames, knit_job_nodecount

Inside a job body, knit_job_hostnames and knit_job_nodecount report the placement the scheduler gave the job — which nodes it got and how many:

Shorthand

@job "julia" "Render a Julia-set fractal as a submitted job."
@with_setup "juliaenv"
@with_optional "width:integer"    "800"    "Image width in pixels."
@with_optional "height:integer"   "600"    "Image height in pixels."
@with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
@with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
@with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    local png="${KNIT_JOB_PREFIX%/}/fractal.png"

    # Report where the scheduler placed this job.
    printf 'The job is running on hosts: %s\n' \
        "$(knit_job_hostnames --separator ', ')"

    # Was (Step 4): julia-fractal "${width}" ... "${png}" "${colormap}"
    # Now: launch the render app instead of running the binary here, with one rank
    # per allocated node. knit_job_nodecount is 1 on a laptop, so this runs
    # anywhere; on a cluster it scales with the nodes the job was given (submit
    # --nodes N). The values we used to pass positionally become named parameters.
    knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render \
        --width "${width}" --height "${height}" \
        --c-re "${c_re}" --c-im "${c_im}" --max-iter "${max_iter}" \
        --colormap "${colormap}" --output "${png}"
}
@done

Long form

knit_register_job "julia" _julia "Render a Julia-set fractal as a submitted job."
knit_with_setup "juliaenv"
knit_with_optional "width:integer"    "800"    "Image width in pixels."
knit_with_optional "height:integer"   "600"    "Image height in pixels."
knit_with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
knit_with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
knit_with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    local png="${KNIT_JOB_PREFIX%/}/fractal.png"

    # Report where the scheduler placed this job.
    printf 'The job is running on hosts: %s\n' \
        "$(knit_job_hostnames --separator ', ')"

    # Was (Step 4): julia-fractal "${width}" ... "${png}" "${colormap}"
    # Now: launch the render app instead of running the binary here, with one rank
    # per allocated node. knit_job_nodecount is 1 on a laptop, so this runs
    # anywhere; on a cluster it scales with the nodes the job was given (submit
    # --nodes N). The values we used to pass positionally become named parameters.
    knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render \
        --width "${width}" --height "${height}" \
        --c-re "${c_re}" --c-im "${c_im}" --max-iter "${max_iter}" \
        --colormap "${colormap}" --output "${png}"
}
knit_done

knit_job_hostnames prints the allocated hosts, one per line by default. Shape the output with:

--separator ', ' — join the hosts with a custom separator instead of newlines.
--json — emit a JSON array (handy for passing hosts to another tool).
--raw — keep the scheduler’s verbatim host list, including any repeats (e.g. one line per slot); the default deduplicates and keeps first-seen order.
--select <start>:<length> — take a slice of the (0-based) host list, e.g. --select 0:1 for just the first host.

knit_job_nodecount prints the number of distinct nodes (the deduplicated host count) — so a host offering several slots counts once. It pairs naturally with knit run --procs-per-node 1 to place exactly one rank per node.

Both work off a scheduler too: on a laptop knit_job_hostnames reports the local host and knit_job_nodecount is 1, so the same body runs anywhere. (The knit run line above launches an MPI app across the allocation; that is covered in the Apps category.)

Write results into the job’s directory#

Use KNIT_JOB_PREFIX, the job’s own working directory, to place output files.

APIs: KNIT_JOB_PREFIX

Every job runs in its own directory under the job root (<job-root>/<uuid>), exported to the body as KNIT_JOB_PREFIX and already set as the process’s current directory. Anything the job writes there travels with the run’s records, so this is where output files belong:

Shorthand

@job "julia" "Render a Julia-set fractal as a submitted job."
@with_setup "juliaenv"
@with_optional "width:integer"    "800"    "Image width in pixels."
@with_optional "height:integer"   "600"    "Image height in pixels."
@with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
@with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
@with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    # A submitted job already runs with its working directory set to its own job
    # directory (exported as KNIT_JOB_PREFIX), so a bare relative output PNG
    # would land there too. We build an absolute path anyway: it is explicit
    # about where the image belongs and stays correct even if the body (or a
    # program it launches) changes directory first.
    local png="${KNIT_JOB_PREFIX%/}/fractal.png"
    julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
        "${png}" "${colormap}"
}
@done

Long form

knit_register_job "julia" _julia "Render a Julia-set fractal as a submitted job."
knit_with_setup "juliaenv"
knit_with_optional "width:integer"    "800"    "Image width in pixels."
knit_with_optional "height:integer"   "600"    "Image height in pixels."
knit_with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
knit_with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
knit_with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    # A submitted job already runs with its working directory set to its own job
    # directory (exported as KNIT_JOB_PREFIX), so a bare relative output PNG
    # would land there too. We build an absolute path anyway: it is explicit
    # about where the image belongs and stays correct even if the body (or a
    # program it launches) changes directory first.
    local png="${KNIT_JOB_PREFIX%/}/fractal.png"
    julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
        "${png}" "${colormap}"
}
knit_done

Because KNIT_JOB_PREFIX is already the working directory, a bare relative path (fractal.png) lands there too. Building an absolute path from KNIT_JOB_PREFIX — as the body above does — is still worth it: it is explicit about where the file belongs and stays correct even if the body, or a program it launches, changes directory first.

The basename of KNIT_JOB_PREFIX is the job’s UUID — the same id submit printed and the key used by job show, job status, and the rest of the job commands — so a job can also derive its own id from it if it needs to. KNIT_JOB_PREFIX is set only while a job runs; it is unset on the login/submit side.

Wait for and inspect a job#

Block on a job, check its lifecycle state, and show its recorded parameters.

APIs: submit, job:wait, job:status, job:show

A submitted job moves through lifecycle states — submitted → running → completed, or killed if the scheduler terminates it. Knit records the state in the jobs table keyed by the job’s UUID.

To submit and block in one step, add --wait: submit then returns the job’s own exit code, so it composes in a script like any foreground command:

$ ./exp.sh submit --wait -- julia --width 1920 && echo "render ok"

To block on an already-submitted job, use job wait. It waits on the scheduler itself (not by polling the database), prints the terminal state, and exits non-zero if the job was killed:

$ id=$(./exp.sh submit -- julia --width 1920)
$ ./exp.sh job wait --id "$id"
completed

Check the current state without blocking with job status, and see everything recorded for the job — its submission options and its own parameters — with job show (add --json for machine-readable output):

$ ./exp.sh job status --id "$id"
running
$ ./exp.sh job show --id "$id"
$ ./exp.sh job show --id "$id" --json

Retrieve a job’s output#

Print a job’s captured stdout, stderr, and generated batch script by id.

APIs: job:show:stdout, job:show:stderr, job:show:script

A job’s standard output and error are always captured to files in its working directory, and the batch script knit generated to run it is kept alongside them. Retrieve any of the three by the job’s UUID — no need to know the paths:

$ ./exp.sh job show:stdout --id "$id"
$ ./exp.sh job show:stderr --id "$id"
$ ./exp.sh job show:script --id "$id"

show:stdout and show:stderr accept --follow to stream the file as it grows, like tail -f — handy for watching a still-running job:

$ ./exp.sh job show:stdout --id "$id" --follow

show:script prints the exact scheduler script knit submitted (its directives, the setup activation, and the submit line it re-invokes on the compute node), which is the first thing to read when a job behaves unexpectedly.

Cancel or resubmit a job#

Stop a running job, or replay a past submission as a fresh job.

APIs: job:cancel, job:resubmit

Stop a running job with job cancel. Knit asks the scheduler to terminate it; the job’s compute-side handler records the killed state as it goes down:

$ ./exp.sh job cancel --id "$id"

To run a job again with exactly the same inputs, use job resubmit. Knit reads the old job’s recorded submission options (setup, nodes, walltime, …) and its job parameters from the database and reconstructs an equivalent submit invocation — so a run is reproducible from its id alone, with nothing to remember or retype:

$ new_id=$(./exp.sh job resubmit --id "$id")

Resubmitting always mints a new job UUID; the original id is never reused, so the old run’s records and outputs stay intact. This makes job resubmit the building block for re-running a past experiment exactly as it was first run.

Prepare a job instead of submitting it#

Build and record a job without dispatching it, leaving it queued as a prepared row.

APIs: prepare, job:cancel

submit does two things at once: it builds a job (validate it, create its directory, generate the batch script, record the row) and it dispatches that job to the scheduler. prepare does only the first half. It mirrors submit argument-for-argument — over the same job registry, so you prepare the same jobs you submit — but stops before the scheduler, leaving a row in state prepared:

Shorthand

# A job you prepare now and release later. Nothing about it is special: `prepare`
# and `submit` share one job registry, so any job you can submit you can prepare.
@job "sim" "Run one simulation."
@without_setup
@with_optional "n:integer"      "10"   "Number of steps to run."
@with_optional "label:string"   "run"  "A label recorded with the run."
@with_table
@with_output   "result:integer" "0"    "Twice the step count (a stand-in result)."
_sim() {
    local n label
    n=$(knit_get_parameter "n" "$@")
    label=$(knit_get_parameter "label" "$@")
    knit_output "result" "$(( n * 2 ))"
    printf '%s: result=%s\n' "${label}" "$(( n * 2 ))"
}
@done

Long form

# A job you prepare now and release later. Nothing about it is special: `prepare`
# and `submit` share one job registry, so any job you can submit you can prepare.
knit_register_job "sim" _sim "Run one simulation."
knit_without_setup
knit_with_optional "n:integer"      "10"   "Number of steps to run."
knit_with_optional "label:string"   "run"  "A label recorded with the run."
knit_with_table
knit_with_output   "result:integer" "0"    "Twice the step count (a stand-in result)."
_sim() {
    local n label
    n=$(knit_get_parameter "n" "$@")
    label=$(knit_get_parameter "label" "$@")
    knit_output "result" "$(( n * 2 ))"
    printf '%s: result=%s\n' "${label}" "$(( n * 2 ))"
}
knit_done

Prepare the job the way you would submit it, but with prepare (note there is no --wait — nothing is dispatched yet). --group tags related runs with a free-form label you can filter on later:

$ ./exp.sh prepare --group sweep -- sim --n 5
018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a

The whole submission spec — nodes, walltime, queue, setup, and the job’s own arguments — is frozen at prepare time and baked into the batch script, so releasing the job never re-opens those options. A prepared job is an ordinary jobs row, so it needs no new listing command: job list --status prepared shows them (add --types to narrow by job type):

$ ./exp.sh job list --status prepared

A prepared job has contacted no scheduler, so job cancel simply removes it — its row, its directory, and any --name alias — with nothing to cancel remotely:

$ ./exp.sh job cancel --id 018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a

See Release prepared jobs for how to hand a prepared job to the scheduler, and Prepare many jobs from a plan for preparing a whole batch at once.

Release prepared jobs#

Hand prepared jobs to the scheduler one at a time, by id or oldest-first, with a drain loop.

APIs: submit:next, submit:prepared

Once jobs are prepared (see Prepare a job instead of submitting it), release them to the scheduler with two submit subcommands. submit next releases the oldest prepared job, optionally restricted by job type (--type) or group (--group); submit prepared --id releases one specific job. Both take --wait (block until the job finishes) but no scheduler options — the submission spec was frozen at prepare time:

$ ./exp.sh submit next --group sweep
018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a
$ ./exp.sh submit prepared --id 018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a

The claim is atomic — picking and marking a job is one conditional update under knit’s write lock — so several releasers may run concurrently (cron ticks, or a loop keeping N jobs in flight) without ever double-submitting the same row.

submit next returns non-zero when no prepared job matches, so a fill-the-queue loop can drain a group and stop on its own:

# Release every prepared job in the "sweep" group, oldest first.
while ./exp.sh submit next --group sweep --wait; do :; done

Releasing advances the row prepared -> submitted -> running -> completed (or killed); the id never changes, so the job you prepared and the job that ran are the same recorded artifact.

Prepare many jobs from a plan#

Prepare a whole batch of jobs from a JSON plan, including a GitHub-Actions-style matrix.

APIs: prepare:from

To prepare a whole batch at once, describe it in a JSON plan and feed it to prepare from — from a file with --file, or on stdin when --file is omitted. Each entry is prepared as if you had run prepare -- <job> … by hand:

{
  "group": "sweep",
  "jobs": [
    { "job": "sim", "args": { "n": 5, "label": "baseline" } },

    { "matrix": {
        "job": "sim",
        "axes": {
          "args":  [ {"label": "a"}, {"label": "b"} ],
          "nodes": [ 1, 2 ]
        },
        "exclude": [ { "args": {"label": "b"}, "nodes": 2 } ],
        "include": [ { "args": {"label": "c"}, "nodes": 4 } ]
    } }
  ]
}

Plan schema. A plan is a JSON object with three top-level keys:

group (optional string) — a default group applied to every prepared job (a per-entry group, or a --group on the command line, overrides it).
defaults (optional object) — a field map merged under every entry, so an explicit field on the entry wins. Applies to concrete entries and to every matrix combination.
jobs (required array) — the entries to prepare. Each element is either a concrete entry or a matrix block.

Entry fields. Within a concrete entry (and within each matrix combination):

job (required string) — the registered job name (the token after --).
args (optional) — the job’s own arguments. An object {"n": 5} becomes --n 5 (a boolean true is a bare flag, false is omitted); an array ["--n", "5"] is passed through as raw tokens verbatim.
extra (optional array) — raw tokens appended after args.
any other key — a submission option (nodes, walltime, setup, group, …), exactly as on the prepare command line. An unknown key is a fatal plan error naming the key, so a typo is never silently dropped.

Matrix expansion. A matrix block expands to one prepared job per combination: the cartesian product of its axes, minus every exclude combination, plus each include entry appended as a standalone combination. The block’s fixed fields (like job) are carried into every combination. Because a bare axis key is a submission option, an axis varies a submission field (like nodes) directly; to vary the job’s own arguments, use an args axis whose values are arg objects. The plan above expands to five prepared jobs: the baseline entry, three surviving product combinations (a/1, a/2, b/1 — b/2 is excluded), and the c/4 include.

Feed the plan on stdin (a here-doc, here-string, or pipe) or from a file:

$ ./exp.sh prepare from --file plan.json
$ ./exp.sh prepare from < plan.json

The whole plan is validated before any job is prepared, so a malformed plan leaves nothing half-prepared. Jobs are prepared in plan order (matrix combinations in product order, then includes), so submit next later releases them in that order — see Release prepared jobs.

Jobs

Contents

Jobs#

Register and submit a job#

List the job types you can submit#

Request resources for a job#

Name a job and give it a stable alias#

Set the scheduler account, project, and queue#

Find the job’s hosts and node count#

Write results into the job’s directory#

Wait for and inspect a job#

Retrieve a job’s output#

Cancel or resubmit a job#

Prepare a job instead of submitting it#

Release prepared jobs#

Prepare many jobs from a plan#