Apps

Launching MPI applications across a job’s allocation with knit run.

Register an MPI app

Declare an app with knit_register_app; its body runs once per MPI rank, and knit records outputs from rank 0 only.

APIs: knit_register_app, knit_output

An app is the unit knit run launches across an MPI world: knit starts one copy of its body per rank. Register it with knit_register_app — it takes parameters, types, and outputs exactly like a job or a plain command:

Shorthand

@app "render" "Render one MPI-parallel Julia-set image."
@with_optional "width:integer"    "800"    "Image width in pixels."
@with_optional "height:integer"   "600"    "Image height in pixels."
@with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
@with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
@with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
@with_required "output:string"             "Absolute PNG path (the job supplies one per run)."
@with_output   "inside:integer"   "0"      "Grid points inside the set (recorded by rank 0)."
_render_app() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")
    output=$(knit_get_parameter "output" "$@")

    # This body runs on EVERY rank: knit run launched one copy per rank, and
    # julia-fractal --- called as a child, not exec'd --- inherits the launcher's
    # MPI environment, so the copies form one MPI world and split the image. Only
    # rank 0 writes the PNG and prints the inside= line.
    local out
    out=$(julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
        "${output}" "${colormap}")
    printf '%s\n' "${out}"

    # Record the metric. Knit records outputs only from rank 0 (it suppresses
    # recording on the other ranks), so this single knit_output writes one row no
    # matter how many ranks ran. KNIT_MPI_RANK / KNIT_MPI_SIZE / KNIT_MPI_LOCAL_RANK
    # are available too if a body needs to branch on its own rank.
    local inside
    inside=$(sed -n 's/.*inside=\([0-9]*\).*/\1/p' <<< "${out}" | head -1)
    knit_output "inside" "${inside}"
}
@done

Long form

knit_register_app "render" _render_app "Render one MPI-parallel Julia-set image."
knit_with_optional "width:integer"    "800"    "Image width in pixels."
knit_with_optional "height:integer"   "600"    "Image height in pixels."
knit_with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
knit_with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
knit_with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
knit_with_required "output:string"             "Absolute PNG path (the job supplies one per run)."
knit_with_output   "inside:integer"   "0"      "Grid points inside the set (recorded by rank 0)."
_render_app() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")
    output=$(knit_get_parameter "output" "$@")

    # This body runs on EVERY rank: knit run launched one copy per rank, and
    # julia-fractal --- called as a child, not exec'd --- inherits the launcher's
    # MPI environment, so the copies form one MPI world and split the image. Only
    # rank 0 writes the PNG and prints the inside= line.
    local out
    out=$(julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
        "${output}" "${colormap}")
    printf '%s\n' "${out}"

    # Record the metric. Knit records outputs only from rank 0 (it suppresses
    # recording on the other ranks), so this single knit_output writes one row no
    # matter how many ranks ran. KNIT_MPI_RANK / KNIT_MPI_SIZE / KNIT_MPI_LOCAL_RANK
    # are available too if a body needs to branch on its own rank.
    local inside
    inside=$(sed -n 's/.*inside=\([0-9]*\).*/\1/p' <<< "${out}" | head -1)
    knit_output "inside" "${inside}"
}
knit_done

The body above runs on every rank. It calls julia-fractal as a child process (not exec), so that child inherits the launcher’s MPI environment and the copies join one MPI_COMM_WORLD and split the work. A body that needs to branch on its own rank reads the environment knit normalizes for it — see Read a rank’s place in the MPI world.

Recording is automatic but rank-0 only: knit suppresses row and output recording on every rank but rank 0, so a single knit_output (see the inside metric above) writes one row no matter how many ranks ran. You do not need to guard it with a rank check.

An app is not submitted directly — a job launches it with knit run (see Launch an app from a job).

Read a rank’s place in the MPI world

An app body branches on its own rank via the normalized KNIT_MPI_RANK, KNIT_MPI_SIZE, and KNIT_MPI_LOCAL_RANK.

APIs: KNIT_MPI_RANK, KNIT_MPI_SIZE, KNIT_MPI_LOCAL_RANK

An app body runs once per rank. When it needs to know which rank it is — to split work, or to let one rank do something the others don’t — it reads the three environment variables knit exports into every rank:

• KNIT_MPI_RANK — this rank’s index in MPI_COMM_WORLD (0-based).

• KNIT_MPI_SIZE — the total number of ranks in MPI_COMM_WORLD.

• KNIT_MPI_LOCAL_RANK — this rank’s index among the ranks on its own node.

if [[ "${KNIT_MPI_RANK}" == "0" ]]; then
    printf 'world has %s ranks\n' "${KNIT_MPI_SIZE}"
fi

Knit fills these from whichever launcher actually ran — Open MPI (OMPI_COMM_WORLD_*), MPICH/Hydra (PMI_*), Slurm (SLURM_PROCID / SLURM_NTASKS / SLURM_LOCALID), PALS (PALS_*), and so on. Reading the normalized KNIT_MPI_* names means the same app body works unchanged across backends, instead of hard-coding one launcher’s variables.

On a laptop (the none launcher, a single process) KNIT_MPI_RANK is 0, KNIT_MPI_SIZE is 1, and KNIT_MPI_LOCAL_RANK is 0, so a body written against these variables still runs correctly with no launcher at all.

You rarely need to guard knit_output with a rank check: knit already records only from rank 0 (see Register an MPI app). These variables are for the app’s own logic — deciding which slice of the problem each rank computes.

Launch an app from a job

Call knit run –procs N – <app> from a job body to launch an app across the job’s allocation.

APIs: run, knit_register_app, knit_job_nodecount

Apps are launched from inside a job body with knit run. The job holds the scheduler allocation; knit run places the app’s ranks across it:

Shorthand

@job "julia" "Render a Julia-set fractal as a submitted job."
@with_setup "juliaenv"
@with_optional "width:integer"    "800"    "Image width in pixels."
@with_optional "height:integer"   "600"    "Image height in pixels."
@with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
@with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
@with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    local png="${KNIT_JOB_PREFIX%/}/fractal.png"

    # Report where the scheduler placed this job.
    printf 'The job is running on hosts: %s\n' \
        "$(knit_job_hostnames --separator ', ')"

    # Was (Step 4): julia-fractal "${width}" ... "${png}" "${colormap}"
    # Now: launch the render app instead of running the binary here, with one rank
    # per allocated node. knit_job_nodecount is 1 on a laptop, so this runs
    # anywhere; on a cluster it scales with the nodes the job was given (submit
    # --nodes N). The values we used to pass positionally become named parameters.
    knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render \
        --width "${width}" --height "${height}" \
        --c-re "${c_re}" --c-im "${c_im}" --max-iter "${max_iter}" \
        --colormap "${colormap}" --output "${png}"
}
@done

Long form

knit_register_job "julia" _julia "Render a Julia-set fractal as a submitted job."
knit_with_setup "juliaenv"
knit_with_optional "width:integer"    "800"    "Image width in pixels."
knit_with_optional "height:integer"   "600"    "Image height in pixels."
knit_with_optional "c-re:real"        "-0.8"   "Real part of the Julia constant c."
knit_with_optional "c-im:real"        "0.156"  "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000"   "Maximum iterations per pixel."
knit_with_optional "colormap:string"  "fire"   "Palette: grayscale | fire | ocean."
_julia() {
    local width height c_re c_im max_iter colormap output
    width=$(knit_get_parameter "width" "$@")
    height=$(knit_get_parameter "height" "$@")
    c_re=$(knit_get_parameter "c-re" "$@")
    c_im=$(knit_get_parameter "c-im" "$@")
    max_iter=$(knit_get_parameter "max-iter" "$@")
    colormap=$(knit_get_parameter "colormap" "$@")

    local png="${KNIT_JOB_PREFIX%/}/fractal.png"

    # Report where the scheduler placed this job.
    printf 'The job is running on hosts: %s\n' \
        "$(knit_job_hostnames --separator ', ')"

    # Was (Step 4): julia-fractal "${width}" ... "${png}" "${colormap}"
    # Now: launch the render app instead of running the binary here, with one rank
    # per allocated node. knit_job_nodecount is 1 on a laptop, so this runs
    # anywhere; on a cluster it scales with the nodes the job was given (submit
    # --nodes N). The values we used to pass positionally become named parameters.
    knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render \
        --width "${width}" --height "${height}" \
        --c-re "${c_re}" --c-im "${c_im}" --max-iter "${max_iter}" \
        --colormap "${colormap}" --output "${png}"
}
knit_done

The launch line is the last statement of the job:

knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render \
    --width "${width}" --output "${png}"

Everything before the -- configures the launch (how many ranks, where); everything after names the app and its own parameters. Deriving --procs from knit_job_nodecount makes the same body scale: it is 1 on a laptop (one local rank, no MPI needed) and grows to the node count on a cluster, set by submit --nodes N.

knit run records the launch as a row in the runs table — the resolved placement and the app name travel with it — and it must run inside a job (it reads the surrounding job’s allocation). See Control process placement to pin ranks precisely and Pick a launcher backend to choose the launcher.

Control process placement

Shape how knit run places ranks with –procs, –procs-per-node, –hostnames, and per-rank CPU/GPU binding options.

APIs: run

knit run takes a set of placement options, all before the --. Give as few or as many as you like: knit fills the rest from the job’s allocation and the machine’s per-node core count.

$ knit run --procs 32 --procs-per-node 8 -- render --output "${png}"

How many ranks and where:

• --procs — total number of ranks. If omitted, knit uses one rank per allocated core (per-node core count × nodes), or one rank per node when the core count is unknown.

• --procs-per-node — ranks per node. If omitted, the launcher’s default distribution applies; --procs and --procs-per-node must be consistent with the node count (--procs divisible by --procs-per-node).

• --hostnames — a comma-separated subset of the job’s allocated hosts to place on. Each must be one of the job’s hosts (see knit_job_hostnames); useful to run on part of a larger allocation.

Per-rank CPU and GPU binding (all optional, best-effort — each is translated to the chosen launcher’s native flag, and a backend with no equivalent warns and skips rather than failing):

• --cpus-per-proc — hardware threads reserved per rank.

• --bind — CPU binding policy. Knit normalizes the vocabulary none | core | socket | numa | thread to each launcher’s spelling; any other value is passed through verbatim.

• --gpus-per-proc — GPUs reserved per rank.

• --gpu-bind — GPU binding policy, passed through to the launcher.

The resolved placement is recorded with the run, so a run’s records show exactly how its ranks were laid out. For anything a flag doesn’t cover, pass launcher arguments verbatim (see Pick a launcher backend).

Pick a launcher backend

Override the launcher knit run uses with –launcher, and pass launcher-native flags with –launcher-args.

APIs: run, bootstrap

knit run selects a launcher automatically — the one detected at bootstrap or set with bootstrap --launcher (see Select the MPI launcher). Override it for a single launch with --launcher:

$ knit run --launcher openmpi --procs 16 -- render --output "${png}"

The supported backends are:

• none — no launcher; runs the app as a single rank-0 process (rejects --procs > 1 and remote hosts). This is the laptop default.

• openmpi / mpich — MPI-native launchers (mpirun / mpiexec).

• slurm / pbs / pals / flux — the scheduler’s own launcher (srun, PBS mpiexec, PALS mpiexec, flux run), which reads the allocation natively.

Knit translates --procs, --procs-per-node, --hostnames, and the binding options into each backend’s native flags. When a flag isn’t covered, or you need a launcher option knit doesn’t model, pass it through verbatim with --launcher-args:

$ knit run --launcher openmpi --procs 16 \
    --launcher-args "--map-by socket --report-bindings" -- render --output "${png}"

--launcher-args is the escape hatch: its contents are appended to the launcher command line unchanged, after the flags knit generates.