Part I — The guided tour#
This first part builds the experiment from beginning to end, where each step introduces exactly one new concept. We build around a real program, julia-fractal, a small C++ renderer for Julia-set fractals. We start with a plain command that runs it in a serial way and, step by step, turn it into a Spack-backed, MPI-parallel, fully recorded experiment. By the end you will have used setups, jobs, apps, and the query and provenance surfaces together on a single realistic workload.
Note
This tutorial is meant to run on a laptop, with Step 9 moving it to an actual supercomputer. Your laptop does not need MPI. However it needs to be able to run Spack (i.e. have python), compile C programs (i.e., gcc, make), untar archives (tar) and download from the internet (curl). The code for each step of this tutorial is shown in full and was validated end to end during development; the console output is illustrative.
The program#
julia-fractal renders a Julia-set fractal to a PNG image. Its arguments are
positional — width height c_re c_im max_iter are required, the rest have
defaults:
julia-fractal <width> <height> <c_re> <c_im> <max_iter> \
[output.png] [colormap] [center_x] [center_y] [zoom]
On completion it prints, on standard output, the number of grid points found to
be inside the set, including a stable machine-readable line inside=<count>
that we will record and aggregate later. The program builds either as a plain
serial binary or, when MPI is available, as a parallel one — a property the
tutorial relies on when it introduces MPI. This program is representative of
your typical HPC simulation: a bulk-synchronous program that outputs files
and prints some results.
Step 1 — A command that calls the program#
The simplest thing we can do is wrap the program in a Knit command. As in the
Quickstart, the experiment (call it exp.sh) sources knit.sh, registers a
command, and ends with knit "$@". Here is the command — every argument is
optional with a sensible default, so it can be run bare:
#!/usr/bin/env bash
source knit.sh
knit_set_program_description "Render a Julia-set fractal."
@command "julia" "Render a Julia-set fractal to a PNG."
@with_optional "width:integer" "800" "Image width in pixels."
@with_optional "height:integer" "600" "Image height in pixels."
@with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
@with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
@with_optional "colormap:string" "fire" "Palette: grayscale | fire | ocean."
@with_optional "output:string" "" "PNG file to write (empty = no file)."
_julia() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
# The binary takes its arguments positionally:
# width height c_re c_im max_iter [output] [colormap]
julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
"${output}" "${colormap}"
}
@done
knit "$@"
#!/usr/bin/env bash
source knit.sh
knit_set_program_description "Render a Julia-set fractal."
knit_register "julia" _julia "Render a Julia-set fractal to a PNG."
knit_with_optional "width:integer" "800" "Image width in pixels."
knit_with_optional "height:integer" "600" "Image height in pixels."
knit_with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
knit_with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
knit_with_optional "colormap:string" "fire" "Palette: grayscale | fire | ocean."
knit_with_optional "output:string" "" "PNG file to write (empty = no file)."
_julia() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
# The binary takes its arguments positionally:
# width height c_re c_im max_iter [output] [colormap]
julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
"${output}" "${colormap}"
}
knit_done
knit "$@"
The body reads each parameter with knit_get_parameter and calls
julia-fractal with the arguments in the order the program expects.
Note
The function is named _julia, not julia. Knit binds a command to the
function defined just below its declaration, so the two names are independent
— the function name does not have to match the command name. A leading
underscore is good practice: it keeps the command (julia) and its Bash
function (_julia) distinct and marks the function as an internal helper of
the experiment. This tutorial follows that convention throughout
(_aggregate, _render_app, _juliaenv_setup, …).
Bootstrap the experiment once, then run the command:
$ ./exp.sh bootstrap
$ ./exp.sh julia
./exp.sh: line 39: julia-fractal: command not found
This failure is normal at this stage. Knit found our command and ran it, but
the command itself failed, because nothing has built or installed
julia-fractal yet. It is not on the PATH, so the shell cannot find it.
Note
You may wonder why we wrap julia-fractal in a command like we did. After all,
it looks more complicated than just calling it as-is. By wrapping it like we
did, we (1) told Knit about it, in particular what its inputs and outputs are
so they can be recorded, (2) made it available as a command so Knit has built
a nice CLI for us to invoke it, and (3) forced all parameters to be explicit
instead of positional, which prevents mistakes (e.g. swapping c-re and
c-im).
Note
You will find that ./exp.sh bootstrap may have created multiple folders in the
current directory: .knit contains Knit-maintained databases and tools;
setups contains environment instances (it should currently only contain
a default subfolder), and jobs will eventually contain a subfolder
for each job we will submit.
Step 2 — A setup that builds the program#
We could build the julia-fractal executable by hand, but this would mean documenting
extra steps for anyone who wants to reproduce our experiment. A setup is Knit’s
answer: a recorded, reproducible step that builds an environment once, to be reused by later
commands. Ours builds julia-fractal from source and installs it.
A setup is registered with @setup. This one declares the
software it needs with @with_spack_specs — cmake and libpng
(no MPI yet; that comes later) — and its body clones the source, builds it, and
installs it. This setup must come before our earlier definition of the julia
command:
@setup "juliaenv" "Build and install julia-fractal from source."
@with_spack_specs "cmake" "libpng"
@with_optional "ref:string" "v1.1.0" "git ref to build (tag, branch, or commit)."
_juliaenv_setup() {
# The Spack environment declared above is already built and activated here, so
# cmake and libpng are on PATH / LD_LIBRARY_PATH. Everything we install goes
# under KNIT_SETUP_PREFIX --- the private directory Knit created for this setup.
local ref
ref="$(knit_get_parameter "ref" "$@")"
git clone "https://github.com/knit-sh/julia-fractal-example.git" \
"${KNIT_SETUP_PREFIX}/src"
git -C "${KNIT_SETUP_PREFIX}/src" checkout "${ref}"
# Configure, build, and install into the setup prefix. No MPI is present in
# the environment, so CMake builds the serial binary. RPATH_USE_LINK_PATH
# bakes the libpng location into the binary so it also runs on a machine with
# no system libpng.
cmake -S "${KNIT_SETUP_PREFIX}/src" -B "${KNIT_SETUP_PREFIX}/build" \
-DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
cmake --build "${KNIT_SETUP_PREFIX}/build"
cmake --install "${KNIT_SETUP_PREFIX}/build"
# Put the installed binary on the PATH of every command that depends on this
# setup. knit_setup_env_prepend records a composable line, so each dependent
# command keeps its own PATH and gains this entry.
knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
@done
knit_register_setup "juliaenv" _juliaenv_setup "Build and install julia-fractal from source."
knit_with_spack_specs "cmake" "libpng"
knit_with_optional "ref:string" "v1.1.0" "git ref to build (tag, branch, or commit)."
_juliaenv_setup() {
# The Spack environment declared above is already built and activated here, so
# cmake and libpng are on PATH / LD_LIBRARY_PATH. Everything we install goes
# under KNIT_SETUP_PREFIX --- the private directory Knit created for this setup.
local ref
ref="$(knit_get_parameter "ref" "$@")"
git clone "https://github.com/knit-sh/julia-fractal-example.git" \
"${KNIT_SETUP_PREFIX}/src"
git -C "${KNIT_SETUP_PREFIX}/src" checkout "${ref}"
# Configure, build, and install into the setup prefix. No MPI is present in
# the environment, so CMake builds the serial binary. RPATH_USE_LINK_PATH
# bakes the libpng location into the binary so it also runs on a machine with
# no system libpng.
cmake -S "${KNIT_SETUP_PREFIX}/src" -B "${KNIT_SETUP_PREFIX}/build" \
-DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
cmake --build "${KNIT_SETUP_PREFIX}/build"
cmake --install "${KNIT_SETUP_PREFIX}/build"
# Put the installed binary on the PATH of every command that depends on this
# setup. knit_setup_env_prepend records a composable line, so each dependent
# command keeps its own PATH and gains this entry.
knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
knit_done
Two things make this reproducible. @with_spack_specs declares a
Spack environment: Knit will use Spack to build
cmake and libpng and activates them before the body runs, so
the body finds them regardless of what the host has installed.
And KNIT_SETUP_PREFIX is a private directory
Knit creates for a setup instance; everything the body installs there stays with
the setup. The body declares the environment it
wants dependent commands to inherit. knit_setup_env_prepend PATH
"${KNIT_SETUP_PREFIX}/bin" records a composable line, so each command that
depends on the setup adds the installed binary to its own PATH instead of
overwriting it. This is similar to how environment-modules or lmod work.
Note
Because a Spack-backed setup is now declared, the next time
exp.sh is executed, Knit will download Spack.
We can now instanciate our setup:
$ ./exp.sh setup --name myenv -- juliaenv
setup is a dispatcher: the options before -- configure the setup
instance (--name myenv names this build), and the arguments after --
select the setup type (juliaenv) and pass its own parameters. To build a
different revision, append --ref, e.g:
$ ./exp.sh setup --name myenv -- juliaenv --ref main
Once built, you should see a new folder named myenv in the setups directory. This is where the environment was installed.
Now tell the julia command to depend on the setup. One line, between its
@command and @done, does it:
@with_setup "juliaenv"
knit_with_setup "juliaenv"
@with_setup "juliaenv" adds a --setup option to the command and,
before the body runs, activates the named setup’s environment — putting the
installed julia-fractal on PATH. Run it, pointing at the setup we built:
$ ./exp.sh julia --setup myenv
Number of grid points within the set: 181
inside=181
The command now finds the binary and runs it. Notice the inside=181 line:
that is the metric we will record and aggregate later. So far we have run
julia directly, in the foreground. Real HPC work is submitted to run in the
background (and, on a cluster, on other nodes) — which is what the next step
turns julia into.
Note
The run above prints the count but writes no image, because output
defaults to the empty string (empty = no file). Pass --output to get a
PNG of the Julia set:
$ ./exp.sh julia --setup myenv --output fractal.png
which leaves a fractal.png in the current directory.
Step 3 — Submit it as a job#
A job is a command you submit rather than run yourself: Knit hands it to the
machine’s batch scheduler (Slurm, PBS, …), which runs it in the background —
on a laptop, as a local background process; on a cluster, on compute nodes
allocated for it. The experiment does not change shape; the julia command
simply becomes a job.
The change is one word: register with @job instead of
@command. That makes julia a subcommand of the builtin submit
command rather than a top-level command. The body is almost the same as before,
with one addition — it writes its image under KNIT_JOB_PREFIX (we also
remove the output parameter of the command, alway outputing fractal.png):
@job "julia" "Render a Julia-set fractal as a submitted job."
@with_setup "juliaenv"
@with_optional "width:integer" "800" "Image width in pixels."
@with_optional "height:integer" "600" "Image height in pixels."
@with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
@with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
@with_optional "colormap:string" "fire" "Palette: grayscale | fire | ocean."
_julia() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
# A submitted job already runs with its working directory set to its own job
# directory (exported as KNIT_JOB_PREFIX), so a bare relative output PNG
# would land there too. We build an absolute path anyway: it is explicit
# about where the image belongs and stays correct even if the body (or a
# program it launches) changes directory first.
local png="${KNIT_JOB_PREFIX%/}/fractal.png"
julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
"${png}" "${colormap}"
}
@done
knit_register_job "julia" _julia "Render a Julia-set fractal as a submitted job."
knit_with_setup "juliaenv"
knit_with_optional "width:integer" "800" "Image width in pixels."
knit_with_optional "height:integer" "600" "Image height in pixels."
knit_with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
knit_with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
knit_with_optional "colormap:string" "fire" "Palette: grayscale | fire | ocean."
_julia() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
# A submitted job already runs with its working directory set to its own job
# directory (exported as KNIT_JOB_PREFIX), so a bare relative output PNG
# would land there too. We build an absolute path anyway: it is explicit
# about where the image belongs and stays correct even if the body (or a
# program it launches) changes directory first.
local png="${KNIT_JOB_PREFIX%/}/fractal.png"
julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
"${png}" "${colormap}"
}
knit_done
Two things are worth calling out. KNIT_JOB_PREFIX is a private working
directory Knit creates for each submission, so every run keeps its own output
alongside the standard output and error Knit captures for the job. Knit runs the
job’s body with its current directory already set to that job directory, so a
bare relative fractal.png would land in the same place — we could have
written just that. We still build an absolute path from KNIT_JOB_PREFIX on
purpose: it says explicitly where the output belongs, and it keeps working even
if the body (or a program it launches) changes directory first. That habit pays
off in a later step, where the body launches its work as separate processes
through knit run — an explicit, absolute path stays unambiguous across that
boundary.
The @with_setup "juliaenv" line we added in Step 2 still applies, but for
a job it behaves differently: instead of adding a --setup option to
julia, it makes --setup a required option of submit (a job always
runs against a setup), and the setup’s environment is re-activated on the machine
where the job actually runs.
Submitting uses the submit dispatcher. Its grammar mirrors setup: the
options before -- configure the submission (which setup, how many nodes,
whether to wait, …), and the arguments after -- select the job and pass
its own parameters:
$ ./exp.sh submit --setup myenv --wait -- julia
018f2a1b-9c3d-7e4f-8a1b-2c3d4e5f6a7b
submit prints the job’s UUID — its stable, scheduler-independent
identifier. --wait blocks until the job finishes (without it, submit
returns immediately and the job runs in the background). --setup myenv points
at the setup instance we built in Step 2.
Because the job ran elsewhere, its output was captured rather than printed to our
terminal. Retrieve it with job show stdout, passing the UUID:
$ ./exp.sh job show stdout --id 018f2a1b-9c3d-7e4f-8a1b-2c3d4e5f6a7b
Number of grid points within the set: 181
inside=181
Knit also tracks each job’s lifecycle. job list shows every submission and
its state (submitted — running — completed), and job status
--id <uuid> reports one job’s current state:
$ ./exp.sh job list
id job state
------------------------------------ ----- ---------
018f2a1b-9c3d-7e4f-8a1b-2c3d4e5f6a7b julia completed
The same submit command works unchanged on a laptop and on a supercomputer
— only the machine (and its scheduler) differ. That portability is the subject
of the final step. For now the job still runs the binary exactly as Step 2 did:
a single, serial process. The next step brings MPI into the picture, starting
with the setup.
Step 4 — MPI in the setup#
julia-fractal can be built as an MPI program: its CMake calls
find_package(MPI) and, when an MPI is present, compiles the parallel binary;
when none is present (Steps 2 and 3) it compiles the serial one. The source is
identical either way — whether we get a parallel build is decided entirely by
what is in the environment. So making the experiment MPI-capable is a change to
the setup, not to the experiment code. Two lines change:
@setup "juliaenv" "Build and install julia-fractal with MPI."
@with_spack_specs "cmake" "libpng" "mpi"
@provides_launcher
@with_optional "ref:string" "v1.1.0" "git ref to build (tag, branch, or commit)."
_juliaenv_setup() {
# cmake, libpng, and an MPI (mpicc/mpicxx, mpirun/mpiexec) are on PATH here,
# provided by the Spack environment declared above. Everything we install goes
# under KNIT_SETUP_PREFIX.
local ref
ref="$(knit_get_parameter "ref" "$@")"
git clone "https://github.com/knit-sh/julia-fractal-example.git" \
"${KNIT_SETUP_PREFIX}/src"
git -C "${KNIT_SETUP_PREFIX}/src" checkout "${ref}"
# CMake runs find_package(MPI). Because an MPI is now in the environment, it
# finds one and builds the parallel binary; with no MPI (Step 2) it built the
# serial one. The experiment code is identical --- only the environment changed.
cmake -S "${KNIT_SETUP_PREFIX}/src" -B "${KNIT_SETUP_PREFIX}/build" \
-DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
cmake --build "${KNIT_SETUP_PREFIX}/build"
cmake --install "${KNIT_SETUP_PREFIX}/build"
knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
@done
knit_register_setup "juliaenv" _juliaenv_setup "Build and install julia-fractal with MPI."
knit_with_spack_specs "cmake" "libpng" "mpi"
knit_provides_launcher
knit_with_optional "ref:string" "v1.1.0" "git ref to build (tag, branch, or commit)."
_juliaenv_setup() {
# cmake, libpng, and an MPI (mpicc/mpicxx, mpirun/mpiexec) are on PATH here,
# provided by the Spack environment declared above. Everything we install goes
# under KNIT_SETUP_PREFIX.
local ref
ref="$(knit_get_parameter "ref" "$@")"
git clone "https://github.com/knit-sh/julia-fractal-example.git" \
"${KNIT_SETUP_PREFIX}/src"
git -C "${KNIT_SETUP_PREFIX}/src" checkout "${ref}"
# CMake runs find_package(MPI). Because an MPI is now in the environment, it
# finds one and builds the parallel binary; with no MPI (Step 2) it built the
# serial one. The experiment code is identical --- only the environment changed.
cmake -S "${KNIT_SETUP_PREFIX}/src" -B "${KNIT_SETUP_PREFIX}/build" \
-DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
cmake --build "${KNIT_SETUP_PREFIX}/build"
cmake --install "${KNIT_SETUP_PREFIX}/build"
knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
knit_done
The two highlighted lines are the whole change. The first adds mpi to the
Spack specs. mpi is a Spack virtual:
it stands for “some MPI implementation” and Spack picks a concrete provider
(OpenMPI, MPICH, …) — either building one or reusing one the site already
exposes to Spack. With an MPI in the environment, CMake now finds it and builds
the parallel binary, and the compiler wrappers (mpicc / mpicxx) and the
launcher (mpirun / mpiexec) are on PATH while the body runs.
The second change is @provides_launcher. Launching an MPI program takes
two things: the program, and a launcher to start its ranks and place them on
nodes. On a real cluster the launcher is the site’s own (srun, the batch
mpiexec), and Knit uses it. But a plain laptop has no integrated launcher —
and here the setup just built an MPI that ships one. @provides_launcher
declares exactly that: “this setup supplies a launcher where the machine has
none.” When the setup builds, Knit detects the MPI it put on PATH, freezes
that choice into the setup’s environment, and records it. Later, when a job runs
an MPI application (Step 5), Knit uses the site launcher if there is one and
otherwise falls back to the launcher this setup provided — so the same
experiment launches ranks on a cluster and on a laptop.
Note
@provides_launcher is a fallback, not an override: a launcher the
machine already integrates always wins, because a site’s launcher cooperates
with its scheduler as intended. The setup’s launcher is used only where the
machine offers none.
The setup now builds the MPI binary. We instantiate it under a new name,
mympienv, rather than reusing myenv from Step 2:
$ ./exp.sh setup --name mympienv -- juliaenv
Note
Setup instance names are unique — setup --name myenv -- juliaenv a
second time is refused because myenv already exists. Deleting a setup is
not as simple as removing its folder: jobs and commands are recorded as having
used it, so deleting the folder by hand would leave that provenance
dangling. Knit removes a setup and everything recorded against it safely with
knit remove, which Part II’s removal step covers.
In this first pass we simply give each build a fresh name (here, mympienv).
Nothing else changes yet. The julia job is exactly the one from Step 3, and
submitting it against the new setup still runs a single process — the parallel
binary launched as one rank:
$ ./exp.sh submit --setup mympienv --wait -- julia
018f2a1b-9c3d-7e4f-8a1b-2c3d4e5f6a7b
We have built the parallel program and arranged for a launcher, but we are not
launching it in parallel yet: the job body still calls the binary directly. The
next step wraps the binary as an app and has the job launch it across ranks
with knit run.
Step 5 — Launch it in parallel with knit run#
Until now the job body ran julia-fractal itself, as a single process. To run
it across MPI ranks, the job stops calling the binary directly and instead asks
Knit to launch it, with knit run. Its parameters are unchanged from Step 4;
only the body changes:
@job "julia" "Render a Julia-set fractal as a submitted job."
@with_setup "juliaenv"
@with_optional "width:integer" "800" "Image width in pixels."
@with_optional "height:integer" "600" "Image height in pixels."
@with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
@with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
@with_optional "colormap:string" "fire" "Palette: grayscale | fire | ocean."
_julia() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
local png="${KNIT_JOB_PREFIX%/}/fractal.png"
# Report where the scheduler placed this job.
printf 'The job is running on hosts: %s\n' \
"$(knit_job_hostnames --separator ', ')"
# Was (Step 4): julia-fractal "${width}" ... "${png}" "${colormap}"
# Now: launch the render app instead of running the binary here, with one rank
# per allocated node. knit_job_nodecount is 1 on a laptop, so this runs
# anywhere; on a cluster it scales with the nodes the job was given (submit
# --nodes N). The values we used to pass positionally become named parameters.
knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render \
--width "${width}" --height "${height}" \
--c-re "${c_re}" --c-im "${c_im}" --max-iter "${max_iter}" \
--colormap "${colormap}" --output "${png}"
}
@done
knit_register_job "julia" _julia "Render a Julia-set fractal as a submitted job."
knit_with_setup "juliaenv"
knit_with_optional "width:integer" "800" "Image width in pixels."
knit_with_optional "height:integer" "600" "Image height in pixels."
knit_with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
knit_with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
knit_with_optional "colormap:string" "fire" "Palette: grayscale | fire | ocean."
_julia() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
local png="${KNIT_JOB_PREFIX%/}/fractal.png"
# Report where the scheduler placed this job.
printf 'The job is running on hosts: %s\n' \
"$(knit_job_hostnames --separator ', ')"
# Was (Step 4): julia-fractal "${width}" ... "${png}" "${colormap}"
# Now: launch the render app instead of running the binary here, with one rank
# per allocated node. knit_job_nodecount is 1 on a laptop, so this runs
# anywhere; on a cluster it scales with the nodes the job was given (submit
# --nodes N). The values we used to pass positionally become named parameters.
knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render \
--width "${width}" --height "${height}" \
--c-re "${c_re}" --c-im "${c_im}" --max-iter "${max_iter}" \
--colormap "${colormap}" --output "${png}"
}
knit_done
Two Knit calls are new, both usable from any job body. knit_job_hostnames
prints the hosts the scheduler allocated to this job (here joined with commas), so
the job can log or reason about where it landed. knit_job_nodecount returns
how many distinct nodes that is. The job uses it to launch one rank per node
— knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render … —
which is one rank on a laptop and scales with the allocation on a cluster, with no
change to the script. Everything else (the parameters, reading them, the absolute
KNIT_JOB_PREFIX output path) is exactly as in Step 4; where the body used to
call julia-fractal directly, it now calls knit run … -- render, forwarding
the same values as named parameters.
knit run’s grammar mirrors submit and setup: options before --
configure the launch (--procs total ranks, --procs-per-node how many land
on each node), and the arguments after -- select what to launch (render)
and pass its parameters.
But knit run launches something new: an app. An app is the unit of
parallel work — knit run starts one copy of it per rank. Our job launches an
app called render that wraps the actual julia-fractal call. So we register
it, with @app:
@app "render" "Render one MPI-parallel Julia-set image."
@with_optional "width:integer" "800" "Image width in pixels."
@with_optional "height:integer" "600" "Image height in pixels."
@with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
@with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
@with_optional "colormap:string" "fire" "Palette: grayscale | fire | ocean."
@with_required "output:string" "Absolute PNG path (the job supplies one per run)."
@with_output "inside:integer" "0" "Grid points inside the set (recorded by rank 0)."
_render_app() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
# This body runs on EVERY rank: knit run launched one copy per rank, and
# julia-fractal --- called as a child, not exec'd --- inherits the launcher's
# MPI environment, so the copies form one MPI world and split the image. Only
# rank 0 writes the PNG and prints the inside= line.
local out
out=$(julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
"${output}" "${colormap}")
printf '%s\n' "${out}"
# Record the metric. Knit records outputs only from rank 0 (it suppresses
# recording on the other ranks), so this single knit_output writes one row no
# matter how many ranks ran. KNIT_MPI_RANK / KNIT_MPI_SIZE / KNIT_MPI_LOCAL_RANK
# are available too if a body needs to branch on its own rank.
local inside
inside=$(sed -n 's/.*inside=\([0-9]*\).*/\1/p' <<< "${out}" | head -1)
knit_output "inside" "${inside}"
}
@done
knit_register_app "render" _render_app "Render one MPI-parallel Julia-set image."
knit_with_optional "width:integer" "800" "Image width in pixels."
knit_with_optional "height:integer" "600" "Image height in pixels."
knit_with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
knit_with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
knit_with_optional "colormap:string" "fire" "Palette: grayscale | fire | ocean."
knit_with_required "output:string" "Absolute PNG path (the job supplies one per run)."
knit_with_output "inside:integer" "0" "Grid points inside the set (recorded by rank 0)."
_render_app() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
# This body runs on EVERY rank: knit run launched one copy per rank, and
# julia-fractal --- called as a child, not exec'd --- inherits the launcher's
# MPI environment, so the copies form one MPI world and split the image. Only
# rank 0 writes the PNG and prints the inside= line.
local out
out=$(julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
"${output}" "${colormap}")
printf '%s\n' "${out}"
# Record the metric. Knit records outputs only from rank 0 (it suppresses
# recording on the other ranks), so this single knit_output writes one row no
# matter how many ranks ran. KNIT_MPI_RANK / KNIT_MPI_SIZE / KNIT_MPI_LOCAL_RANK
# are available too if a body needs to branch on its own rank.
local inside
inside=$(sed -n 's/.*inside=\([0-9]*\).*/\1/p' <<< "${out}" | head -1)
knit_output "inside" "${inside}"
}
knit_done
The app’s body is the julia-fractal call that used to live in the job, plus
one addition. Three things are worth calling out.
First, the body runs on every rank: knit run started one copy per rank,
and because the app calls julia-fractal as a child (rather than exec-ing
it) each copy’s binary inherits the launcher’s MPI environment and joins one
size-N MPI world. That is what makes julia-fractal’s own work-splitting real
— rank 0 gathers the image and writes the PNG, the other ranks compute their
slice. Knit also exports KNIT_MPI_RANK, KNIT_MPI_SIZE, and
KNIT_MPI_LOCAL_RANK for an app that needs to branch on its own rank; this one
lets the binary handle that.
Second, the app records the metric. @with_output "inside:integer"
declares an output column, and knit_output "inside" … fills it from the
binary’s inside= line. Every rank runs that line, but Knit records outputs
only from rank 0 (it suppresses recording on the others), so one clean row is
written no matter how many ranks ran. That recorded inside is what Step 6
queries and aggregates.
Third, the app has no setup of its own.
an app inherits the environment of the job that launched it, so julia-fractal
is already on PATH from the job’s juliaenv setup. The job also hands the
app the output path it built from KNIT_JOB_PREFIX. Knit runs each rank in the
directory where knit run was called — the job’s own directory — so a bare
relative name would land there too; passing the absolute path is the same good
practice as in Step 3: it is explicit, and it does not depend on the job body’s
current directory or on the process boundary the launcher crosses to start each
rank.
Note
The job and the app declare many of the same parameters (width,
height, c-re, …), because the job forwards them to the app. Rather than
repeat the declarations, you can define them once as a parameter set and
import it into both commands. Part II does
exactly this.
Submitting is exactly as before — julia is still the job, so the
submit grammar is unchanged. On a laptop the job runs on one node, so
knit_job_nodecount is 1 and a single rank is launched:
$ ./exp.sh submit --setup mympienv --wait -- julia
018f2a1b-9c3d-7e4f-8a1b-2c3d4e5f6a7b
$ ./exp.sh job show stdout --id 018f2a1b-9c3d-7e4f-8a1b-2c3d4e5f6a7b
The job is running on hosts: my-laptop
Number of grid points within the set: 181
inside=181
The payoff is portability: the same julia job scales out just by asking for
more nodes. Submit across an allocation and knit_job_nodecount reports 2, so
the job launches two ranks, one per node — the launcher places them for you:
$ ./exp.sh submit --setup mympienv --nodes 2 --wait -- julia
018f3b2c-0d4e-7f5a-9b2c-3d4e5f6a7b8c
$ ./exp.sh job show stdout --id 018f3b2c-0d4e-7f5a-9b2c-3d4e5f6a7b8c
The job is running on hosts: node07, node08
Number of grid points within the set: 181
inside=181
Nothing in the experiment script changes between the two; only the node count (and
the machine) differs. The next step turns to the records these runs left behind
— querying and aggregating the inside metric across many renders.
Step 6 — Query and aggregate the results#
Every render recorded one row, from rank 0, in a table named after the app.
Those rows accumulate: submit julia a few times — here with different Julia
constants, so each produces a different image — and the database holds one
render row per run:
$ ./exp.sh submit --setup mympienv --wait -- julia --c-re -0.123 --c-im 0.745 # Douady rabbit
$ ./exp.sh submit --setup mympienv --wait -- julia --c-re -1.0 --c-im 0.0 # Basilica
$ ./exp.sh submit --setup mympienv --wait -- julia --c-re -0.391 --c-im -0.587 # Siegel disk
$ ./exp.sh submit --setup mympienv --wait -- julia --c-re 0.285 --c-im 0.535 # period-3 lobe
$ ./exp.sh submit --setup mympienv --wait -- julia --c-re -0.7 --c-im 0.0 # San Marco dragon
$ ./exp.sh submit --setup mympienv --wait -- julia --c-re -1.25 --c-im 0.0 # period-4 bulb
$ ./exp.sh submit --setup mympienv --wait -- julia --c-re -0.1 --c-im 0.651 # near-dendrite (tiny interior)
Each constant sits inside a hyperbolic component of the Mandelbrot set, so its
Julia set has a genuine black interior for inside to count — unlike a
dendrite constant (e.g. -0.4 0.6), whose picture is intricate but has zero
interior area, so it would record inside=0.
Knit gives you three ways to read those records back.
The schema, with query catalog. It lists every table Knit is recording
into and the columns of each. Our app’s outputs show up under the render
table — notice the inside column we declared, alongside the parameters:
$ ./exp.sh query catalog --ref render
table render (command: run:render)
column id (TEXT)
column output (TEXT)
column c_im (REAL)
column c_re (REAL)
column colormap (TEXT)
column height (INTEGER)
column max_iter (INTEGER)
column width (INTEGER)
column inside (INTEGER)
(Declared parameter names with hyphens become columns with underscores, so
c-re is stored as c_re.)
The raw rows, with query sql. It runs a read-only SQL statement against
that database (any statement that would write is refused), so you can look at
exactly what was recorded:
$ ./exp.sh query sql --format column --header \
--exec "SELECT c_re, c_im, inside FROM render ORDER BY inside"
c_re c_im inside
------ ------ ------
-0.8 0.156 181
-0.1 0.651 890
-1.25 0.0 34164
0.285 0.535 67424
-0.123 0.745 69456
-1.0 0.0 74800
-0.391 -0.587 78043
-0.7 0.0 119960
A packaged fan-in, with your own command. Reading the same table from a
command turns an ad-hoc query into a reusable part of the experiment. This
aggregate command totals inside across every render, no matter which
job produced it — a fan-in, the counterpart to the fan-out of submitting many
jobs:
@command "aggregate" \
"Fan-in: total the inside metric across every recorded render."
@without_provenance
_aggregate() {
# Read back what the renders recorded. Each `knit run -- render` wrote one row
# in the `render` table (rank 0 only), so one SELECT sees every image the
# experiment has produced, across every job submission.
local count total
count=$(knit query sql --exec "SELECT count(*) FROM render;")
total=$(knit query sql --exec "SELECT sum(inside) FROM render;")
printf 'Summed inside=%s over %s render(s).\n' "${total:-0}" "${count:-0}"
}
@done
knit_register "aggregate" _aggregate \
"Fan-in: total the inside metric across every recorded render."
knit_without_provenance
_aggregate() {
# Read back what the renders recorded. Each `knit run -- render` wrote one row
# in the `render` table (rank 0 only), so one SELECT sees every image the
# experiment has produced, across every job submission.
local count total
count=$(knit query sql --exec "SELECT count(*) FROM render;")
total=$(knit query sql --exec "SELECT sum(inside) FROM render;")
printf 'Summed inside=%s over %s render(s).\n' "${total:-0}" "${count:-0}"
}
knit_done
It calls knit query sql from inside the body, exactly as you would from the
shell. @without_provenance marks the command as read-only bookkeeping: it
reads results but is not itself part of the experiment’s provenance, so running it
records no row and leaves no call edge in the graph. Running it collapses
every render into one number:
$ ./exp.sh aggregate
Summed inside=444918 over 8 render(s).
The provenance graph, with query graph. query sql reads columns from
one table; query graph follows the relationships Knit records between rows
— which submission called which job, which job launched which run, which run
produced which render. It runs Cypher, matching a
path through the graph. Each node’s label is a table name, and -[:call]-> is
the “called” edge Knit draws from a command to what it invoked. So to walk from
each julia job down to the render it produced — two tables that no single
SQL query would join without knowing the provenance — and read the metric back:
$ ./exp.sh query graph --format column --header \
--exec "MATCH (job:julia)-[:call]->(:run)-[:call]->(img:render)
RETURN job.id, img.c_re, img.inside"
id c_re inside
------------------------------------ ------ ------
018f1a0b-8b2c-7d3e-9f0a-1b2c3d4e5f60 -0.8 181
018f2a1b-9c3d-7e4f-8a1b-2c3d4e5f6a7b -0.123 69456
018f3b2c-0d4e-7f5a-9b2c-3d4e5f6a7b8c -1.0 74800
018f4c3d-1e5f-7a6b-0c3d-4e5f6a7b8c9d -0.391 78043
018f5d4e-2f6a-7b7c-1d4e-5f6a7b8c9dae 0.285 67424
018f6e5f-3a7b-7c8d-2e5f-6a7b8c9daebf -0.7 119960
018f7f6a-4b8c-7d9e-3f6a-7b8c9daebfc0 -1.25 34164
018f8a7b-5c9d-7eaf-4a7b-8c9daebfc0d1 -0.1 890
Each output column is headed by the bare property name (the part after the dot),
so RETURN job.id, img.c_re prints columns id and c_re.
The path mirrors how the run actually happened: submit called the julia
job, the job’s body called knit run, and the run launched the render app.
Because that history is recorded as a graph, you can ask questions that span the
whole chain — for example, following the used_by edge back to the setup a
render’s job consumed — rather than only what lives in a single table.
Knit’s experimental model fans out from
a single bootstrap to multiple setups, each used by multiple jobs, each potentially
running one or more apps, before a fan-in command aggregates the result to, for
instance, produce figures. That entire lineage is a single path in Cypher:
starting from a setup, follow its used_by edge to every job that consumed it,
then the call edges down to each job’s run — one row per run:
$ ./exp.sh query graph --format column --header \
--exec 'MATCH (s:`setup:juliaenv`)-[:used_by]->(:submit)-[:call]->(:julia)-[:call]->(r:run) RETURN s.id, r.app, r.procs, r.hostnames'
id app procs hostnames
------------------------------------ ------ ----- -------------
018f27aa-4b1c-7d3e-8f0a-1b2c3d4e5f60 render 1 node07
018f27aa-4b1c-7d3e-8f0a-1b2c3d4e5f60 render 2 node07,node08
018f27aa-4b1c-7d3e-8f0a-1b2c3d4e5f60 render 1 node07
Because a setup: label contains a colon, it is quoted with backticks, and the
whole --exec is single-quoted so the shell leaves those backticks alone. Every
row shares the same s.id — the one setup every job consumed — and
each carries the placement of a distinct run (a few are shown here). Adapt it by changing the setup label
(setup:<name>), projecting other columns (any column of the runs table,
e.g. r.procs_per_node or r.native_cmd), or extending the path one more hop
to the app that recorded the science metric —
-[:call]->(img:render) RETURN r.hostnames, img.inside.
Step 7 — Ask your experiment in natural language#
query sql and query graph are precise, but they ask you to know the
schema and write the query. Knit can also put a language model in front of the
same records, so you can ask in plain English. Everything it does is
read-only: the model may inspect the interface and the recorded runs, but it
cannot run your commands or change anything.
First, point Knit at an OpenAI-compatible provider. Knit never stores your API key — you give it the name of the environment variable that holds the key, and it reads that variable at call time:
$ export OPENAI_API_KEY=sk-...
$ ./exp.sh bootstrap --ai-api-key-env OPENAI_API_KEY --ai-model o4-mini
This records only non-secret configuration (env-var names and defaults) in the
metadata table. --ai-base-url defaults to https://api.openai.com/v1 but
can point at any OpenAI-compatible endpoint. Re-running bootstrap updates only
the --ai-* option you type (bootstrap --ai-model gpt-4o-mini), so one
field can change without clearing the rest.
Open-ended questions, with ai ask. The AI model answers by calling Knit’s own
read-only tools — describe, --help, metadata show, read-only SQL, and
a job’s captured output — so its answers are grounded in this experiment rather
than guessed:
$ ./exp.sh ai ask --question "which Julia constant produced the most interior points?"
The render with c_re=-0.7, c_im=0.0 (the "San Marco" constant) had the most
interior points (inside=119960), ahead of c_re=-0.391 (78043) and c_re=-1.0
(74800).
Pass --verbose to stream each tool call and result to stderr and watch it
work. Because the tools are read-only, ai ask can describe and inspect the
experiment but never submit a job or write to the database.
Auditable answers, with ai query, is narrower: it turns the question
into exactly one read-only query, runs it against the experiment, and prints
the result in the output mode you choose. It picks the language that fits: SQL
for aggregation and sorting within a table, Cypher for
relationships across commands. If the query errors, Knit feeds the error back so
the model can correct it (up to --max-iterations):
$ ./exp.sh ai query --format column \
--question "Show each render's constant and inside count, most interior first"
c_re c_im inside
------ ------ ------
-0.7 0.0 119960
-0.391 -0.587 78043
-1.0 0.0 74800
-0.123 0.745 69456
0.285 0.535 67424
-1.25 0.0 34164
-0.1 0.651 890
-0.8 0.156 181
A relationship question is better answered in Cypher; --lang pins the
language when you want to be sure, and --verbose reports which one was used:
$ ./exp.sh ai query --lang cypher \
--question "which setup did the render job use?"
When you would rather review the query than trust it blindly, --query-only
prints the generated statement (and its detected language) without running it —
handy for pasting into query sql or query graph yourself:
$ ./exp.sh ai query --query-only \
--question "Show each render's constant and inside count, most interior first"
SELECT c_re, c_im, inside FROM render ORDER BY inside DESC;
Both commands need a configured provider and a reachable key; without one they
stop with a clear message pointing you back to bootstrap --ai-*.
Step 8 — Inspect and manage what you have built#
The experiment is complete: it builds its software, submits jobs, runs them in parallel, records every run, and answers questions about the results. Before we take it to a cluster, it is worth meeting the handful of builtin commands that let you look around, retrieve outputs, and tidy up. None of them change the experiment’s code — they all operate on the experiment you have already written, and every one is a Knit builtin available in any experiment.
Know the interface, with describe and --help. describe prints
the command tree — every command with its parameters, types, defaults, and
outputs — built purely from the registrations, so it runs without touching the
database or building anything. Narrow it to one command with --only:
$ ./exp.sh describe --only submit:julia
submit julia
------------
[job, user] Render a Julia-set fractal as a submitted job.
Options
-------
--help Print this help message and exit.
--width <value> [default: '800'] Image width in pixels.
--height <value> [default: '600'] Image height in pixels.
--c-re <value> [default: '-0.8'] Real part of the Julia constant c.
--c-im <value> [default: '0.156'] Imaginary part of the Julia constant c.
--max-iter <value> [default: '1000'] Maximum iterations per pixel.
--colormap <value> [default: 'fire'] Palette: grayscale | fire | ocean.
--output <value> [default: 'fractal.png'] PNG file name, written in the job directory.
Bare, describe walks the whole tree; --format yaml or --format json
give a machine-readable view (both carry the Knit version and the full typed
interface — the same structure the AI tools in Step 7 read), --format
markdown produces a table you can drop into documentation, and
--exclude-builtins hides Knit’s own commands so you see only yours. The
per-command --help shows the same options focused on a single command
(./exp.sh julia --help).
Revisit your jobs. job list shows every job you have submitted with its
lifecycle state:
$ ./exp.sh job list
id job state
------------------------------------ ----- ---------
018f2a1b-9c3d-7e4f-8a1b-2c3d4e5f6a7b julia completed
018f3b2c-0d4e-7f5a-9b2c-3d4e5f6a7b8c julia completed
018f4c3d-1e5f-7a6b-0c3d-4e5f6a7b8c9d julia completed
Filter it with --status running, --types julia, or --setup mympienv,
and add --json for scripting. job status --id <uuid> prints one job’s
state, and job wait --id <uuid> blocks until a job reaches a terminal state
(this is what submit --wait does for you).
Step 3 used job show stdout to read a job’s captured output; the same command
also retrieves its standard error and the exact batch script Knit generated and
handed to the scheduler:
$ ./exp.sh job show script --id 018f2a1b-9c3d-7e4f-8a1b-2c3d4e5f6a7b
#!/bin/bash
export KNIT_JOB_PREFIX=.../jobs/018f2a1b-9c3d-7e4f-8a1b-2c3d4e5f6a7b
export KNIT_SETUP_PREFIX=.../setups/mympienv
source .../setups/mympienv/.activate.sh
cd ...
exec .../exp.sh submit julia
On a laptop that script just activates the setup and runs the job in the
background; on a cluster the very same command shows the #SBATCH / #PBS
directives Knit filled in — which is exactly what the next step introduces.
When a job is no longer wanted, or you want to run it again, two commands close
the loop: job cancel --id <uuid> stops a running job, and
job resubmit --id <uuid> re-runs a job reusing its recorded parameters —
repeating a run without retyping its arguments.
Record notes, with metadata. Knit keeps a small key/value table for
experiment-level facts. Store your own and read one back:
$ ./exp.sh metadata store --key note --value "week-1 parameter sweep"
$ ./exp.sh metadata load --key note
week-1 parameter sweep
metadata show lists everything in the table. Your own keys sit alongside the
__…__ keys Knit records at bootstrap (the project name, the detected scheduler
and launcher, queue and walltime defaults), so this is also where you can see how
Knit configured itself. Because it is an ordinary table, it is queryable like
anything else — query sql --exec "SELECT * FROM metadata" — and ai
can read it too. Use --force with metadata store to overwrite an existing
key.
Inspect the Spack environment, with spack. The setup provisioned its
dependencies with Spack; the spack wrapper runs any Spack command inside that
same provisioned environment, so you can confirm what got built or examine a
concrete spec:
$ ./exp.sh spack find
$ ./exp.sh spack spec cmake
Arguments after spack are forwarded verbatim, so anything Spack understands
works here.
With these in hand you can inspect, retrieve, and manage everything the experiment records. The one thing left is to run it where it is meant to run — a real cluster — which is the final step.
Step 9 — Move to a real HPC machine#
Everything so far ran on a laptop. Moving to a supercomputer changes nothing in
the experiment script: the juliaenv setup, the julia job, and the
render app are all unchanged. What differs is the machine underneath —
its scheduler (PBS, Slurm, or Flux instead of a background process), its MPI
launcher, and how many nodes you can ask for. Knit captures those machine facts
once, at bootstrap, and every submit afterwards uses them. This step shows
how.
Start from a machine profile. A profile is a small, versioned JSON description of a known HPC system — its scheduler, its launcher, its default queue and per-queue limits, and its hardware (cores and GPUs per node). Knit ships profiles for several machines and serves them from its repository. List what is available:
$ ./exp.sh profile list
anl/aurora [github] ALCF Aurora — Intel Xeon Max, 208 cores + 6× Intel GPU Max per node
anl/improv [github] LCRC Improv — 2× AMD EPYC 7713, 128 cores per node (CPU-only)
anl/polaris [github] ALCF Polaris — HPE Cray EX, 32 cores + 4× NVIDIA A100 per node
Each line is a profile name, a bracketed tag for where it came from, and the
profile’s one-line description. The tag is github for the profiles Knit ships;
a site can add its own under /etc/knit/profiles, which show up as admin.
Profiles still being validated on their machine ship hidden and are left out of
this list. Profile names have two or more path segments (a profile may add a
further segment such as .../cpu to name a specific node type). Inspect one
before you commit to it — before bootstrap you pass the spec explicitly:
$ ./exp.sh profile show --profile anl/polaris
{
"description": "ALCF Polaris — HPE Cray EX, 32 cores + 4× NVIDIA A100 per node",
"scheduler": {
"type": "pbs",
"command": "qsub",
"default_queue": "prod",
"queues": {
"prod": { "min_nodes": 10, "max_nodes": 496, "min_walltime": "00:05:00", "max_walltime": "24:00:00", "default_walltime": "01:00:00" },
"debug": { "min_nodes": 1, "max_nodes": 2, "min_walltime": "00:05:00", "max_walltime": "01:00:00", "default_walltime": "01:00:00" }
}
},
"launcher": { "type": "pals", "command": "mpiexec" },
"hardware": { "cores_per_node": 32, "gpus_per_node": 4 }
}
The output is abbreviated here: a full profile also lists the modules to load
and a spack block carrying Spack configuration. Typically that is a
packages section naming vendor packages (the system MPI, for instance) as
non-buildable externals and requiring the mpi virtual to resolve to them, so
a setup’s Spack environment reuses the vendor build instead of recompiling it.
The spack block is handed to Spack verbatim, so it may hold any Spack config
section (packages, mirrors for an air-gapped mirror, concretizer,
…). A profile spec can also be a URL or a path to a local JSON file, so a site
or a collaborator can hand you one that is not in the Knit repository.
Bootstrap under the profile. On the cluster, bootstrap the experiment and point it at the profile. This freezes the profile’s facts into the experiment’s metadata and records the allocation you will charge jobs to:
$ ./exp.sh bootstrap --profile anl/polaris --account MyAllocation
The profile pre-populates the scheduler (pbs), the launcher (pals), the
default queue (prod), and the per-node core and GPU counts, so you do not have
to spell any of them out. If you omit --profile, Knit still detects the
scheduler and launcher from what is on PATH (--scheduler auto and
--launcher auto are the defaults) — a profile simply makes the choice
explicit and adds the queue limits and hardware facts autodetection cannot know.
Either way, the resolved values land in the __scheduler__, __launcher__,
and __account__ metadata keys you met in Step 8, and profile show (no
spec now) prints the profile frozen at bootstrap.
Build the setup on the machine. The setup is where the software is compiled,
so it must run once on the cluster to rebuild julia-fractal with that
machine’s compilers and MPI:
$ ./exp.sh setup --name mympienv -- juliaenv
Because the machine now advertises its own launcher (PALS), that launcher is used
in preference to the Spack-built one the setup provides — exactly the precedence
described in Step 4. The setup’s @provides_launcher remains the laptop
fallback; on a real machine it steps aside.
Submit across an allocation. The submit command is the same one you have
used all along; a cluster just gives its scheduler-facing options something real
to do:
$ ./exp.sh submit --nodes 2 --queue debug --walltime 00:30:00 --wait \
-- julia --c-re -0.8 --c-im 0.156
018f5d4e-2f6a-7b7c-1d4e-5f6a7b8c9d0e
$ ./exp.sh job show stdout --id 018f5d4e-2f6a-7b7c-1d4e-5f6a7b8c9d0e
The job is running on hosts: x3006c0s1b0n0, x3006c0s1b1n0
Number of grid points within the set: 181
inside=181
--nodes 2 allocates two whole nodes; --queue and --walltime override
the profile’s defaults for this one submission (leave them off and the profile’s
prod queue is used, and an unset --walltime falls back to the selected
queue’s default_walltime — a modest limit, not the queue’s max_walltime
cap — so switching queues never carries one queue’s ceiling onto another). When
the walltime is defaulted this way on a batch scheduler, Knit warns and suggests
passing --walltime explicitly. Knit does not enforce a queue’s node or
walltime limits — the scheduler is the sole authority on those; the profile
records them only for reference. The account you gave at bootstrap is reused
automatically, so you do not retype it — override it per-submit with
--account (the allocation charged, #PBS -A / Slurm --account) or
--project (a project tag, #PBS -P / Slurm --wckey) when you need to.
For GPU nodes, --gpus-per-node requests accelerators. Knit allocates whole
nodes exclusively and fills the scheduler directives (#PBS, #SBATCH, or
the # flux: comments Flux uses — the ones job show script reveals) from
the profile and these options.
The job body itself is untouched. It still launches one rank per node with
knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render …, so on
this two-node allocation knit_job_nodecount is 2 and PBS/PALS places one rank
on each node — the same code that launched a single rank on your laptop.
That is the whole point. The experiment you wrote on a laptop — one script, its setup, its job, its app, its queries — moves to Polaris, Aurora, or Frontier by changing only the profile you bootstrap with. No line of the experiment refers to a scheduler, a launcher, a node count, or a queue; those live in the machine’s profile and in the options you pass at submit time. Write once, reproduce anywhere.
That completes Part I. You have built a portable, self-provisioning, fully-recorded HPC experiment from an empty directory. Part II returns to this same experiment and refines it, piece by piece, with the Knit capabilities that make each part cleaner and more reproducible. To go deeper on any single piece — the recording and provenance model, setups and Spack, parallel placement, querying, or the AI tools — you can also continue to the Stitch Guide.
The complete experiment (Part I)#
Here is the whole thing in one file. Save it as exp.sh next to a copy of
knit.sh, make it executable (chmod +x exp.sh), and you have the finished
experiment this part built:
#!/bin/bash
# doc-check: source-only
#
# The complete tutorial experiment: the setup that builds the software, the
# `julia` job that submits work, the `render` app that runs one MPI image, and
# the `aggregate` fan-in that sums the recorded metric. This is the final state
# of the experiment built up over Steps 1--9.
#
# Source-only: building the setup needs a live Spack, a compiler, an MPI, and
# network access, and launching needs an MPI launcher, none of which run in plain
# CI, so check-docs only syntax-checks this file. Behavior was validated live
# during development.
source knit.sh
knit_set_program_description "Render a Julia-set fractal."
# --- Setup ------------------------------------------------------------------
# Builds and installs julia-fractal with MPI. Spack provisions the build
# dependencies; on a machine that offers no launcher of its own, the setup also
# provides one (laptop-safe fallback). The whole thing installs into
# KNIT_SETUP_PREFIX and puts the binary on PATH.
@setup "juliaenv" "Build and install julia-fractal with MPI."
@with_spack_specs "cmake" "libpng" "mpi"
@provides_launcher
@with_optional "ref:string" "v1.1.0" "git ref to build (tag, branch, or commit)."
_juliaenv_setup() {
local ref
ref="$(knit_get_parameter "ref" "$@")"
git clone "https://github.com/knit-sh/julia-fractal-example.git" \
"${KNIT_SETUP_PREFIX}/src"
git -C "${KNIT_SETUP_PREFIX}/src" checkout "${ref}"
cmake -S "${KNIT_SETUP_PREFIX}/src" -B "${KNIT_SETUP_PREFIX}/build" \
-DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
cmake --build "${KNIT_SETUP_PREFIX}/build"
cmake --install "${KNIT_SETUP_PREFIX}/build"
knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
@done
# --- Job --------------------------------------------------------------------
# Submitted to the scheduler. Its body launches the render app across one rank
# per allocated node with `knit run`, so the same job runs one rank on a laptop
# and one rank per node across a multi-node allocation.
@job "julia" "Render a Julia-set fractal as a submitted job."
@with_setup "juliaenv"
@with_optional "width:integer" "800" "Image width in pixels."
@with_optional "height:integer" "600" "Image height in pixels."
@with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
@with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
@with_optional "colormap:string" "fire" "Palette: grayscale | fire | ocean."
@with_optional "output:string" "fractal.png" "PNG file name, written in the job directory."
_julia() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
local png="${KNIT_JOB_PREFIX%/}/${output}"
printf 'The job is running on hosts: %s\n' \
"$(knit_job_hostnames --separator ', ')"
knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render \
--width "${width}" --height "${height}" \
--c-re "${c_re}" --c-im "${c_im}" --max-iter "${max_iter}" \
--colormap "${colormap}" --output "${png}"
}
@done
# --- App --------------------------------------------------------------------
# One MPI-parallel image. Launched by the job via `knit run`; it inherits the
# job's setup environment, so julia-fractal is already on PATH. Rank 0 records
# the `inside` metric into the `render` table.
@app "render" "Render one MPI-parallel Julia-set image."
@with_optional "width:integer" "800" "Image width in pixels."
@with_optional "height:integer" "600" "Image height in pixels."
@with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
@with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
@with_optional "colormap:string" "fire" "Palette: grayscale | fire | ocean."
@with_required "output:string" "Absolute PNG path (the job supplies one per run)."
@with_output "inside:integer" "0" "Grid points inside the set (recorded by rank 0)."
_render_app() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
local out
out=$(julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
"${output}" "${colormap}")
printf '%s\n' "${out}"
local inside
inside=$(sed -n 's/.*inside=\([0-9]*\).*/\1/p' <<< "${out}" | head -1)
knit_output "inside" "${inside}"
}
@done
# --- Fan-in -----------------------------------------------------------------
# Reads back what the renders recorded. `@without_provenance` marks this as
# read-only bookkeeping, so it writes no row or edge of its own. Each
# `knit run -- render` wrote one row in the `render` table, so a single SELECT
# sees every image the experiment has produced.
@command "aggregate" \
"Fan-in: total the inside metric across every recorded render."
@without_provenance
_aggregate() {
local count total
count=$(knit query sql --exec "SELECT count(*) FROM render;")
total=$(knit query sql --exec "SELECT sum(inside) FROM render;")
printf 'Summed inside=%s over %s render(s).\n' "${total:-0}" "${count:-0}"
}
@done
knit "$@"
#!/bin/bash
# doc-check: source-only
#
# The complete tutorial experiment: the setup that builds the software, the
# `julia` job that submits work, the `render` app that runs one MPI image, and
# the `aggregate` fan-in that sums the recorded metric. This is the final state
# of the experiment built up over Steps 1--9.
#
# Source-only: building the setup needs a live Spack, a compiler, an MPI, and
# network access, and launching needs an MPI launcher, none of which run in plain
# CI, so check-docs only syntax-checks this file. Behavior was validated live
# during development.
source knit.sh
knit_set_program_description "Render a Julia-set fractal."
# --- Setup ------------------------------------------------------------------
# Builds and installs julia-fractal with MPI. Spack provisions the build
# dependencies; on a machine that offers no launcher of its own, the setup also
# provides one (laptop-safe fallback). The whole thing installs into
# KNIT_SETUP_PREFIX and puts the binary on PATH.
knit_register_setup "juliaenv" _juliaenv_setup "Build and install julia-fractal with MPI."
knit_with_spack_specs "cmake" "libpng" "mpi"
knit_provides_launcher
knit_with_optional "ref:string" "v1.1.0" "git ref to build (tag, branch, or commit)."
_juliaenv_setup() {
local ref
ref="$(knit_get_parameter "ref" "$@")"
git clone "https://github.com/knit-sh/julia-fractal-example.git" \
"${KNIT_SETUP_PREFIX}/src"
git -C "${KNIT_SETUP_PREFIX}/src" checkout "${ref}"
cmake -S "${KNIT_SETUP_PREFIX}/src" -B "${KNIT_SETUP_PREFIX}/build" \
-DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
cmake --build "${KNIT_SETUP_PREFIX}/build"
cmake --install "${KNIT_SETUP_PREFIX}/build"
knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
knit_done
# --- Job --------------------------------------------------------------------
# Submitted to the scheduler. Its body launches the render app across one rank
# per allocated node with `knit run`, so the same job runs one rank on a laptop
# and one rank per node across a multi-node allocation.
knit_register_job "julia" _julia "Render a Julia-set fractal as a submitted job."
knit_with_setup "juliaenv"
knit_with_optional "width:integer" "800" "Image width in pixels."
knit_with_optional "height:integer" "600" "Image height in pixels."
knit_with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
knit_with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
knit_with_optional "colormap:string" "fire" "Palette: grayscale | fire | ocean."
knit_with_optional "output:string" "fractal.png" "PNG file name, written in the job directory."
_julia() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
local png="${KNIT_JOB_PREFIX%/}/${output}"
printf 'The job is running on hosts: %s\n' \
"$(knit_job_hostnames --separator ', ')"
knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render \
--width "${width}" --height "${height}" \
--c-re "${c_re}" --c-im "${c_im}" --max-iter "${max_iter}" \
--colormap "${colormap}" --output "${png}"
}
knit_done
# --- App --------------------------------------------------------------------
# One MPI-parallel image. Launched by the job via `knit run`; it inherits the
# job's setup environment, so julia-fractal is already on PATH. Rank 0 records
# the `inside` metric into the `render` table.
knit_register_app "render" _render_app "Render one MPI-parallel Julia-set image."
knit_with_optional "width:integer" "800" "Image width in pixels."
knit_with_optional "height:integer" "600" "Image height in pixels."
knit_with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
knit_with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
knit_with_optional "colormap:string" "fire" "Palette: grayscale | fire | ocean."
knit_with_required "output:string" "Absolute PNG path (the job supplies one per run)."
knit_with_output "inside:integer" "0" "Grid points inside the set (recorded by rank 0)."
_render_app() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
local out
out=$(julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
"${output}" "${colormap}")
printf '%s\n' "${out}"
local inside
inside=$(sed -n 's/.*inside=\([0-9]*\).*/\1/p' <<< "${out}" | head -1)
knit_output "inside" "${inside}"
}
knit_done
# --- Fan-in -----------------------------------------------------------------
# Reads back what the renders recorded. `@without_provenance` marks this as
# read-only bookkeeping, so it writes no row or edge of its own. Each
# `knit run -- render` wrote one row in the `render` table, so a single SELECT
# sees every image the experiment has produced.
knit_register "aggregate" _aggregate \
"Fan-in: total the inside metric across every recorded render."
knit_without_provenance
_aggregate() {
local count total
count=$(knit query sql --exec "SELECT count(*) FROM render;")
total=$(knit query sql --exec "SELECT sum(inside) FROM render;")
printf 'Summed inside=%s over %s render(s).\n' "${total:-0}" "${count:-0}"
}
knit_done
knit "$@"
To run it from scratch:
$ ./exp.sh bootstrap # on a laptop; add --profile <machine> --allocation <alloc> on a cluster
$ ./exp.sh setup --name mympienv -- juliaenv
$ ./exp.sh submit --setup mympienv --wait -- julia --c-re -0.8 --c-im 0.156
$ ./exp.sh aggregate
The first three commands provision, build, and run; aggregate totals the
inside metric across every image you have rendered. Submit julia again
with different constants, on more nodes, or on another machine — the script
never changes.
Continue to Part II — Refining the experiment to refine this experiment.