Part II — Refining the experiment#
Part I built a complete experiment: a Spack-backed setup that
compiles julia-fractal, a julia job that submits it, a render app
that launches it across MPI ranks, and the query, provenance, and AI surfaces on
top. It works end to end, on a laptop and on a supercomputer.
This second part returns to that same experiment and improves it, piece by piece. It follows Part I’s order, from setup to run, and each section takes one part of the experiment you already understand and shows a Knit capability that makes it cleaner, safer, or more reproducible. Each section shows the change against the Part I code, and the finished Part II experiment is collected in one file at the end.
Important
Start Part II from a fresh folder. We recommend that you begin Part II
from a clean slate — copy your experiment script (or the one shown at
the end of Part I) and knit.sh into a new folder and
run ./exp.sh bootstrap there. Part II then runs against a fresh
.knit/knit.db, the various setup names are free again, and every example on
this page (fetch, setup --name mympienv, the submits) runs cleanly with no
remove-first step.
Step 1 — Resources: fetch the source instead of cloning it#
Part I’s juliaenv setup clones the julia-fractal source in its body:
git clone "https://github.com/knit-sh/julia-fractal-example.git" \
"${KNIT_SETUP_PREFIX}/src"
git -C "${KNIT_SETUP_PREFIX}/src" checkout "${ref}"
That clone is invisible to Knit. Nothing records which commit was used, and every fresh build of the setup downloads the source again. A resource fixes both problems: it is a named input that Knit fetches once, caches, and records.
Declare the source as a resource. A resource has no body function. One download decorator says where it comes from:
# A resource declares HOW to acquire an input --- here the julia-fractal source,
# as a git repository pinned to a default ref. It has no body: `knit fetch`
# clones it once into the resources directory, makes it read-only, and records
# the resolved commit for provenance. Fetch a named instance with
# ./exp.sh fetch --name julia_src -- julia_code
# and pin a different revision with `--ref` when you fetch.
@resource "julia_code" "The julia-fractal source (github.com/knit-sh/julia-fractal-example)."
@with_git "https://github.com/knit-sh/julia-fractal-example.git" "v1.1.0"
@done
# A resource declares HOW to acquire an input --- here the julia-fractal source,
# as a git repository pinned to a default ref. It has no body: `knit fetch`
# clones it once into the resources directory, makes it read-only, and records
# the resolved commit for provenance. Fetch a named instance with
# ./exp.sh fetch --name julia_src -- julia_code
# and pin a different revision with `--ref` when you fetch.
knit_register_resource "julia_code" "The julia-fractal source (github.com/knit-sh/julia-fractal-example)."
knit_with_git "https://github.com/knit-sh/julia-fractal-example.git" "v1.1.0"
knit_done
@with_git selects the git backend and pins a default ref; @with_local
and @with_url are the other two backends. Fetch a named instance with the
fetch dispatcher:
$ ./exp.sh fetch --name julia_src -- julia_code
$ ./exp.sh fetch --name julia_src -- julia_code --ref main # pin another ref
fetch clones the repository once at its pinned ref into the resources
directory, makes it read-only, records the resolved commit in the
resource:julia_code table, and prints the on-disk path. Fetching is
idempotent by name: re-fetching julia_src from the same source does nothing.
Now the setup declares the resource it needs instead of cloning it:
# The setup no longer clones anything. It DECLARES that it needs the source with
# @with_resource "src:julia_code": Knit checks the named instance was fetched
# (and is of type julia_code) before the body runs, and records a used_by edge
# from the resource to the setup. The body turns the instance name into its
# on-disk path with knit_resource_path.
@setup "juliaenv" "Build and install julia-fractal with MPI from fetched source."
@with_resource "src:julia_code" "Name of the fetched julia-fractal source to build."
@with_spack_specs "cmake" "libpng" "mpi"
@provides_launcher
_juliaenv_setup() {
# A fetched instance is read-only, so we build OUT of source --- a separate
# build/ tree under the setup prefix --- and never write back into it.
local src
src="$(knit_resource_path "$(knit_get_parameter "src" "$@")")"
cmake -S "${src}" -B "${KNIT_SETUP_PREFIX}/build" \
-DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
cmake --build "${KNIT_SETUP_PREFIX}/build"
cmake --install "${KNIT_SETUP_PREFIX}/build"
knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
@done
# The setup no longer clones anything. It DECLARES that it needs the source with
# @with_resource "src:julia_code": Knit checks the named instance was fetched
# (and is of type julia_code) before the body runs, and records a used_by edge
# from the resource to the setup. The body turns the instance name into its
# on-disk path with knit_resource_path.
knit_register_setup "juliaenv" _juliaenv_setup "Build and install julia-fractal with MPI from fetched source."
knit_with_resource "src:julia_code" "Name of the fetched julia-fractal source to build."
knit_with_spack_specs "cmake" "libpng" "mpi"
knit_provides_launcher
_juliaenv_setup() {
# A fetched instance is read-only, so we build OUT of source --- a separate
# build/ tree under the setup prefix --- and never write back into it.
local src
src="$(knit_resource_path "$(knit_get_parameter "src" "$@")")"
cmake -S "${src}" -B "${KNIT_SETUP_PREFIX}/build" \
-DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
cmake --build "${KNIT_SETUP_PREFIX}/build"
cmake --install "${KNIT_SETUP_PREFIX}/build"
knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
knit_done
@with_resource "src:julia_code" adds a --src option to the setup and
validates it before the body runs: the named instance must exist and be of
type julia_code, or Knit stops and prints the exact fetch command to run
first. Knit also records a used_by edge from the resource to the setup, so
the provenance graph now ties each build to the source revision it consumed.
Inside the body, knit_resource_path turns the instance name into its
directory. Fetch first, then build against the fetched instance:
$ ./exp.sh fetch --name julia_src -- julia_code
$ ./exp.sh setup --name mympienv -- juliaenv --src julia_src
Note
A resource is meant for an input artifact — a dataset or a reference
input — not source code. The better way to handle the julia-fractal
source is to make it a Spack package and let the
Spack-backed setup build it, exactly as the setup already builds cmake,
libpng, and mpi. We fetch the code repository as a resource here only
because it is the input Part I already downloads by hand, so the improvement
is easy to see side by side. Do not read this section as “clone your source
with a resource” — read it as “acquire your inputs once, and let Knit record
them.”
Step 3 — Enums: make colormap a validated type#
colormap accepts only three palettes — grayscale, fire and
ocean — but in Part I it is typed string, so nothing checks the value.
A wrong palette (--colormap purple) passes the command line, reaches
julia-fractal, and fails deep in the render, far from the typo. An enum
moves that check to the front. It is a named type with a fixed set of values,
declared once:
# An enum is a named type with a fixed set of values. Declaring `colormap` here
# lets any parameter be typed `:colormap`; Knit then rejects a value outside the
# set before the command body runs.
@enum "colormap" "grayscale" "fire" "ocean"
# An enum is a named type with a fixed set of values. Declaring `colormap` here
# lets any parameter be typed `:colormap`; Knit then rejects a value outside the
# set before the command body runs.
knit_enum "colormap" "grayscale" "fire" "ocean"
With the type in hand, the shared parameter changes from colormap:string to
colormap:colormap — a single edit in the julia-params set from Step 2,
so the job and the app both pick it up. knit_enum_values builds the help text
from the same declaration, so the list of palettes in the description can never
drift from the type:
# The shared set is unchanged except for `colormap`, which is now typed with the
# enum above instead of a free `string`. `knit_enum_values` builds the help text
# from the same source, so the list of palettes cannot drift from the type.
@parameter_set "julia-params"
@with_optional "width:integer" "800" "Image width in pixels."
@with_optional "height:integer" "600" "Image height in pixels."
@with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
@with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
@with_optional "colormap:colormap" "fire" "Palette: $(knit_enum_values colormap ", ")."
@done
# The shared set is unchanged except for `colormap`, which is now typed with the
# enum above instead of a free `string`. `knit_enum_values` builds the help text
# from the same source, so the list of palettes cannot drift from the type.
knit_parameter_set "julia-params"
knit_with_optional "width:integer" "800" "Image width in pixels."
knit_with_optional "height:integer" "600" "Image height in pixels."
knit_with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
knit_with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
knit_with_optional "colormap:colormap" "fire" "Palette: $(knit_enum_values colormap ", ")."
knit_done
Now a value outside the set is refused up front, before the body runs, with a message that names the accepted values:
$ ./exp.sh submit --setup mympienv -- julia --colormap purple
[knit:fatal] Parameter --colormap of "submit:julia" expects one of: grayscale, fire, ocean (got "purple").
The choices are also advertised, so a user does not have to guess them.
describe inlines them next to the default, and --help shows them through
the description built with knit_enum_values:
$ ./exp.sh describe --only submit:julia
...
--colormap <value> [default: 'fire', one of: grayscale, fire, ocean] Palette: grayscale, fire, ocean.
...
An enum is a definition-time type like the built-in integer or real:
declare it once, name it on as many parameters as you like, and every command
that takes one of those parameters — here both the job and the app, through the
shared set — validates and advertises the same fixed set of values.
Step 4 — File output: record the PNG as a checksummed intermediate#
The render app produces a PNG, but Part I does not record it. Its output
parameter is typed string, which accepts any value and carries no meaning;
nothing checks that the file appears or captures what it contains. If a render
silently writes nothing, the row still looks fine.
Knit has a small family of path-shaped types that say more. They come in two pairs — one for a single file, one for a directory tree — and each pair has a checked member and an unchecked counterpart:
fileis checked;filenameis its unchecked counterpart.directoryis checked;pathis its unchecked counterpart.
All four accept a non-empty string. The difference is what Knit does with the
value. A checked type (file, directory) verifies the target exists
— direction-aware, so an input must exist before the body runs and an output
must exist when it finishes — and records a SHA-256 checksum of its content. An
unchecked type (filename, path) only asserts the value is non-empty;
Knit never touches the filesystem for it. Use an unchecked type when a value
merely names a location, and a checked type when Knit should confirm and
fingerprint what is actually there.
Typing the app’s input and its new output from this family puts each value in the
right category. Bind the produced PNG with knit_output:
# `output` is a `filename` INPUT: the location the job hands in. `filename` is the
# unchecked counterpart of `file` --- Knit does not require it to exist, because
# the render has not written it yet. `image:file` is the OUTPUT: `knit_output
# "image"` binds the produced PNG, and because it is a `file`, Knit checks it
# exists at completion and records its SHA-256 in the `image_checksum` column.
@app "render" "Render one MPI-parallel Julia-set image."
@with_parameter_set "julia-params"
@with_required "output:filename" "Absolute PNG path (the job supplies one per run)."
@with_output "inside:integer" "0" "Grid points inside the set (recorded by rank 0)."
@with_output "image:file" "" "The PNG this render produced (checksummed intermediate)."
_render_app() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
local out
out=$(julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
"${output}" "${colormap}")
printf '%s\n' "${out}"
local inside
inside=$(sed -n 's/.*inside=\([0-9]*\).*/\1/p' <<< "${out}" | head -1)
knit_output "inside" "${inside}"
# Bind the produced PNG. Knit verifies it exists and records its checksum.
knit_output "image" "${output}"
}
@done
# `output` is a `filename` INPUT: the location the job hands in. `filename` is the
# unchecked counterpart of `file` --- Knit does not require it to exist, because
# the render has not written it yet. `image:file` is the OUTPUT: `knit_output
# "image"` binds the produced PNG, and because it is a `file`, Knit checks it
# exists at completion and records its SHA-256 in the `image_checksum` column.
knit_register_app "render" _render_app "Render one MPI-parallel Julia-set image."
knit_with_parameter_set "julia-params"
knit_with_required "output:filename" "Absolute PNG path (the job supplies one per run)."
knit_with_output "inside:integer" "0" "Grid points inside the set (recorded by rank 0)."
knit_with_output "image:file" "" "The PNG this render produced (checksummed intermediate)."
_render_app() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
local out
out=$(julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
"${output}" "${colormap}")
printf '%s\n' "${out}"
local inside
inside=$(sed -n 's/.*inside=\([0-9]*\).*/\1/p' <<< "${out}" | head -1)
knit_output "inside" "${inside}"
# Bind the produced PNG. Knit verifies it exists and records its checksum.
knit_output "image" "${output}"
}
knit_done
The output parameter is now a filename: the destination the job hands in.
It must not be a file, because a file input is checked for existence
before the body runs — and the render has not written the PNG yet, so that
would fail every run. filename — the unchecked counterpart of file —
is exactly right for “somewhere to write.” (The job types its own bare output
name the same way.)
The image output, by contrast, is a file. Because the value is a path to
a file that must exist when the app finishes, Knit does two things Part I did
not. First, it enforces existence: a render that produced no file is now caught
at once, with a fatal error, not discovered later. Second, it records the
content: it hashes the file with SHA-256 and writes the digest into a companion
image_checksum column that it adds to the render table automatically.
describe shows both the output and its synthesized column:
$ ./exp.sh describe --only run:render
...
Outputs
-------
inside [integer, default: '0'] Grid points inside the set (recorded by rank 0).
image [file, default: ''] The PNG this render produced (checksummed intermediate).
image-checksum [string, default: ''] SHA-256 checksum of "image", recorded automatically.
The checksum lets a reproducer confirm that a re-run produced byte-for-byte the
same image. Hashing a large file has a cost, so a file output can opt out of the
digest with --no-checksum on the declaration; the existence check still
applies (it is a property of the type), only the image_checksum column is
dropped.
The PNG stays exactly where the job wrote it, in the job directory. A file output does not move the file or copy it anywhere — it records a path and a digest against the row that produced it. That makes the image a checksummed intermediate: tracked, verifiable, and tied to its render, but local to the run. The next section introduces its counterpart, an artifact, which is a result Knit relocates and records under a common root; the contrast between a checksummed file output that stays put and an artifact that is collected is the point of Step 5.
Note
The directory pair mirrors the file pair. directory is checked like
file — same direction-aware existence check, with the checksum taken over
the tree’s contents — and path is its unchecked counterpart, the way
filename is for file. Reach for directory when an output is a tree
rather than a single file, and for path when you only need to name one.
Step 5 — Results and artifacts: record the table as a result artifact#
Part I’s aggregate command is read-only bookkeeping. It is marked
@without_provenance, so it records nothing: it runs two knit query sql
counts, prints a one-line summary, and leaves no trace that it ran or what it
found. The numbers scroll past and are gone.
This section turns aggregate into the command that produces the experiment’s
headline result, recorded as an artifact — a durable, relocatable output
Knit tracks in a table of its own (how that differs from the file output of Step
4 comes later in this section). An artifact has a physical type — file or
directory — and, optionally, a semantic kind: a label backed by one
physical type that says what the file means. Declare the kind once at the top
level. Here insidecsv marks a file as the inside-metric table, so a
consumer can later require that kind rather than merely “a file”:
# A KIND gives an artifact a semantic label backed by one physical type. Declared
# once at the top level, `insidecsv` marks a file as *the inside-metric table*, so
# a consumer can require that kind rather than merely "a file".
@artifact "insidecsv:file" "Per-render inside metric, one row per image (CSV)."
# A KIND gives an artifact a semantic label backed by one physical type. Declared
# once at the top level, `insidecsv` marks a file as *the inside-metric table*, so
# a consumer can require that kind rather than merely "a file".
knit_register_artifact "insidecsv:file" "Per-render inside metric, one row per image (CSV)."
With the kind declared, two changes turn aggregate into its producer: drop
@without_provenance so the command records its own row and provenance, and
declare the table it writes as an output artifact of that kind:
# Part I's aggregate was read-only bookkeeping (@without_provenance): it printed a
# sum and left no trace. Now it produces a result. Dropping @without_provenance
# lets it record its own row and provenance; @with_table gives it that row, and
# @with_output_artifact declares the CSV it writes as the headline RESULT --- and
# names the `insidecsv` KIND, not the bare `file` type.
@command "aggregate" "Fan-in: tabulate every render into a result CSV."
@with_table
@with_output_artifact "table:insidecsv" "Per-render inside metric, one row per image (CSV)." --result
_aggregate() {
# knit_artifact_dir is the artifacts/ root: write into it, then declare.
local out
out="$(knit_artifact_dir)"
mkdir -p "${out}"
# Build the table straight from the provenance graph with a Cypher query:
# walk from each run to the render it launched, projecting the constant and
# the inside metric. --format csv emits comma-separated rows; --header adds
# the column names.
knit query graph --format csv --header --exec \
"MATCH (r:run)-[:call]->(img:render)
RETURN img.c_re AS c_re, img.c_im AS c_im, img.inside AS inside" \
> "${out}/inside.csv"
# Bind the CSV as a result artifact: Knit records it as a row in the
# framework-owned artifacts table, linked to THIS invocation by a `produced`
# edge, with its content checksummed and its path stored relative to the
# artifacts root (never an absolute machine path).
knit_artifact "table" "inside.csv"
printf 'wrote %s\n' "${out}/inside.csv"
}
@done
# Part I's aggregate was read-only bookkeeping (@without_provenance): it printed a
# sum and left no trace. Now it produces a result. Dropping @without_provenance
# lets it record its own row and provenance; @with_table gives it that row, and
# @with_output_artifact declares the CSV it writes as the headline RESULT --- and
# names the `insidecsv` KIND, not the bare `file` type.
knit_register "aggregate" _aggregate "Fan-in: tabulate every render into a result CSV."
knit_with_table
knit_with_output_artifact "table:insidecsv" "Per-render inside metric, one row per image (CSV)." --result
_aggregate() {
# knit_artifact_dir is the artifacts/ root: write into it, then declare.
local out
out="$(knit_artifact_dir)"
mkdir -p "${out}"
# Build the table straight from the provenance graph with a Cypher query:
# walk from each run to the render it launched, projecting the constant and
# the inside metric. --format csv emits comma-separated rows; --header adds
# the column names.
knit query graph --format csv --header --exec \
"MATCH (r:run)-[:call]->(img:render)
RETURN img.c_re AS c_re, img.c_im AS c_im, img.inside AS inside" \
> "${out}/inside.csv"
# Bind the CSV as a result artifact: Knit records it as a row in the
# framework-owned artifacts table, linked to THIS invocation by a `produced`
# edge, with its content checksummed and its path stored relative to the
# artifacts root (never an absolute machine path).
knit_artifact "table" "inside.csv"
printf 'wrote %s\n' "${out}/inside.csv"
}
knit_done
Note
How the CSV is built is incidental to the artifact — any command that writes
a file into the artifacts root will do. This body happens to build it with
knit query graph (a Cypher query, --format csv) instead of Part I’s
knit query sql, just as an example of a query that yields the table: the
pattern (r:run)-[:call]->(img:render) walks from each run to the render it
launched. See the Query the provenance graph stitch for what Cypher can do.
An artifact records the result. @with_output_artifact "table:insidecsv"
... --result declares that the command produces an artifact of the
insidecsv kind and that it is the result — what the experiment was for.
Inside the body, knit_artifact_dir gives the artifacts root, and
knit_artifact "table" "inside.csv" binds the file that was written there. The
kind is recorded in the artifacts row and describe shows it (not the bare
physical type) in the command’s own section, flagged as a result:
$ ./exp.sh describe --only aggregate
...
Artifacts
---------
table [insidecsv, result] Per-render inside metric, one row per image (CSV).
An artifact is not a file output. The distinction is the point of this section:
A file output (Step 4) is a column on the producing command’s own table. It records a path and a checksum, and the file stays exactly where it was written — the PNG never moves. It is a tracked intermediate, local to the run.
An artifact is a row in the framework-owned
artifactstable, a node of its own, linked to the invocation that made it by aproducededge. Its path is stored relative to the artifacts root — never an absolute machine path — so the wholeartifacts/tree can be moved or shipped and the records still resolve. That relocatable, self-describing entry is the durable record of what the run produced.--resultchanges nothing about how the artifact is stored; it is only a marker that flags the entry as a headline output, sodescribeand other surfaces can draw attention to it.
Knit always checksums an artifact (there is no --no-checksum for it), and
each artifacts-relative path is write-once: binding inside.csv twice is a
fatal error, so a re-run either uses a fresh name or removes the old entry first
— Step 7 erases recorded entities and their artifacts.
In Part I, aggregate recorded nothing — no row, no result, no file. In Part II the same
command records a row, marks a result, writes a relocatable table, and leaves a
produced edge from that row to the table’s node. That edge is queryable like
any other. To find which command produced a given artifact, walk the
produced edge backwards:
$ ./exp.sh query graph --format column --header --exec \
"MATCH (t)-[e:produced]->(a:artifacts)
WHERE a.name = 'table' RETURN e.source_name, a.path, a.checksum"
Consuming an artifact. An artifact may not be the final product of an experiment. For instance, a CSV file could be an artifact, and another command could consume it to produce a figure, or print out a report on the standard output.
To showcase this, we will now add a report command that consumes the CSV table aggregate
produced and names the render with the largest inside metric:
# `report` CONSUMES the artifact `aggregate` produced. @with_input_artifact
# requires a recorded artifact of kind `insidecsv`; the parameter value is its
# artifacts-relative path. Before the body runs, Knit resolves the path, refuses a
# missing artifact or a wrong kind, and --- with --verify-checksum --- re-hashes
# the bytes and refuses a table that changed since it was produced. Consuming the
# artifact records a `used_by` edge from it to this command, the mirror of the
# producer's `produced` edge.
@command "report" "Read the inside-metric table and name the peak render."
@with_input_artifact "table:insidecsv" "Artifacts-relative path of the table to read." --verify-checksum
@with_table
@with_output "peak:real" "0" "Largest inside metric across the rendered images." --result
_report() {
# Resolve the recorded artifacts-relative path to the on-disk file.
local csv
csv="$(knit_input_artifact_path "$(knit_get_parameter "table" "$@")")"
# The inside column is the third field; skip the header, take the maximum.
local peak
peak=$(tail -n +2 "${csv}" | cut -d, -f3 | sort -n | tail -1)
knit_output "peak" "${peak:-0}"
printf 'peak inside metric: %s\n' "${peak:-0}"
}
@done
# `report` CONSUMES the artifact `aggregate` produced. @with_input_artifact
# requires a recorded artifact of kind `insidecsv`; the parameter value is its
# artifacts-relative path. Before the body runs, Knit resolves the path, refuses a
# missing artifact or a wrong kind, and --- with --verify-checksum --- re-hashes
# the bytes and refuses a table that changed since it was produced. Consuming the
# artifact records a `used_by` edge from it to this command, the mirror of the
# producer's `produced` edge.
knit_register "report" _report "Read the inside-metric table and name the peak render."
knit_with_input_artifact "table:insidecsv" "Artifacts-relative path of the table to read." --verify-checksum
knit_with_table
knit_with_output "peak:real" "0" "Largest inside metric across the rendered images." --result
_report() {
# Resolve the recorded artifacts-relative path to the on-disk file.
local csv
csv="$(knit_input_artifact_path "$(knit_get_parameter "table" "$@")")"
# The inside column is the third field; skip the header, take the maximum.
local peak
peak=$(tail -n +2 "${csv}" | cut -d, -f3 | sort -n | tail -1)
knit_output "peak" "${peak:-0}"
printf 'peak inside metric: %s\n' "${peak:-0}"
}
knit_done
@with_input_artifact "table:insidecsv" registers a required parameter whose
value is the artifacts-relative path of a recorded artifact of kind
insidecsv. Before the body runs, Knit resolves the path to its artifacts row
and refuses the run when the value is empty, when no artifact is recorded there,
or when the recorded kind is not insidecsv — so report never reads the
wrong kind of input. --verify-checksum adds one more guard: it re-hashes the
bytes and refuses a table that changed since it was produced, against the digest
aggregate recorded. In the body, knit_input_artifact_path turns the stored
path into the on-disk file to read. Run it against the table by its
artifacts-relative path:
$ ./exp.sh report --table inside.csv
peak inside metric: 119960
Consuming the artifact records a used_by edge from the artifact’s row to
report, the mirror of the produced edge aggregate left. The two
edges meet at the same artifact node, so one query walks the whole chain
aggregate --produced--> table --used_by--> report:
$ ./exp.sh query graph --format column --header --exec \
"MATCH (p)-[pr:produced]->(a:artifacts)-[ub:used_by]->(c)
WHERE a.name = 'table'
RETURN pr.source_name, a.kind, ub.target_name"
The experiment now has a result that outlives the run — a checksummed, relocatable table — and a lineage that runs both ways from it: back to the command that built it and every render that went into it, and forward to every command that read it.
Note
A command sometimes produces or consumes an unknown number of artifacts —
one shard per seed, one frame per step. An artifact kind takes a quantifier
for this: * (zero or more) or + (one or more) turns the name into a
collection the body binds once per member, and a matching quantifier on an
input artifact gathers the whole set through a single glob argument with
knit_input_artifact_paths. See the Fan out a variadic output, Consume
many artifacts with a glob, and Discover and merge a fan-out stitches for
the full recipes.
Step 6 — Prepare: build a batch of runs, release them on your terms#
Part I’s sweep fired one job at a time:
$ ./exp.sh submit --setup mympienv --wait -- julia --c-re -0.123 --c-im 0.745
$ ./exp.sh submit --setup mympienv --wait -- julia --c-re -1.0 --c-im 0.0
... five more ...
submit does two things at once: it builds the job — validates it, creates
its directory, writes the batch script, records the row — and then dispatches
it to the scheduler. Fusing the two is fine for one job, but a sweep wants them
apart: describe every run up front, then release the runs at a rate you control
(a few at a time, a cron tick, or a loop that keeps N in flight). prepare
does exactly the build half.
Prepare one job. prepare mirrors submit argument-for-argument over
the same job registry — minus --wait, since nothing is dispatched — and
adds a --group label you can filter on later. It records the jobs row in a
new prepared state and stops before the scheduler:
$ ./exp.sh prepare --setup mympienv --group julia-sweep -- julia --c-re -0.123 --c-im 0.745
018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a
The whole submission spec — setup, nodes, walltime, queue, and the job’s own
arguments — is frozen at prepare time and baked into the batch script, so
releasing the job never re-opens those options. The UUID it prints is the same
stable identifier submit would have given it.
Prepare the whole sweep from a plan. Typing one prepare per constant is
no better than typing one submit per constant. prepare from reads the
whole batch as a JSON plan — from a file with --file, or on stdin — and
prepares each entry as if you had run prepare -- julia … by hand. Part I’s
seven-line sweep becomes one plan:
{
"group": "julia-sweep",
"defaults": { "setup": "mympienv" },
"jobs": [
{ "job": "julia", "args": { "c-re": -0.123, "c-im": 0.745 } },
{ "job": "julia", "args": { "c-re": -1.0, "c-im": 0.0 } },
{ "job": "julia", "args": { "c-re": -0.391, "c-im": -0.587 } },
{ "job": "julia", "args": { "c-re": 0.285, "c-im": 0.535 } },
{ "job": "julia", "args": { "c-re": -0.7, "c-im": 0.0 } },
{ "job": "julia", "args": { "c-re": -1.25, "c-im": 0.0 } },
{ "job": "julia", "args": { "c-re": -0.1, "c-im": 0.651 } }
]
}
group tags every prepared job; defaults is a field map merged under
each entry (so --setup mympienv is written once, not seven times); jobs
lists the entries. Within an entry, args is the job’s own arguments (an object
{"c-re": -1.0} becomes --c-re -1.0), and any other key is a submission
option (nodes, walltime, …), exactly as on the prepare command line.
Feed the plan on stdin or from a file:
$ ./exp.sh prepare from --file sweep.json
$ ./exp.sh prepare from < sweep.json
The whole plan is validated before any job is prepared, so a malformed plan — an
unknown key, a missing job — leaves nothing half-prepared. The jobs are
prepared in plan order.
Vary a submission option with a matrix. A jobs entry may be a matrix
block instead of a concrete entry. It expands to one prepared job per combination:
the cartesian product of its axes, minus every exclude, plus each
include. An args axis varies the job’s own arguments; a bare axis key
varies a submission option. To render a few constants at both one and two nodes,
give an args axis and a nodes axis:
{ "matrix": {
"job": "julia",
"axes": {
"args": [ { "c-re": -0.123, "c-im": 0.745 }, { "c-re": -1.0, "c-im": 0.0 } ],
"nodes": [ 1, 2 ]
}
} }
That single block expands to four prepared jobs (two constants times two node counts).
Inspect the batch. Prepared jobs can be listed as follows.
$ ./exp.sh job list --status prepared
Release on your terms. Two submit subcommands hand prepared jobs to the
scheduler. submit prepared --id releases one specific job; submit next
releases the oldest prepared job, optionally narrowed by --type (the job name)
or --group. The claim is atomic, so concurrent releasers never double-submit a
row, and submit next returns non-zero when nothing matches — so a
fill-the-queue loop drains a group and stops on its own:
$ ./exp.sh submit prepared --id 018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a --wait
$ while ./exp.sh submit next --group julia-sweep --wait; do :; done
Releasing advances the same row through prepared — submitted —
running — completed; the UUID never changes, so the job you prepared and
the job that ran are one recorded entity, and knit query sees the whole group
at once:
$ ./exp.sh query sql --format column --header --exec \
"SELECT id, job, state, \"group\" FROM jobs WHERE \"group\" = 'julia-sweep'"
Preparing separates describing a batch of runs from releasing them: the plan is the durable description, and the release policy is yours.
Step 7 — Remove: erase recorded entities and their dependents#
Sooner or later you may want to take something back out of the experiment: a resource
you fetched, a setup you built, a job or a plain command you ran, an artifact one
of them produced. knit remove erases any of these. The catch is that a
recorded entity is never isolated — Part I built a provenance graph tying it all
together, and the earlier steps of Part II added edges of their own. A job used
a setup; a run belongs to a job; a command produced an artifact. Deleting one
row without regard for those edges would leave the graph dangling, its records
pointing at things that no longer exist.
knit remove is provenance-aware, so it never does that. It erases the entity
you name and everything recorded downstream of it — the rows it called, the
runs and app invocations under them, and the artifacts they produced — in one
transaction. What the entity used (a setup, a resource) is left
alone: those edges point into it, not out of it, so removing a job never takes
its setup with it.
Preview before you delete. Every remove prints an itemized report of
exactly what it will erase, and --dry-run prints that report and stops —
nothing is touched. Point it at one finished job by its id:
$ ./exp.sh remove job --id 018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a --dry-run
The following will be permanently erased:
Data rows (4):
job julia 018f9c3a-... (state: completed)
job julia 01926b7a-... (body row)
run render 01926b7c-...
app render 01926b7d-...
Provenance edges (5):
submit 018f9c3a-... --call--> julia 01926b7a-...
...
Directories and artifacts removed (1):
jobs/018f9c3a-...
The render app writes fractal.png into the job directory, so the PNG goes
with jobs/018f9c3a-... above; the report does not list it a second time. A
plain output written outside every removed directory would appear under a
separate “Left on disk” heading, and so would an artifact kept by
--keep-artifacts (described below).
Delete, with a confirmation. Drop --dry-run and remove prints the same
report, then prompts before touching anything:
$ ./exp.sh remove job --id 018f9c3a-7b2e-7c41-9d0a-1f2e3d4c5b6a
The following will be permanently erased:
...
Erase these? [y/N]
Answer y to proceed; anything else aborts, untouched. Pass --yes to skip
the prompt for a script or a non-interactive shell.
Prune the batch by group. The --group selector names the whole
julia-sweep batch Step 6 prepared, so one command erases all of it:
$ ./exp.sh remove job --group julia-sweep --dry-run # preview the whole batch
$ ./exp.sh remove job --group julia-sweep --yes
A live job is never erased out from under you: a job still prepared,
submitted, or running is refused, with a pointer to job cancel. Cancel
it, or let it finish, then remove it. Only a completed or killed job — a
job that is truly done — can be erased. This refusal applies to --dry-run
too: when a live entity would block the removal, --dry-run stops with the same
job cancel pointer rather than printing a preview.
Remove a setup and rebuild. Part I treated a setup as write-once: with no way
to delete one, the advice was to give each rebuild a fresh name and let the old
ones pile up. That constraint is gone. Removing a setup cascades downward to
every job that used it — and each job’s runs and renders — in one step, while
the fetched julia_code resource above it stays put:
$ ./exp.sh remove setup --name mympienv --dry-run # preview the cascade
$ ./exp.sh remove setup --name mympienv --yes
Now setup --name mympienv -- juliaenv builds a clean one against the same
resource. Retire and rebuild, rather than accumulate mympienv, mympienv2,
mympienv3 forever.
Erase a whole lineage. An artifact cannot be removed on its own: its producer
is kept, so deleting just the file would dangle the produced edge, and Knit
refuses with a hint. --from-root widens the erase set to the entire
call/produced lineage the artifact belongs to — up to the root and back down —
so the producing command and the artifact go together:
$ ./exp.sh remove artifact --path inside.csv --from-root --yes
Keep the files, prune the records. Two flags stop remove short of the
disk, at two scopes. --keep-artifacts erases the rows and edges and still
removes the job, setup, and resource directories, but leaves the recorded
artifact entries under artifacts/ in place, each listed under “Left on
disk”; use it to reclaim the database while keeping the results a run produced.
--keep-files goes further: it erases the rows and edges only and makes no
filesystem change at all — every directory, artifact, and plain output stays,
each listed under “Left on disk”; use it to prune the provenance while leaving
the whole on-disk tree untouched.
Step 8 — Bundle: ship the finished experiment#
The experiment is done: it fetched its source, built an environment, rendered images across a cluster, aggregated a result artifact, and recorded every step. Now package it to send to a collaborator or deposit in a repository such as Zenodo. Like Steps 6 and 7, this adds no code — it is a console workflow over the same script.
bundle packs the whole experiment into one archive that unpacks anywhere,
carrying a complete record of what was done:
$ ./exp.sh bundle
[knit:info] Wrote bundle to ./exp-bundle.tar.gz
Every path inside the archive is relative to exp.sh at the root, so the tree
relocates cleanly. A symlink that points outside the tree — on an HPC machine some data often live on a faster parallel filesystem and are linked back in — is
dereferenced, so its content travels rather than a link that would dangle on
the reproducer’s machine.
Embedded vs referenced. The bundle draws a deliberate line. It embeds what is small and defining, and references (records how to rebuild) what is large and derivable:
Embedded by default: the code (
exp.shandknit.sh), the provenance database (a pruned.knit/knit.db— just the database, not the provisioned tooling around it), the job logs and scripts, the setup manifests (.activate.shand thespack.yaml/spack.lockthat pin the environment), and the declared artifacts (inside.csvand the images).Referenced, not embedded: the built Spack environment and the provisioned
.knit/spack,.knit/sqlite,.knit/jqtoolchain (gigabytes, regenerable), and the fetchedjulia_codegit resource (re-fetchable from the commit Knit recorded).
So the archive is a compact, durable trace of the experiment: the exact code
(exp.sh and knit.sh), the provenance database of what ran, the pinned
setup manifests (spack.yaml / spack.lock), and the recorded source commit
— everything a reader needs to see how each result was produced and to
reconstruct the environment, rather than a multi-gigabyte copy of the machine.
Tune the line with the include/exclude flags: --no-knit / --no-db /
--no-artifacts and friends drop a default group, while
--include-job-content and --include-resources <name> pull bulky content
in. Files Knit does not track — a config, a plotting script — ride along when
the script declares them with knit_bundle_requires (see the Bundle & export
stitches).
Note
The bundle records what a run needs to be reproduced, but Knit does not yet
replay it for you: there is no single command that rebuilds the Spack
environment from the bundled spack.lock and re-runs the recorded steps. For
now, treat the bundle as a trace — read its provenance, bootstrap a fresh
experiment, re-fetch the recorded commit, and rebuild the environment from the
pinned manifests by hand. Turnkey reproduction from a bundle is planned for a
future release.
Preview the exact contents before writing anything with --dry-run, and write
a zip (what Zenodo and WorkflowHub expect) with --zip:
$ ./exp.sh bundle --dry-run --size
$ ./exp.sh bundle --zip --output julia.zip
A standard research object. Add --ro-crate and Knit writes an
ro-crate-metadata.json at the archive root, describing the packed files with
the RO-Crate 1.1 / Process Run
Crate vocabulary — one CreateAction per recorded run, wired to its inputs
and outputs. The archive is then self-describing and FAIR-friendly, and
--ro-crate --zip is ready to upload as-is:
$ ./exp.sh bundle --ro-crate --zip --output julia.zip
To inspect or regenerate that manifest on its own — no archive, no copied files
— use knit export ro-crate (--output - writes it to stdout). Both read
knit.db and record nothing, so they are safe to run at any time after
bootstrap.
Step 9 — Querying across machines#
Every step so far ran against one experiment on one machine. Part II is meant to
run in multiple places, though: render on your laptop while you develop, then move
the same script to a supercomputer and bootstrap it there (bootstrap
--profile <machine> --allocation <alloc>, as Part I’s move-to-HPC step covers)
and render at scale. Each machine keeps its own, independent .knit/knit.db
— the laptop’s runs never mix into the cluster’s, and neither database knows
the other exists.
knit query --extra reads them together without merging them. It assembles
a throwaway, read-only union of the current database and the others you name — a
lens — runs one query against the whole set, then discards the union. Nothing
is copied, and each source database stays single-platform. --extra takes a
comma-separated list, and each entry is a directory (its .knit/knit.db is
used), a database file, or a bundle (a .tar.gz from Step 8, so a collaborator’s results join the lens directly):
$ ./exp.sh query graph --extra ../run-on-cluster --format column --header --exec \
"MATCH (p:platform)-[:executed]->(r:run)-[:call]->(img:render)
RETURN p.id AS platform, p.arch AS arch, img.c_re AS c_re, img.inside AS inside"
Inside the lens every database contributes a platform node whose properties
are that machine’s fingerprint (arch, scheduler, launcher,
profile, knit_version), joined to every row that ran on it by an
executed edge. So one flat hop tags each render with the machine that
produced it, and the query above lists the laptop’s renders and the cluster’s
side by side. The platform node is synthesized from each database’s own metadata
at query time, so it needs nothing recorded up front and works even against a
single database — with no --extra the lens is just the current one.
Because the whole comparison happens in one lens, count, max, ORDER
BY and DISTINCT are correct across every platform at once — which a shell
loop that ran the query once per database and concatenated the output could not
guarantee. Filter by a machine attribute the same way, for instance WHERE
p.arch = 'aarch64' to keep only the cluster’s rows. The Query across
platforms stitch carries the full recipe, including the aggregate SQL form.
Important
Until Knit reaches version 1.0 and a stable database schema, we recommend that you rely on the same version of Knit across machines if you intent to run cross-platform queries.
The complete experiment (Part II)#
Here is the whole refined experiment in one file. It is the Part I script with
every Part II change from Steps 1–5 folded in: the git resource feeding the
setup, the shared parameter set, the colormap enum, the app’s checksummed
file output, the fan-in that produces a result artifact of kind insidecsv,
and the report consumer that reads it back. Steps 6 through 9 add no code —
prepare, remove, bundle and the cross-platform query are console
workflows over this same script.
Save it as exp.sh next to a copy of knit.sh and make it executable
(chmod +x exp.sh):
#!/bin/bash
# doc-check: source-only
#
# The complete Part II experiment: the same julia-fractal experiment Part I
# built, refined piece by piece. A git RESOURCE feeds the setup, a shared
# PARAMETER SET and a colormap ENUM remove the duplicated parameters, the app
# records its PNG as a checksummed FILE OUTPUT, and the fan-in produces a result
# ARTIFACT from a graph query. This is the final state after Part II's Steps 1--5;
# Steps 6 (prepare) and 7 (remove) are console workflows over this same script.
#
# Source-only: building the setup needs a live Spack, a compiler, an MPI, and
# network access, and launching needs an MPI launcher, none of which run in plain
# CI, so check-docs only syntax-checks this file. Behavior was validated live
# during development.
source knit.sh
knit_set_program_description "Render a Julia-set fractal."
# --- Resource ---------------------------------------------------------------
# The julia-fractal source, acquired once with `knit fetch` and recorded at its
# resolved commit. A resource has no body; a download decorator says where it
# comes from. Fetch a named instance with `fetch --name <n> -- julia_code`.
@resource "julia_code" "The julia-fractal source (github.com/knit-sh/julia-fractal-example)."
@with_git "https://github.com/knit-sh/julia-fractal-example.git" "v1.1.0"
@done
# --- Setup ------------------------------------------------------------------
# Builds and installs julia-fractal with MPI from the fetched source instead of
# cloning it inline. It declares the resource it needs (up-front validation and a
# `used_by` edge) and resolves the name to a path. Spack provisions the build
# dependencies; on a machine with no launcher of its own, the setup provides one.
@setup "juliaenv" "Build and install julia-fractal with MPI from fetched source."
@with_resource "src:julia_code" "Name of the fetched julia-fractal source to build."
@with_spack_specs "cmake" "libpng" "mpi"
@provides_launcher
_juliaenv_setup() {
local src
src="$(knit_resource_path "$(knit_get_parameter "src" "$@")")"
cmake -S "${src}" -B "${KNIT_SETUP_PREFIX}/build" \
-DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
cmake --build "${KNIT_SETUP_PREFIX}/build"
cmake --install "${KNIT_SETUP_PREFIX}/build"
knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
@done
# --- Enum -------------------------------------------------------------------
# A user-defined, validated type. `--colormap purple` is refused up front, and
# `describe`/`--help` advertise the choices.
@enum "colormap" "grayscale" "fire" "ocean"
# --- Parameter set ----------------------------------------------------------
# The six shared parameters, defined once and imported by both the job and the
# app. A conflicting redefinition on import is fatal. `colormap` uses the enum.
@parameter_set "julia-params"
@with_optional "width:integer" "800" "Image width in pixels."
@with_optional "height:integer" "600" "Image height in pixels."
@with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
@with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
@with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
@with_optional "colormap:colormap" "fire" "Palette: $(knit_enum_values colormap ", ")."
@done
# --- Job --------------------------------------------------------------------
# Submitted to the scheduler. Imports the shared set and keeps its own `output`
# name. Its body launches the render app across one rank per allocated node with
# `knit run`, so the same job runs one rank on a laptop and one rank per node
# across a multi-node allocation.
@job "julia" "Render a Julia-set fractal as a submitted job."
@with_setup "juliaenv"
@with_parameter_set "julia-params"
@with_optional "output:filename" "fractal.png" "PNG file name, written in the job directory."
_julia() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
local png="${KNIT_JOB_PREFIX%/}/${output}"
printf 'The job is running on hosts: %s\n' \
"$(knit_job_hostnames --separator ', ')"
knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render \
--width "${width}" --height "${height}" \
--c-re "${c_re}" --c-im "${c_im}" --max-iter "${max_iter}" \
--colormap "${colormap}" --output "${png}"
}
@done
# --- App --------------------------------------------------------------------
# One MPI-parallel image. Imports the shared set. Rank 0 records the `inside`
# metric and the produced PNG as a checksummed `file` output (an intermediate
# that stays in the job directory, with an `image_checksum` column on this table).
@app "render" "Render one MPI-parallel Julia-set image."
@with_parameter_set "julia-params"
@with_required "output:filename" "Absolute PNG path (the job supplies one per run)."
@with_output "inside:integer" "0" "Grid points inside the set (recorded by rank 0)."
@with_output "image:file" "" "The PNG this render produced (checksummed intermediate)."
_render_app() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
local out
out=$(julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
"${output}" "${colormap}")
printf '%s\n' "${out}"
local inside
inside=$(sed -n 's/.*inside=\([0-9]*\).*/\1/p' <<< "${out}" | head -1)
knit_output "inside" "${inside}"
knit_output "image" "${output}"
}
@done
# --- Artifact kind ----------------------------------------------------------
# A KIND is a semantic label backed by one physical type. `insidecsv` marks a
# file as the inside-metric table, so a consumer can require that kind.
@artifact "insidecsv:file" "Per-render inside metric, one row per image (CSV)."
# --- Fan-in -----------------------------------------------------------------
# Produces a result instead of just printing one. It records its own row and
# provenance (no @without_provenance), builds a CSV straight from the provenance
# graph with a Cypher query, and declares that CSV as a result ARTIFACT of kind
# `insidecsv`: a row in the artifacts table, a `produced` edge, a content
# checksum, and a path stored relative to the artifacts root.
@command "aggregate" "Fan-in: tabulate every render into a result CSV."
@with_table
@with_output_artifact "table:insidecsv" "Per-render inside metric, one row per image (CSV)." --result
_aggregate() {
local out
out="$(knit_artifact_dir)"
mkdir -p "${out}"
knit query graph --format csv --header --exec \
"MATCH (r:run)-[:call]->(img:render)
RETURN img.c_re AS c_re, img.c_im AS c_im, img.inside AS inside" \
> "${out}/inside.csv"
knit_artifact "table" "inside.csv"
printf 'wrote %s\n' "${out}/inside.csv"
}
@done
# --- Consumer ---------------------------------------------------------------
# Consumes the artifact `aggregate` produced. @with_input_artifact requires a
# recorded artifact of kind `insidecsv` (the value is its artifacts-relative
# path); --verify-checksum re-hashes the bytes and refuses a table that changed
# since it was produced. Reading it records a `used_by` edge from the artifact to
# this command, mirroring the producer's `produced` edge.
@command "report" "Read the inside-metric table and name the peak render."
@with_input_artifact "table:insidecsv" "Artifacts-relative path of the table to read." --verify-checksum
@with_table
@with_output "peak:real" "0" "Largest inside metric across the rendered images." --result
_report() {
local csv
csv="$(knit_input_artifact_path "$(knit_get_parameter "table" "$@")")"
local peak
peak=$(tail -n +2 "${csv}" | cut -d, -f3 | sort -n | tail -1)
knit_output "peak" "${peak:-0}"
printf 'peak inside metric: %s\n' "${peak:-0}"
}
@done
knit "$@"
#!/bin/bash
# doc-check: source-only
#
# The complete Part II experiment: the same julia-fractal experiment Part I
# built, refined piece by piece. A git RESOURCE feeds the setup, a shared
# PARAMETER SET and a colormap ENUM remove the duplicated parameters, the app
# records its PNG as a checksummed FILE OUTPUT, and the fan-in produces a result
# ARTIFACT from a graph query. This is the final state after Part II's Steps 1--5;
# Steps 6 (prepare) and 7 (remove) are console workflows over this same script.
#
# Source-only: building the setup needs a live Spack, a compiler, an MPI, and
# network access, and launching needs an MPI launcher, none of which run in plain
# CI, so check-docs only syntax-checks this file. Behavior was validated live
# during development.
source knit.sh
knit_set_program_description "Render a Julia-set fractal."
# --- Resource ---------------------------------------------------------------
# The julia-fractal source, acquired once with `knit fetch` and recorded at its
# resolved commit. A resource has no body; a download decorator says where it
# comes from. Fetch a named instance with `fetch --name <n> -- julia_code`.
knit_register_resource "julia_code" "The julia-fractal source (github.com/knit-sh/julia-fractal-example)."
knit_with_git "https://github.com/knit-sh/julia-fractal-example.git" "v1.1.0"
knit_done
# --- Setup ------------------------------------------------------------------
# Builds and installs julia-fractal with MPI from the fetched source instead of
# cloning it inline. It declares the resource it needs (up-front validation and a
# `used_by` edge) and resolves the name to a path. Spack provisions the build
# dependencies; on a machine with no launcher of its own, the setup provides one.
knit_register_setup "juliaenv" _juliaenv_setup "Build and install julia-fractal with MPI from fetched source."
knit_with_resource "src:julia_code" "Name of the fetched julia-fractal source to build."
knit_with_spack_specs "cmake" "libpng" "mpi"
knit_provides_launcher
_juliaenv_setup() {
local src
src="$(knit_resource_path "$(knit_get_parameter "src" "$@")")"
cmake -S "${src}" -B "${KNIT_SETUP_PREFIX}/build" \
-DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
cmake --build "${KNIT_SETUP_PREFIX}/build"
cmake --install "${KNIT_SETUP_PREFIX}/build"
knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
knit_done
# --- Enum -------------------------------------------------------------------
# A user-defined, validated type. `--colormap purple` is refused up front, and
# `describe`/`--help` advertise the choices.
knit_enum "colormap" "grayscale" "fire" "ocean"
# --- Parameter set ----------------------------------------------------------
# The six shared parameters, defined once and imported by both the job and the
# app. A conflicting redefinition on import is fatal. `colormap` uses the enum.
knit_parameter_set "julia-params"
knit_with_optional "width:integer" "800" "Image width in pixels."
knit_with_optional "height:integer" "600" "Image height in pixels."
knit_with_optional "c-re:real" "-0.8" "Real part of the Julia constant c."
knit_with_optional "c-im:real" "0.156" "Imaginary part of the Julia constant c."
knit_with_optional "max-iter:integer" "1000" "Maximum iterations per pixel."
knit_with_optional "colormap:colormap" "fire" "Palette: $(knit_enum_values colormap ", ")."
knit_done
# --- Job --------------------------------------------------------------------
# Submitted to the scheduler. Imports the shared set and keeps its own `output`
# name. Its body launches the render app across one rank per allocated node with
# `knit run`, so the same job runs one rank on a laptop and one rank per node
# across a multi-node allocation.
knit_register_job "julia" _julia "Render a Julia-set fractal as a submitted job."
knit_with_setup "juliaenv"
knit_with_parameter_set "julia-params"
knit_with_optional "output:filename" "fractal.png" "PNG file name, written in the job directory."
_julia() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
local png="${KNIT_JOB_PREFIX%/}/${output}"
printf 'The job is running on hosts: %s\n' \
"$(knit_job_hostnames --separator ', ')"
knit run --procs "$(knit_job_nodecount)" --procs-per-node 1 -- render \
--width "${width}" --height "${height}" \
--c-re "${c_re}" --c-im "${c_im}" --max-iter "${max_iter}" \
--colormap "${colormap}" --output "${png}"
}
knit_done
# --- App --------------------------------------------------------------------
# One MPI-parallel image. Imports the shared set. Rank 0 records the `inside`
# metric and the produced PNG as a checksummed `file` output (an intermediate
# that stays in the job directory, with an `image_checksum` column on this table).
knit_register_app "render" _render_app "Render one MPI-parallel Julia-set image."
knit_with_parameter_set "julia-params"
knit_with_required "output:filename" "Absolute PNG path (the job supplies one per run)."
knit_with_output "inside:integer" "0" "Grid points inside the set (recorded by rank 0)."
knit_with_output "image:file" "" "The PNG this render produced (checksummed intermediate)."
_render_app() {
local width height c_re c_im max_iter colormap output
width=$(knit_get_parameter "width" "$@")
height=$(knit_get_parameter "height" "$@")
c_re=$(knit_get_parameter "c-re" "$@")
c_im=$(knit_get_parameter "c-im" "$@")
max_iter=$(knit_get_parameter "max-iter" "$@")
colormap=$(knit_get_parameter "colormap" "$@")
output=$(knit_get_parameter "output" "$@")
local out
out=$(julia-fractal "${width}" "${height}" "${c_re}" "${c_im}" "${max_iter}" \
"${output}" "${colormap}")
printf '%s\n' "${out}"
local inside
inside=$(sed -n 's/.*inside=\([0-9]*\).*/\1/p' <<< "${out}" | head -1)
knit_output "inside" "${inside}"
knit_output "image" "${output}"
}
knit_done
# --- Artifact kind ----------------------------------------------------------
# A KIND is a semantic label backed by one physical type. `insidecsv` marks a
# file as the inside-metric table, so a consumer can require that kind.
knit_register_artifact "insidecsv:file" "Per-render inside metric, one row per image (CSV)."
# --- Fan-in -----------------------------------------------------------------
# Produces a result instead of just printing one. It records its own row and
# provenance (no @without_provenance), builds a CSV straight from the provenance
# graph with a Cypher query, and declares that CSV as a result ARTIFACT of kind
# `insidecsv`: a row in the artifacts table, a `produced` edge, a content
# checksum, and a path stored relative to the artifacts root.
knit_register "aggregate" _aggregate "Fan-in: tabulate every render into a result CSV."
knit_with_table
knit_with_output_artifact "table:insidecsv" "Per-render inside metric, one row per image (CSV)." --result
_aggregate() {
local out
out="$(knit_artifact_dir)"
mkdir -p "${out}"
knit query graph --format csv --header --exec \
"MATCH (r:run)-[:call]->(img:render)
RETURN img.c_re AS c_re, img.c_im AS c_im, img.inside AS inside" \
> "${out}/inside.csv"
knit_artifact "table" "inside.csv"
printf 'wrote %s\n' "${out}/inside.csv"
}
knit_done
# --- Consumer ---------------------------------------------------------------
# Consumes the artifact `aggregate` produced. @with_input_artifact requires a
# recorded artifact of kind `insidecsv` (the value is its artifacts-relative
# path); --verify-checksum re-hashes the bytes and refuses a table that changed
# since it was produced. Reading it records a `used_by` edge from the artifact to
# this command, mirroring the producer's `produced` edge.
knit_register "report" _report "Read the inside-metric table and name the peak render."
knit_with_input_artifact "table:insidecsv" "Artifacts-relative path of the table to read." --verify-checksum
knit_with_table
knit_with_output "peak:real" "0" "Largest inside metric across the rendered images." --result
_report() {
local csv
csv="$(knit_input_artifact_path "$(knit_get_parameter "table" "$@")")"
local peak
peak=$(tail -n +2 "${csv}" | cut -d, -f3 | sort -n | tail -1)
knit_output "peak" "${peak:-0}"
printf 'peak inside metric: %s\n' "${peak:-0}"
}
knit_done
knit "$@"
The Step 6 sweep is data, not code. Save it beside the script as sweep.json
so the run below can prepare the whole batch in one call:
{
"group": "julia-sweep",
"defaults": { "setup": "mympienv" },
"jobs": [
{ "job": "julia", "args": { "c-re": -0.123, "c-im": 0.745 } },
{ "job": "julia", "args": { "c-re": -1.0, "c-im": 0.0 } },
{ "job": "julia", "args": { "c-re": -0.391, "c-im": -0.587 } },
{ "job": "julia", "args": { "c-re": 0.285, "c-im": 0.535 } },
{ "job": "julia", "args": { "c-re": -0.7, "c-im": 0.0 } },
{ "job": "julia", "args": { "c-re": -1.25, "c-im": 0.0 } },
{ "job": "julia", "args": { "c-re": -0.1, "c-im": 0.651 } }
]
}
To run it from scratch:
$ ./exp.sh bootstrap # on a laptop; add --profile <machine> --allocation <alloc> on a cluster
$ ./exp.sh fetch --name julia_src -- julia_code
$ ./exp.sh setup --name mympienv -- juliaenv --src julia_src
$ ./exp.sh prepare from --file sweep.json
$ while ./exp.sh submit next --group julia-sweep --wait; do :; done
$ ./exp.sh aggregate
$ ./exp.sh report --table inside.csv
fetch acquires the source once and records its commit; setup builds
against that named instance; prepare from builds the whole sweep from the
plan, and the loop over submit next releases the prepared jobs one at a time
until the julia-sweep group is drained; aggregate writes inside.csv
as a result artifact and records its own row; report consumes that artifact
by kind and records a used_by edge back to it. From here the rest of the Part
II workflows apply to this same script: remove what you
no longer need, bundle it to ship, or query
across machines. Compare this file with
Part I’s version to see, in one diff, what the refinements
bought.