Setup

Setup#

Defining and provisioning software environments.

Register a setup#

Register a setup that builds and installs software into its own prefix.

APIs: knit_register_setup, knit_with_spack_specs, KNIT_SETUP_PREFIX

Use knit_register_setup to declare a setup: a reproducible software environment that Knit builds once and reuses. The setup function installs everything it needs under KNIT_SETUP_PREFIX — the private directory Knit creates for it — so commands that depend on the setup find the software on their PATH.

A setup can lean on an environment manager for that software: a plugin that provisions and activates an environment for the setup, so the body does not build every dependency by hand. Spack is the first such manager — import it with knit_import spack — and a setup names its needs through the manager’s directive, generically knit_with_<manager>_env (for Spack, knit_with_spack_env or the knit_with_spack_specs shorthand). The setup below asks Spack for cmake and libpng, then builds and installs the program into KNIT_SETUP_PREFIX; see the Spack recipes for the manager details.

@setup "juliaenv" "Build and install julia-fractal from source."
@with_spack_specs "cmake" "libpng"
@with_optional "ref:string" "v1.1.0" "git ref to build (tag, branch, or commit)."
_juliaenv_setup() {
    # The Spack environment declared above is already built and activated here, so
    # cmake and libpng are on PATH / LD_LIBRARY_PATH. Everything we install goes
    # under KNIT_SETUP_PREFIX --- the private directory Knit created for this setup.
    local ref
    ref="$(knit_get_parameter "ref" "$@")"

    git clone "https://github.com/knit-sh/julia-fractal-example.git" \
        "${KNIT_SETUP_PREFIX}/src"
    git -C "${KNIT_SETUP_PREFIX}/src" checkout "${ref}"

    # Configure, build, and install into the setup prefix. No MPI is present in
    # the environment, so CMake builds the serial binary. RPATH_USE_LINK_PATH
    # bakes the libpng location into the binary so it also runs on a machine with
    # no system libpng.
    cmake -S "${KNIT_SETUP_PREFIX}/src" -B "${KNIT_SETUP_PREFIX}/build" \
        -DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
        -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
    cmake --build "${KNIT_SETUP_PREFIX}/build"
    cmake --install "${KNIT_SETUP_PREFIX}/build"

    # Put the installed binary on the PATH of every command that depends on this
    # setup. knit_setup_env_prepend records a composable line, so each dependent
    # command keeps its own PATH and gains this entry.
    knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
@done

knit_register_setup "juliaenv" _juliaenv_setup "Build and install julia-fractal from source."
@with_spack_specs "cmake" "libpng"
knit_with_optional "ref:string" "v1.1.0" "git ref to build (tag, branch, or commit)."
_juliaenv_setup() {
    # The Spack environment declared above is already built and activated here, so
    # cmake and libpng are on PATH / LD_LIBRARY_PATH. Everything we install goes
    # under KNIT_SETUP_PREFIX --- the private directory Knit created for this setup.
    local ref
    ref="$(knit_get_parameter "ref" "$@")"

    git clone "https://github.com/knit-sh/julia-fractal-example.git" \
        "${KNIT_SETUP_PREFIX}/src"
    git -C "${KNIT_SETUP_PREFIX}/src" checkout "${ref}"

    # Configure, build, and install into the setup prefix. No MPI is present in
    # the environment, so CMake builds the serial binary. RPATH_USE_LINK_PATH
    # bakes the libpng location into the binary so it also runs on a machine with
    # no system libpng.
    cmake -S "${KNIT_SETUP_PREFIX}/src" -B "${KNIT_SETUP_PREFIX}/build" \
        -DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
        -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
    cmake --build "${KNIT_SETUP_PREFIX}/build"
    cmake --install "${KNIT_SETUP_PREFIX}/build"

    # Put the installed binary on the PATH of every command that depends on this
    # setup. knit_setup_env_prepend records a composable line, so each dependent
    # command keeps its own PATH and gains this entry.
    knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
knit_done

Keep setup functions thin#

Delegate dependency installation to a package manager and keep the setup body minimal.

APIs: knit_with_spack_specs, knit_with_spack_env, knit_setup_env_set

A setup body should do as little as possible. Every command it runs by hand — cloning a repository, ./configure, make, downloading a tarball — is one more thing a reproducer has to trust to behave identically on a different machine, and one more thing knit cannot capture as provenance. The more the body does, the less reproducible the setup.

Prefer declaring dependencies with knit_with_spack_specs (or a full knit_with_spack_env manifest) and letting Spack build them. Spack pins versions, compilers, and variants, records a concrete lockfile (captured as a provenance output on the setup’s table), and reuses already-built packages across setups — far more reproducible than an imperative build script. In the ideal case the body is empty: everything the experiment needs is a Spack spec, so the setup is just a list of specs.

When your own program is not yet a Spack package — as in the julia-fractal example, whose setup clones the source and builds it with CMake by hand — the most reproducible path is to write a small Spack package for it (a package.py in a custom repo) and add it to the specs. The hand-written clone/configure/build steps then become one more spec that Spack builds, versions, and locks like any other dependency, and the setup body shrinks to nothing.

Keep in the body only what genuinely cannot be a package: writing a config or params file, declaring an environment variable with knit_setup_env_set, or creating a directory under KNIT_SETUP_PREFIX.

Make a spec depend on a parameter#

Reference a setup parameter from a spec or manifest with a {{placeholder}}.

APIs: knit_with_spack_specs, knit_with_spack_env, knit_with_uv_specs, knit_with_required, knit_with_optional

A setup’s environment can depend on its parameters. Write a {{parameter-name}} placeholder anywhere in a spec (or a manifest) and Knit replaces it, when the setup runs, with the value of that setup parameter — so one setup definition builds whatever version the invocation asks for:

# A "{{parameter}}" placeholder in a spec is replaced, when the setup runs, with
# the value of the named setup parameter. @with_spack_specs takes one or more
# specs, and several calls add up into one spec list.
@setup "mathlibs" "Build math libraries at chosen versions."
@with_required "openblas-version:string" "OpenBLAS version to build."
@with_optional "fftw-spec:string" "fftw" "fftw spec (e.g. 'fftw@3.3.10 +openmp')."
@with_spack_specs "openblas@{{openblas-version}}" "{{fftw-spec}}"
_mathlibs_setup() {
    # openblas and fftw at the chosen versions are on PATH / LD_LIBRARY_PATH here.
    echo "building in ${KNIT_SETUP_PREFIX}"
}
@done

# A "{{parameter}}" placeholder in a spec is replaced, when the setup runs, with
# the value of the named setup parameter. @with_spack_specs takes one or more
# specs, and several calls add up into one spec list.
knit_register_setup "mathlibs" _mathlibs_setup "Build math libraries at chosen versions."
knit_with_required "openblas-version:string" "OpenBLAS version to build."
knit_with_optional "fftw-spec:string" "fftw" "fftw spec (e.g. 'fftw@3.3.10 +openmp')."
@with_spack_specs "openblas@{{openblas-version}}" "{{fftw-spec}}"
_mathlibs_setup() {
    # openblas and fftw at the chosen versions are on PATH / LD_LIBRARY_PATH here.
    echo "building in ${KNIT_SETUP_PREFIX}"
}
knit_done

Invoking knit setup -- mathlibs --openblas-version 0.3.28 renders the spec to openblas@0.3.28 before the environment is built. The rendered (concrete) manifest is what Knit records as provenance, so a reproducer reads openblas@0.3.28, not the template.

knit_with_spack_specs takes one or more specs, and several calls on the same setup add up into one spec list (group them however reads best). A setup can instead declare a complete manifest with knit_with_spack_env; the two forms are mutually exclusive on one setup. To choose specs structurally — one spec per computed item, or a spec added only under a flag — a function can generate them (see the Generate specs or a manifest with a function recipe).

The same {{placeholder}} works wherever an environment comes from — the knit_with_spack_specs sugar above, a here-doc or file manifest passed to knit_with_spack_env, and the matching directives of any other environment manager (for example knit_with_uv_specs "numpy=={{numpy-version}}"). A placeholder whose name does not match a declared parameter is a fatal error when the setup runs, naming the offending {{token}}. The placeholder name follows Knit’s usual rule that hyphens and underscores are interchangeable ({{openblas-version}} is the same as {{openblas_version}}).

Generate specs or a manifest with a function#

Let a function produce a setup’s specs (or whole manifest) at invocation time.

APIs: knit_with_spack_specs_from, knit_with_spack_env_from, knit_with_uv_specs_from, knit_with_uv_env_from, knit_get_parameter, knit_with_flag, knit_with_required

A {{placeholder}} fills a value into a fixed shape. When the shape itself depends on a parameter — add a spec only under a flag, emit one spec per computed item — let a function build it. knit_with_<manager>_specs_from names a function that prints newline-separated specs on stdout when the setup runs:

# A spec-generating function is called when the setup runs, with the setup's
# arguments as "$@"; it reads parameters with knit_get_parameter and prints
# newline-separated specs on stdout. It chooses specs *structurally* --- here a
# spec is added only when the --use-cuda flag is set --- and "{{...}}" still fills
# in values, so the two compose.
mllibs_specs() {
    echo "openblas@{{openblas-version}}"
    [[ "$(knit_get_parameter use-cuda "$@")" == true ]] && echo "py-torch +cuda"
}

@setup "mllibs" "Build an ML environment, optionally with CUDA."
@with_flag     "use-cuda" "Build the CUDA-enabled variant."
@with_required "openblas-version:string" "OpenBLAS version to build."
@with_spack_specs_from mllibs_specs
_mllibs_setup() {
    echo "building in ${KNIT_SETUP_PREFIX}"
}
@done

# A spec-generating function is called when the setup runs, with the setup's
# arguments as "$@"; it reads parameters with knit_get_parameter and prints
# newline-separated specs on stdout. It chooses specs *structurally* --- here a
# spec is added only when the --use-cuda flag is set --- and "{{...}}" still fills
# in values, so the two compose.
mllibs_specs() {
    echo "openblas@{{openblas-version}}"
    [[ "$(knit_get_parameter use-cuda "$@")" == true ]] && echo "py-torch +cuda"
}

knit_register_setup "mllibs" _mllibs_setup "Build an ML environment, optionally with CUDA."
knit_with_flag     "use-cuda" "Build the CUDA-enabled variant."
knit_with_required "openblas-version:string" "OpenBLAS version to build."
@with_spack_specs_from mllibs_specs
_mllibs_setup() {
    echo "building in ${KNIT_SETUP_PREFIX}"
}
knit_done

The function is called with the setup’s arguments as "$@", so it reads parameters with knit_get_parameter exactly as a setup body does (a flag arrives as the string true or false). Each non-blank line it prints is a spec, still rendered for {{...}} — so the function chooses which specs and {{...}} fills in values. The printed specs join the setup’s spec list, so knit_with_spack_specs_from and plain knit_with_spack_specs compose in any order.

To generate a whole manifest (a complete spack.yaml or pyproject.toml) rather than a spec list, use knit_with_<manager>_env_from — the programmatic twin of knit_with_spack_env:

# For a whole document (a complete spack.yaml / pyproject.toml), a function prints
# it on stdout. @with_spack_env_from is the programmatic twin of @with_spack_env;
# "{{...}}" is rendered on the printed document too.
mlenv_manifest() {
    cat <<YAML
spack:
  specs:
    - openblas@{{openblas-version}}
  view: true
  concretizer:
    unify: true
YAML
}

@setup "mlenv" "Build an ML environment from a generated manifest."
@with_required "openblas-version:string" "OpenBLAS version to build."
@with_spack_env_from mlenv_manifest
_mlenv_setup() {
    echo "building in ${KNIT_SETUP_PREFIX}"
}
@done

# For a whole document (a complete spack.yaml / pyproject.toml), a function prints
# it on stdout. @with_spack_env_from is the programmatic twin of @with_spack_env;
# "{{...}}" is rendered on the printed document too.
mlenv_manifest() {
    cat <<YAML
spack:
  specs:
    - openblas@{{openblas-version}}
  view: true
  concretizer:
    unify: true
YAML
}

knit_register_setup "mlenv" _mlenv_setup "Build an ML environment from a generated manifest."
knit_with_required "openblas-version:string" "OpenBLAS version to build."
@with_spack_env_from mlenv_manifest
_mlenv_setup() {
    echo "building in ${KNIT_SETUP_PREFIX}"
}
knit_done

A setup declares either a spec list (any number of ..._specs / ..._specs_from calls, which add up) or one complete manifest (a single ..._env / ..._env_from); the two forms are mutually exclusive on one setup. A function that exits non-zero aborts the setup before anything is built, and an empty final spec list is a fatal error. The _from directives exist for every environment manager (knit_with_uv_specs_from, knit_with_uv_env_from, and so on), each with its @ shorthand.

Declare a setup’s environment#

Declare environment changes a setup passes to dependent commands, composably.

APIs: knit_setup_env_set, knit_setup_env_prepend, knit_setup_env_append, knit_setup_env_unset, knit_setup_activate_line

A setup does not export its environment by taking a snapshot of the build shell. Instead the setup body declares each environment change, and Knit records it as a line in the setup’s .activate.sh. Every command that depends on the setup sources that file, so it runs with exactly what the setup declared — nothing more.

Each function changes the build shell now and records a composable line:

  • knit_setup_env_set VAR value — set a variable.

  • knit_setup_env_prepend VAR entry / knit_setup_env_append VAR entry — add one entry to a colon-separated search path such as PATH. The recorded line keeps the ${VAR} reference literal, so a dependent command extends its own PATH instead of overwriting it.

  • knit_setup_env_unset VAR — remove a variable.

  • knit_setup_activate_line 'line' — record a verbatim line (for example a module load). It runs in declaration order, so it can build on a variable set above it.

@setup "toolenv" "Declare a composable software environment."
_toolenv_setup() {
    # Each call changes THIS build shell now and records one composable line in
    # <setup>/.activate.sh for every dependent command to replay.

    # Set a plain variable.
    knit_setup_env_set TOOL_GREETING "hello"

    # Prepend to PATH. The recorded line keeps ${PATH} literal, so a dependent
    # job adds this entry to its OWN PATH instead of overwriting it.
    knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"

    # Append to a search path. Empty-safe: no leading separator on an empty var.
    knit_setup_env_append TOOL_DATA_PATH "${KNIT_SETUP_PREFIX}/share"

    # Remove a variable from every dependent command's environment.
    knit_setup_env_unset TOOL_LEGACY

    # Record a verbatim line. It runs in declaration order, so it can build on a
    # variable set above.
    knit_setup_activate_line 'export TOOL_BANNER="${TOOL_GREETING}-from-setup"'
}
@done

knit_register_setup "toolenv" _toolenv_setup "Declare a composable software environment."
_toolenv_setup() {
    # Each call changes THIS build shell now and records one composable line in
    # <setup>/.activate.sh for every dependent command to replay.

    # Set a plain variable.
    knit_setup_env_set TOOL_GREETING "hello"

    # Prepend to PATH. The recorded line keeps ${PATH} literal, so a dependent
    # job adds this entry to its OWN PATH instead of overwriting it.
    knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"

    # Append to a search path. Empty-safe: no leading separator on an empty var.
    knit_setup_env_append TOOL_DATA_PATH "${KNIT_SETUP_PREFIX}/share"

    # Remove a variable from every dependent command's environment.
    knit_setup_env_unset TOOL_LEGACY

    # Record a verbatim line. It runs in declaration order, so it can build on a
    # variable set above.
    knit_setup_activate_line 'export TOOL_BANNER="${TOOL_GREETING}-from-setup"'
}
knit_done

A dependent command reads the composed environment with no extra work:

@job "probe" "Run inside the setup and report the composed environment."
@with_setup "toolenv"
@with_output "greeting:string"  "" "TOOL_GREETING declared by the setup."
@with_output "banner:string"    "" "TOOL_BANNER built by the activate line."
@with_output "data_path:string" "" "TOOL_DATA_PATH appended by the setup."
@with_output "legacy:string"    "" "TOOL_LEGACY (unset by the setup; empty here)."
@with_output "path:string"      "" "The job's PATH after activation."
_probe() {
    # The setup's environment is already composed onto this job's own shell: its
    # variables are set and its PATH entry is prepended to the job's own PATH.
    knit_output "greeting"   "${TOOL_GREETING:-<unset>}"
    knit_output "banner"     "${TOOL_BANNER:-<unset>}"
    knit_output "data_path"  "${TOOL_DATA_PATH:-<unset>}"
    knit_output "legacy"     "${TOOL_LEGACY:-<unset>}"
    knit_output "path"       "${PATH}"
}
@done

knit_register_job "probe" _probe "Run inside the setup and report the composed environment."
knit_with_setup "toolenv"
knit_with_output "greeting:string"  "" "TOOL_GREETING declared by the setup."
knit_with_output "banner:string"    "" "TOOL_BANNER built by the activate line."
knit_with_output "data_path:string" "" "TOOL_DATA_PATH appended by the setup."
knit_with_output "legacy:string"    "" "TOOL_LEGACY (unset by the setup; empty here)."
knit_with_output "path:string"      "" "The job's PATH after activation."
_probe() {
    # The setup's environment is already composed onto this job's own shell: its
    # variables are set and its PATH entry is prepended to the job's own PATH.
    knit_output "greeting"   "${TOOL_GREETING:-<unset>}"
    knit_output "banner"     "${TOOL_BANNER:-<unset>}"
    knit_output "data_path"  "${TOOL_DATA_PATH:-<unset>}"
    knit_output "legacy"     "${TOOL_LEGACY:-<unset>}"
    knit_output "path"       "${PATH}"
}
knit_done

Depend on a setup#

Bind a command to a setup so it runs inside that software environment.

APIs: knit_with_setup

Declare knit_with_setup right after registering a command (or job) to bind it to a setup. Before the command’s body runs, Knit activates that setup’s environment — its PATH, libraries, and any Spack packages — so the command finds the software the setup installed.

@with_setup "juliaenv"

knit_with_setup "juliaenv"

Build an MPI environment and provide a launcher#

Ask Spack for an MPI provider in a setup and expose it as the run launcher.

APIs: knit_register_setup, knit_with_spack_specs, knit_provides_launcher

knit_with_spack_specs comes from the Spack plugin — knit_import spack just after source knit.sh makes it available; knit_provides_launcher is part of Knit’s core.

Add mpi (or a concrete provider like mpich or openmpi) to a setup’s knit_with_spack_specs so Spack provisions an MPI implementation as part of the environment. Calling knit_provides_launcher then advertises that Spack-built MPI as the launcher, so knit run can place ranks even on a machine that has no MPI of its own:

# Ask Spack for MPICH and expose it as the launcher. Activating the Spack
# environment (which knit does before the body runs) puts mpicc / mpirun on PATH,
# and @provides_launcher advertises that MPI to "knit run" --- so the body has
# nothing left to build.
@setup "mpienv" "Provide an MPI implementation via Spack."
@with_spack_specs "mpich"
@provides_launcher
_mpienv_setup() {
    : # nothing to do: Spack already installed and activated MPICH
}
@done

# Ask Spack for MPICH and expose it as the launcher. Activating the Spack
# environment (which knit does before the body runs) puts mpicc / mpirun on PATH,
# and @provides_launcher advertises that MPI to "knit run" --- so the body has
# nothing left to build.
knit_register_setup "mpienv" _mpienv_setup "Provide an MPI implementation via Spack."
@with_spack_specs "mpich"
knit_provides_launcher
_mpienv_setup() {
    : # nothing to do: Spack already installed and activated MPICH
}
knit_done

Because knit activates the Spack environment before the setup body runs, the MPI compilers and launcher (mpicc / mpirun) are already on PATH — the body here has nothing left to do. In a real setup this is exactly where the environment pays off: a downstream build (say a CMake find_package(MPI)) now finds the Spack MPI and produces a parallel binary, with no MPI-specific logic in the setup itself.

The provided launcher is detected once at setup-build time and frozen into the setup’s .activate.sh (recorded as the __mpi_launcher__ provenance output). It sits below a machine’s own launcher in precedence, so a profile’s launcher still wins when one exists; it only fills the gap on a machine that offers none.

Provide the default setup#

Understand the builtin default setup that jobs adopt, and how to opt out.

APIs: knit_without_setup, knit_with_setup, knit_register_job

Bootstrap auto-instantiates a builtin default setup that carries only the platform activation (the profile’s modules and environment, if any). A job that declares neither knit_with_setup nor knit_without_setup runs inside this default setup automatically, so it inherits the platform environment with no boilerplate:

# A job that declares NEITHER @with_setup NOR @without_setup adopts the
# builtin "default" setup automatically, inheriting the platform environment.
@job "adopt" "Runs in the builtin default setup."
_adopt() { _report_marker; }
@done

# A job that declares NEITHER @with_setup NOR @without_setup adopts the
# builtin "default" setup automatically, inheriting the platform environment.
knit_register_job "adopt" _adopt "Runs in the builtin default setup."
_adopt() { _report_marker; }
knit_done

To run a job with no setup at all — no setup directory and no platform activation — opt out with knit_without_setup (mutually exclusive with knit_with_setup):

# @without_setup makes a job run with no setup at all: no setup directory and
# no platform activation. It is mutually exclusive with @with_setup.
@job "optout" "Runs with no setup."
@without_setup
_optout() { _report_marker; }
@done

# @without_setup makes a job run with no setup at all: no setup directory and
# no platform activation. It is mutually exclusive with @with_setup.
knit_register_job "optout" _optout "Runs with no setup."
knit_without_setup
_optout() { _report_marker; }
knit_done

The default setup is a setup type like any other, so a command can also require it explicitly by name:

# Any command (not just jobs) can require the default setup explicitly, exactly
# like a named setup type.
@command "report" "Requires the builtin default setup."
@with_setup "default"
_report() { _report_marker; }
@done

# Any command (not just jobs) can require the default setup explicitly, exactly
# like a named setup type.
knit_register "report" _report "Requires the builtin default setup."
knit_with_setup "default"
_report() { _report_marker; }
knit_done

Only jobs adopt the default setup implicitly; a plain command is setup-less unless it declares knit_with_setup, so on a plain command knit_without_setup is a no-op. The default setup is materialized at bootstrap, but you can (re)build a named copy of any setup type with the setup dispatcher:

$ ./exp.sh setup --name default -- default