Spack

Spack#

Working with Spack environments and package specs, via the Spack environment-manager plugin (knit_import spack).

Pin a setup’s Spack specs#

List the Spack packages a setup needs and let knit build them.

APIs: knit_with_spack_specs, knit_register_setup, KNIT_SETUP_PREFIX

Spack is a Knit plugin: import it once with knit_import spack just after source knit.sh, and the knit_with_spack_* setup directives become available.

For the common “just install these packages” case, list the Spack specs a setup needs with knit_with_spack_specs. Knit synthesizes a minimal spack.yaml and, before the setup body runs, installs the specs into a Spack environment and activates it — so the tools are on PATH / LD_LIBRARY_PATH when you build into KNIT_SETUP_PREFIX. The Spack the plugin uses is Knit’s own, installed under .knit/ and isolated from any Spack you have on the machine (see Run Spack directly and Isolate Spack from your personal config); declaring a Spack-backed setup makes bootstrap provision that Spack automatically:

@setup "juliaenv" "Build and install julia-fractal from source."
@with_spack_specs "cmake" "libpng"
@with_optional "ref:string" "v1.1.0" "git ref to build (tag, branch, or commit)."
_juliaenv_setup() {
    # The Spack environment declared above is already built and activated here, so
    # cmake and libpng are on PATH / LD_LIBRARY_PATH. Everything we install goes
    # under KNIT_SETUP_PREFIX --- the private directory Knit created for this setup.
    local ref
    ref="$(knit_get_parameter "ref" "$@")"

    git clone "https://github.com/knit-sh/julia-fractal-example.git" \
        "${KNIT_SETUP_PREFIX}/src"
    git -C "${KNIT_SETUP_PREFIX}/src" checkout "${ref}"

    # Configure, build, and install into the setup prefix. No MPI is present in
    # the environment, so CMake builds the serial binary. RPATH_USE_LINK_PATH
    # bakes the libpng location into the binary so it also runs on a machine with
    # no system libpng.
    cmake -S "${KNIT_SETUP_PREFIX}/src" -B "${KNIT_SETUP_PREFIX}/build" \
        -DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
        -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
    cmake --build "${KNIT_SETUP_PREFIX}/build"
    cmake --install "${KNIT_SETUP_PREFIX}/build"

    # Put the installed binary on the PATH of every command that depends on this
    # setup. knit_setup_env_prepend records a composable line, so each dependent
    # command keeps its own PATH and gains this entry.
    knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
@done

knit_register_setup "juliaenv" _juliaenv_setup "Build and install julia-fractal from source."
@with_spack_specs "cmake" "libpng"
knit_with_optional "ref:string" "v1.1.0" "git ref to build (tag, branch, or commit)."
_juliaenv_setup() {
    # The Spack environment declared above is already built and activated here, so
    # cmake and libpng are on PATH / LD_LIBRARY_PATH. Everything we install goes
    # under KNIT_SETUP_PREFIX --- the private directory Knit created for this setup.
    local ref
    ref="$(knit_get_parameter "ref" "$@")"

    git clone "https://github.com/knit-sh/julia-fractal-example.git" \
        "${KNIT_SETUP_PREFIX}/src"
    git -C "${KNIT_SETUP_PREFIX}/src" checkout "${ref}"

    # Configure, build, and install into the setup prefix. No MPI is present in
    # the environment, so CMake builds the serial binary. RPATH_USE_LINK_PATH
    # bakes the libpng location into the binary so it also runs on a machine with
    # no system libpng.
    cmake -S "${KNIT_SETUP_PREFIX}/src" -B "${KNIT_SETUP_PREFIX}/build" \
        -DCMAKE_INSTALL_PREFIX="${KNIT_SETUP_PREFIX}" \
        -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
    cmake --build "${KNIT_SETUP_PREFIX}/build"
    cmake --install "${KNIT_SETUP_PREFIX}/build"

    # Put the installed binary on the PATH of every command that depends on this
    # setup. knit_setup_env_prepend records a composable line, so each dependent
    # command keeps its own PATH and gains this entry.
    knit_setup_env_prepend PATH "${KNIT_SETUP_PREFIX}/bin"
}
knit_done

Each argument is a full Spack spec, so you can pin versions and variants the same way you would on the spack install command line (for example knit_with_spack_specs "hdf5@1.14 +mpi" "fftw"). The concrete manifest and lockfile are captured as provenance outputs on the setup’s table. For finer control over the environment, use a full manifest with knit_with_spack_env instead.

Make a spec depend on a parameter#

Reference a setup parameter from a spec or manifest with a {{placeholder}}.

APIs: knit_with_spack_specs, knit_with_spack_env, knit_with_uv_specs, knit_with_required, knit_with_optional

A setup’s environment can depend on its parameters. Write a {{parameter-name}} placeholder anywhere in a spec (or a manifest) and Knit replaces it, when the setup runs, with the value of that setup parameter — so one setup definition builds whatever version the invocation asks for:

# A "{{parameter}}" placeholder in a spec is replaced, when the setup runs, with
# the value of the named setup parameter. @with_spack_specs takes one or more
# specs, and several calls add up into one spec list.
@setup "mathlibs" "Build math libraries at chosen versions."
@with_required "openblas-version:string" "OpenBLAS version to build."
@with_optional "fftw-spec:string" "fftw" "fftw spec (e.g. 'fftw@3.3.10 +openmp')."
@with_spack_specs "openblas@{{openblas-version}}" "{{fftw-spec}}"
_mathlibs_setup() {
    # openblas and fftw at the chosen versions are on PATH / LD_LIBRARY_PATH here.
    echo "building in ${KNIT_SETUP_PREFIX}"
}
@done

# A "{{parameter}}" placeholder in a spec is replaced, when the setup runs, with
# the value of the named setup parameter. @with_spack_specs takes one or more
# specs, and several calls add up into one spec list.
knit_register_setup "mathlibs" _mathlibs_setup "Build math libraries at chosen versions."
knit_with_required "openblas-version:string" "OpenBLAS version to build."
knit_with_optional "fftw-spec:string" "fftw" "fftw spec (e.g. 'fftw@3.3.10 +openmp')."
@with_spack_specs "openblas@{{openblas-version}}" "{{fftw-spec}}"
_mathlibs_setup() {
    # openblas and fftw at the chosen versions are on PATH / LD_LIBRARY_PATH here.
    echo "building in ${KNIT_SETUP_PREFIX}"
}
knit_done

Invoking knit setup -- mathlibs --openblas-version 0.3.28 renders the spec to openblas@0.3.28 before the environment is built. The rendered (concrete) manifest is what Knit records as provenance, so a reproducer reads openblas@0.3.28, not the template.

knit_with_spack_specs takes one or more specs, and several calls on the same setup add up into one spec list (group them however reads best). A setup can instead declare a complete manifest with knit_with_spack_env; the two forms are mutually exclusive on one setup. To choose specs structurally — one spec per computed item, or a spec added only under a flag — a function can generate them (see the Generate specs or a manifest with a function recipe).

The same {{placeholder}} works wherever an environment comes from — the knit_with_spack_specs sugar above, a here-doc or file manifest passed to knit_with_spack_env, and the matching directives of any other environment manager (for example knit_with_uv_specs "numpy=={{numpy-version}}"). A placeholder whose name does not match a declared parameter is a fatal error when the setup runs, naming the offending {{token}}. The placeholder name follows Knit’s usual rule that hyphens and underscores are interchangeable ({{openblas-version}} is the same as {{openblas_version}}).

Generate specs or a manifest with a function#

Let a function produce a setup’s specs (or whole manifest) at invocation time.

APIs: knit_with_spack_specs_from, knit_with_spack_env_from, knit_with_uv_specs_from, knit_with_uv_env_from, knit_get_parameter, knit_with_flag, knit_with_required

A {{placeholder}} fills a value into a fixed shape. When the shape itself depends on a parameter — add a spec only under a flag, emit one spec per computed item — let a function build it. knit_with_<manager>_specs_from names a function that prints newline-separated specs on stdout when the setup runs:

# A spec-generating function is called when the setup runs, with the setup's
# arguments as "$@"; it reads parameters with knit_get_parameter and prints
# newline-separated specs on stdout. It chooses specs *structurally* --- here a
# spec is added only when the --use-cuda flag is set --- and "{{...}}" still fills
# in values, so the two compose.
mllibs_specs() {
    echo "openblas@{{openblas-version}}"
    [[ "$(knit_get_parameter use-cuda "$@")" == true ]] && echo "py-torch +cuda"
}

@setup "mllibs" "Build an ML environment, optionally with CUDA."
@with_flag     "use-cuda" "Build the CUDA-enabled variant."
@with_required "openblas-version:string" "OpenBLAS version to build."
@with_spack_specs_from mllibs_specs
_mllibs_setup() {
    echo "building in ${KNIT_SETUP_PREFIX}"
}
@done

# A spec-generating function is called when the setup runs, with the setup's
# arguments as "$@"; it reads parameters with knit_get_parameter and prints
# newline-separated specs on stdout. It chooses specs *structurally* --- here a
# spec is added only when the --use-cuda flag is set --- and "{{...}}" still fills
# in values, so the two compose.
mllibs_specs() {
    echo "openblas@{{openblas-version}}"
    [[ "$(knit_get_parameter use-cuda "$@")" == true ]] && echo "py-torch +cuda"
}

knit_register_setup "mllibs" _mllibs_setup "Build an ML environment, optionally with CUDA."
knit_with_flag     "use-cuda" "Build the CUDA-enabled variant."
knit_with_required "openblas-version:string" "OpenBLAS version to build."
@with_spack_specs_from mllibs_specs
_mllibs_setup() {
    echo "building in ${KNIT_SETUP_PREFIX}"
}
knit_done

The function is called with the setup’s arguments as "$@", so it reads parameters with knit_get_parameter exactly as a setup body does (a flag arrives as the string true or false). Each non-blank line it prints is a spec, still rendered for {{...}} — so the function chooses which specs and {{...}} fills in values. The printed specs join the setup’s spec list, so knit_with_spack_specs_from and plain knit_with_spack_specs compose in any order.

To generate a whole manifest (a complete spack.yaml or pyproject.toml) rather than a spec list, use knit_with_<manager>_env_from — the programmatic twin of knit_with_spack_env:

# For a whole document (a complete spack.yaml / pyproject.toml), a function prints
# it on stdout. @with_spack_env_from is the programmatic twin of @with_spack_env;
# "{{...}}" is rendered on the printed document too.
mlenv_manifest() {
    cat <<YAML
spack:
  specs:
    - openblas@{{openblas-version}}
  view: true
  concretizer:
    unify: true
YAML
}

@setup "mlenv" "Build an ML environment from a generated manifest."
@with_required "openblas-version:string" "OpenBLAS version to build."
@with_spack_env_from mlenv_manifest
_mlenv_setup() {
    echo "building in ${KNIT_SETUP_PREFIX}"
}
@done

# For a whole document (a complete spack.yaml / pyproject.toml), a function prints
# it on stdout. @with_spack_env_from is the programmatic twin of @with_spack_env;
# "{{...}}" is rendered on the printed document too.
mlenv_manifest() {
    cat <<YAML
spack:
  specs:
    - openblas@{{openblas-version}}
  view: true
  concretizer:
    unify: true
YAML
}

knit_register_setup "mlenv" _mlenv_setup "Build an ML environment from a generated manifest."
knit_with_required "openblas-version:string" "OpenBLAS version to build."
@with_spack_env_from mlenv_manifest
_mlenv_setup() {
    echo "building in ${KNIT_SETUP_PREFIX}"
}
knit_done

A setup declares either a spec list (any number of ..._specs / ..._specs_from calls, which add up) or one complete manifest (a single ..._env / ..._env_from); the two forms are mutually exclusive on one setup. A function that exits non-zero aborts the setup before anything is built, and an empty final spec list is a fatal error. The _from directives exist for every environment manager (knit_with_uv_specs_from, knit_with_uv_env_from, and so on), each with its @ shorthand.

Use a full Spack manifest#

Describe a setup’s Spack environment with a complete spack.yaml.

APIs: knit_with_spack_env, knit_register_setup

Spack is a Knit plugin: import it once with knit_import spack just after source knit.sh, and the knit_with_spack_* setup directives become available.

When you need full control over the environment — versions, variants, compilers, views, package preferences — describe it with a complete spack.yaml manifest via knit_with_spack_env. Point it at a file checked in next to the experiment:

# Point @with_spack_env at a spack.yaml checked in next to the experiment.
# The manifest gives full control (versions, variants, compilers, views).
@setup "libs" "Build dependencies from a spack.yaml."
@with_spack_env "spack.yaml"
_libs_setup() {
    # cmake/hdf5/... from the manifest are on PATH here; build into the prefix.
    echo "building in ${KNIT_SETUP_PREFIX}"
}
@done

# Point @with_spack_env at a spack.yaml checked in next to the experiment.
# The manifest gives full control (versions, variants, compilers, views).
knit_register_setup "libs" _libs_setup "Build dependencies from a spack.yaml."
@with_spack_env "spack.yaml"
_libs_setup() {
    # cmake/hdf5/... from the manifest are on PATH here; build into the prefix.
    echo "building in ${KNIT_SETUP_PREFIX}"
}
knit_done

Or, when the manifest is short and specific to the experiment, inline it with a here-doc (any redirected stdin works). The manifest is read at registration time, so it must actually be redirected — an interactive terminal or an empty manifest is rejected rather than left to block:

# Or inline the manifest with a here-doc when it is short and experiment-specific.
@setup "libs-inline" "Build deps from an inline manifest."
@with_spack_env <<'EOF'
spack:
  specs:
    - hdf5@1.14 +mpi
    - fftw
  view: true
EOF
_libs_inline_setup() {
    echo "building in ${KNIT_SETUP_PREFIX}"
}
@done

# Or inline the manifest with a here-doc when it is short and experiment-specific.
knit_register_setup "libs-inline" _libs_inline_setup "Build deps from an inline manifest."
@with_spack_env <<'EOF'
spack:
  specs:
    - hdf5@1.14 +mpi
    - fftw
  view: true
EOF
_libs_inline_setup() {
    echo "building in ${KNIT_SETUP_PREFIX}"
}
knit_done

A setup may declare at most one Spack environment, and knit_with_spack_env is mutually exclusive with the knit_with_spack_specs sugar (which funnels through it). As with specs, the concrete manifest and lockfile are captured as provenance outputs on the setup’s table. The Spack itself is the plugin’s own, installed under .knit/ at bootstrap time — declaring any Spack-backed setup makes bootstrap detect that the experiment needs Spack and provision it, and the plugin’s bootstrap --spack option provisions (and pins) it explicitly.

Build an MPI environment and provide a launcher#

Ask Spack for an MPI provider in a setup and expose it as the run launcher.

APIs: knit_register_setup, knit_with_spack_specs, knit_provides_launcher

knit_with_spack_specs comes from the Spack plugin — knit_import spack just after source knit.sh makes it available; knit_provides_launcher is part of Knit’s core.

Add mpi (or a concrete provider like mpich or openmpi) to a setup’s knit_with_spack_specs so Spack provisions an MPI implementation as part of the environment. Calling knit_provides_launcher then advertises that Spack-built MPI as the launcher, so knit run can place ranks even on a machine that has no MPI of its own:

# Ask Spack for MPICH and expose it as the launcher. Activating the Spack
# environment (which knit does before the body runs) puts mpicc / mpirun on PATH,
# and @provides_launcher advertises that MPI to "knit run" --- so the body has
# nothing left to build.
@setup "mpienv" "Provide an MPI implementation via Spack."
@with_spack_specs "mpich"
@provides_launcher
_mpienv_setup() {
    : # nothing to do: Spack already installed and activated MPICH
}
@done

# Ask Spack for MPICH and expose it as the launcher. Activating the Spack
# environment (which knit does before the body runs) puts mpicc / mpirun on PATH,
# and @provides_launcher advertises that MPI to "knit run" --- so the body has
# nothing left to build.
knit_register_setup "mpienv" _mpienv_setup "Provide an MPI implementation via Spack."
@with_spack_specs "mpich"
knit_provides_launcher
_mpienv_setup() {
    : # nothing to do: Spack already installed and activated MPICH
}
knit_done

Because knit activates the Spack environment before the setup body runs, the MPI compilers and launcher (mpicc / mpirun) are already on PATH — the body here has nothing left to do. In a real setup this is exactly where the environment pays off: a downstream build (say a CMake find_package(MPI)) now finds the Spack MPI and produces a parallel binary, with no MPI-specific logic in the setup itself.

The provided launcher is detected once at setup-build time and frozen into the setup’s .activate.sh (recorded as the __mpi_launcher__ provenance output). It sits below a machine’s own launcher in precedence, so a profile’s launcher still wins when one exists; it only fills the gap on a machine that offers none.

Run Spack directly#

Drive the experiment’s private Spack with the knit spack wrapper.

APIs: spack, bootstrap

The spack command comes from the Spack plugin — knit_import spack just after source knit.sh adds it. It is a thin wrapper that forwards every argument verbatim to the experiment’s private Spack, so you can inspect and drive it exactly as you would a system Spack — without it leaking onto your PATH:

$ ./exp.sh spack find            # what is installed
$ ./exp.sh spack spec zlib       # how a spec would concretize
$ ./exp.sh spack --help          # Spack's own help, forwarded

Because the wrapper forwards --help too, ./exp.sh spack --help shows Spack’s help rather than knit’s. The private Spack is provisioned automatically the first time a setup declares a Spack environment; run knit spack before Spack is provisioned and it fails with a hint to run bootstrap --spack first. To provision it eagerly and pin its version at bootstrap, see the Pin the Spack version recipe in the Bootstrap category.

Isolate Spack from your personal config#

Knit’s Spack ignores ~/.spack and site config so the environment stays reproducible.

APIs: spack

The Spack the plugin provisions (after knit_import spack) is deliberately walled off from your personal and site-wide Spack configuration, so an experiment concretizes the same way no matter whose machine or account it runs on. The plugin exports two environment variables for its Spack:

  • SPACK_DISABLE_LOCAL_CONFIG=true — Spack ignores both the system scope and your ~/.spack user scope.

  • SPACK_USER_CONFIG_PATH pointing at a directory inside the experiment (.knit/.spack) — any user-scope config Spack reads or writes lives with the experiment, not in your home directory.

The upshot is that what the experiment builds is determined only by the specs/manifest in the script plus knit’s own config under .knit — both of which travel with the experiment — so a stray package preference, mirror, or compiler default in ~/.spack can neither leak into nor silently change the build. To customize the environment, put the settings in the manifest (see Use a full Spack manifest) or edit knit’s own scope with knit spack config, rather than in ~/.spack.

Pin the Spack version#

Provision a specific Spack (and package repo) git ref at bootstrap.

APIs: bootstrap

--spack and --spack-packages are added to bootstrap by the Spack plugin, so they appear once the experiment imports it with knit_import spack (just after source knit.sh).

The plugin provisions a private Spack automatically the first time a setup declares a Spack environment, defaulting to the latest Spack release. Pass --spack to pin a specific Spack git ref (a tag, branch, or commit) instead, and --spack-packages to pin the package repository ref:

$ ./exp.sh bootstrap \
    --spack v0.22.1 \
    --spack-packages v2024.11.0

Both take a git ref as their value; an empty value (the default) uses the latest release. Passing --spack also forces Spack to be provisioned now rather than lazily on first use.