Parameter search

Exploring a declared parameter space: exhaustive sweeps with knit sweep and Bayesian optimization with the knit-ax plugin.

Declare a parameter’s search domain

Attach a value set or range to a parameter with –domain for sweeps and optimization.

APIs: knit_with_optional, knit_with_required, knit_parameter_space, knit_parameter_space_json

A tunable parameter can carry a search domain — the set or range of values a parameter sweep or a Bayesian-optimization run may explore. Declare it with --domain <descriptor> on knit_with_optional / knit_with_required, quoting the descriptor so the shell does not brace-expand it:

Shorthand

# Declare each tunable parameter's search domain with --domain. Quote the
# descriptor: the shell would otherwise brace-expand {a,b,c}. --log marks a real
# range as log-scaled (a hint a Bayesian-optimization proposer reads; a sweep
# ignores it). The domain is validated against the parameter's type the first
# time it is read, not at registration.
@command "train" "Train a model over a search space."
@with_optional "lr:real"        "0.01" "Learning rate." --domain '[1e-5..1.0]' --log
@with_optional "batch:integer"  "32"   "Batch size."    --domain '{16,32,64,128}'
@with_optional "layers:integer" "4"    "Hidden layers." --domain '[2..12]'
@with_optional "data:string"    ""     "Training data." --domain '@glob:data/*.csv'
_train() {
    echo "lr=$(knit_get_parameter lr "$@") batch=$(knit_get_parameter batch "$@")"
}
@done

Long form

# Declare each tunable parameter's search domain with --domain. Quote the
# descriptor: the shell would otherwise brace-expand {a,b,c}. --log marks a real
# range as log-scaled (a hint a Bayesian-optimization proposer reads; a sweep
# ignores it). The domain is validated against the parameter's type the first
# time it is read, not at registration.
knit_register "train" _train "Train a model over a search space."
knit_with_optional "lr:real"        "0.01" "Learning rate." --domain '[1e-5..1.0]' --log
knit_with_optional "batch:integer"  "32"   "Batch size."    --domain '{16,32,64,128}'
knit_with_optional "layers:integer" "4"    "Hidden layers." --domain '[2..12]'
knit_with_optional "data:string"    ""     "Training data." --domain '@glob:data/*.csv'
_train() {
    echo "lr=$(knit_get_parameter lr "$@") batch=$(knit_get_parameter batch "$@")"
}
knit_done

The descriptor grammar is one set of literals: {a,b,c} is a value list, [lo..hi] / [lo..hi:step] a range, @glob:<pattern> a filesystem glob, and @file:<path> one value per line. --log adds a log-scale hint to a real range (a Bayesian-optimization proposer reads it; a sweep ignores it). The domain must match the parameter’s type — a range needs integer or real, a glob needs string, a list suits any type — and it is validated the first time it is read, not at registration, so a type mismatch surfaces at first use.

Discover the declared domains of a command with knit describe --space, which annotates each parameter with its domain (and emits a structured space object in the JSON and YAML formats). A search proposer reads the same domains programmatically through knit_parameter_space (structured, by nameref) and knit_parameter_space_json.

Run a parameter sweep

Run a command, job, or app once per point in a cartesian product of values with knit sweep.

APIs: sweep, submit:drain, knit_with_optional, knit_parameter_space, knit_command_parameters

knit sweep runs a downstream command once per point in the cartesian product of a set of values per parameter. It is the exhaustive, no-model proposer: you write the exact invocation you would run once, and mark any value you want to vary in sweep-syntax. The sweep is opaque to the downstream command — it scans the tokens after -- for sweep-syntax, treats each as an axis, passes everything else through verbatim, and re-emits the reconstructed line once per combination.

A value is a swept axis when it is {a,b,c} (a list), [lo..hi] / [lo..hi:step] (a range), @glob:<pattern> or @file:<path>; quote the braces and brackets so the shell does not expand them. Everything else passes through. The downstream command carries an ordinary job here:

Shorthand

# An ordinary job whose tunable parameter carries a search --domain, so the sweep
# can draw its values from the declaration (--fill domain / --n @domain) as well as
# from an explicit set on the command line (--n '{1,2,4}').
@job "sim" "Run one trivial simulation."
@without_setup
@with_optional "n:integer"      "1" "Number of steps." --domain '{1,2,4}'
@with_table
@with_output   "result:integer" "0" "Twice the step count (a stand-in result)."
_sim() {
    local n
    n=$(knit_get_parameter "n" "$@")
    knit_output "result" "$(( n * 2 ))"
    printf 'sim: n=%s result=%s\n' "${n}" "$(( n * 2 ))"
}
@done

Long form

# An ordinary job whose tunable parameter carries a search --domain, so the sweep
# can draw its values from the declaration (--fill domain / --n @domain) as well as
# from an explicit set on the command line (--n '{1,2,4}').
knit_register_job "sim" _sim "Run one trivial simulation."
knit_without_setup
knit_with_optional "n:integer"      "1" "Number of steps." --domain '{1,2,4}'
knit_with_table
knit_with_output   "result:integer" "0" "Twice the step count (a stand-in result)."
_sim() {
    local n
    n=$(knit_get_parameter "n" "$@")
    knit_output "result" "$(( n * 2 ))"
    printf 'sim: n=%s result=%s\n' "${n}" "$(( n * 2 ))"
}
knit_done

Ephemeral: run the product now. Write the real invocation after -- and mark the swept values. The dispatcher you use (submit, run, or a plain command) selects what each point does; the sweep launches them in-process, keeping at most --max-inflight in flight (default 1):

# a job per point (3 submissions), grouped as a campaign:
$ ./exp.sh sweep -- submit --group mc --nodes {1,2} -- sim --n '{1,2,4}'
# an app per point, inside one allocation:
$ ./exp.sh sweep -- run --procs {1,2,4} -- stencil --tile '{64,128}'
# a plain command per point:
$ ./exp.sh sweep -- sim --n '{1,2,4}'

A --group you put on the submit line tags the jobs the sweep produces, so the campaign’s jobs share a jobs.group you can later drain, inspect, or remove as a unit. The sweep does not forward its own --group onto that line: a --group on sweep labels only the sweeps (parent) row, and the two are kept independent so you can gather jobs from several submits — by hand or under one or more sweeps — under one job-level group. Independently of any group, every point records an automatic call edge back to the sweep’s row, so a query can always walk a campaign to its children. --on-failure stop aborts on the first failed child (continue, the default, runs the whole product). --dry-run prints each reconstructed command line and exits without recording or running — the quickest way to see exactly what a sweep would do.

Vary a declared domain. When a parameter declares a --domain (see Declare a parameter’s search domain), the sweep can draw its values from the declaration instead of the command line: --param @domain opts one parameter in, --fill domain opts in every unspecified parameter with an enumerable domain, and --param @default opts one back out. A real-interval range is bounds-only and cannot be enumerated, so sweeping one is a hard error with a fix message. This is the one point where the sweep introspects the target, through the same accessors a Bayesian-optimization proposer uses (knit_parameter_space / knit_command_parameters):

$ ./exp.sh sweep --fill domain -- submit -- sim          # sim --n over its {1,2,4} domain
$ ./exp.sh sweep -- submit -- sim --n @domain            # same, opting --n in explicitly

Durable: prepare now, drain later (resumable). Swap -- submit for -- prepare and the sweep writes the whole product as a durable to-do list — one prepared job per point, no scheduler contact — then a separate submit drain releases them. This is the form to use for a large or long campaign, or whenever you want to resume after an interruption: a drainer that dies leaves the un-started points still prepared, so re-running drain resumes exactly the remainder (see Drain a prepared batch with submit drain and Retry failed or killed jobs):

# 1. prepare the product once (durable, grouped -- --group on prepare labels
#    the prepared jobs so the drain can select them):
$ ./exp.sh sweep -- prepare --group mc -- sim --n '{1,2,4}'
# 2. drain it -- from the login node, or inside a pilot job:
$ ./exp.sh submit drain --group mc --max-inflight 8 --reconcile --retry killed,failed

Resumability is not a sweep feature at all — it is a property of the prepare/drain queue the sweep writes into. Draining inside a knit_with_pilot job packs the swept jobs onto the pilot’s nodes with no second queue wait; a pilot killed at walltime is resumed by re-submitting it, where --reconcile first corrects the sub-jobs the hard kill stranded at running and --retry re-queues them. The sweep’s own --max-inflight / --on-failure govern only the ephemeral forms; durable concurrency, recovery, and failure policy are drain’s.

Optimize a job with Bayesian search

Tune a job’s recorded output with the knit-ax plugin’s Bayesian ask/tell loop over a declared search domain, or fit a surrogate model and query it with ax predict.

APIs: knit_import, knit_with_optional, knit_with_output, knit_with_table, knit_get_parameter, knit_output

The knit-ax plugin runs Bayesian optimization over a knit command. Where knit sweep tries every point in a product, ax fits a model to the results so far and asks it where to look next — the right proposer when each trial is expensive and the space is large. Import the plugin, then write an ordinary job whose tunable parameter carries a search --domain and whose figure of merit is a declared output:

Shorthand

# An ordinary job with one tunable real knob carrying a search --domain and a
# declared numeric output. Each trial, "ax optimize" proposes an x within the
# domain, runs the job, reads the recorded output, and feeds it back to the
# model. The objective is closed-form here so the example is self-contained; a
# real job would run a simulation and record its figure of merit.
@job "quad" "Evaluate the surrogate objective."
@without_setup
@with_optional "x:real"     "0" "The knob to optimize." --domain '[0..12]'
# @job already declares the job's table (named "quad"); do not add @with_table,
# which would re-point it at the "submit:quad" command name and split recording
# from where the objective is read.
@with_output   "score:real" "0" "Objective = 100 - (x - 6)^2 (max 100 at x = 6)."
_quad() {
    local x score
    x=$(knit_get_parameter "x" "$@")
    score=$(awk -v x="${x}" 'BEGIN { printf "%.6f", 100 - (x - 6) * (x - 6) }')
    knit_output "score" "${score}"
    printf 'quad: x=%s score=%s\n' "${x}" "${score}"
}
@done

Long form

# An ordinary job with one tunable real knob carrying a search --domain and a
# declared numeric output. Each trial, "ax optimize" proposes an x within the
# domain, runs the job, reads the recorded output, and feeds it back to the
# model. The objective is closed-form here so the example is self-contained; a
# real job would run a simulation and record its figure of merit.
knit_register_job "quad" _quad "Evaluate the surrogate objective."
knit_without_setup
knit_with_optional "x:real"     "0" "The knob to optimize." --domain '[0..12]'
# @job already declares the job's table (named "quad"); do not add @with_table,
# which would re-point it at the "submit:quad" command name and split recording
# from where the objective is read.
knit_with_output   "score:real" "0" "Objective = 100 - (x - 6)^2 (max 100 at x = 6)."
_quad() {
    local x score
    x=$(knit_get_parameter "x" "$@")
    score=$(awk -v x="${x}" 'BEGIN { printf "%.6f", 100 - (x - 6) * (x - 6) }')
    knit_output "score" "${score}"
    printf 'quad: x=%s score=%s\n' "${x}" "${score}"
}
knit_done

A run is a campaign: a named, resumable record stored on disk under campaigns/<name>/ (a sibling of setups/ and jobs/, so knit bundle packs it). Two commands drive it, mirroring prepare then submit drain: ax prepare validates the request, resolves the objective, builds the search space, and records the campaign without running a trial; ax optimize starts — or resumes — the ask/tell loop toward the budget. First run and every resume are the same call, so a campaign interrupted at walltime continues from its persisted state:

# define the campaign (runs nothing); the objective is a declared output,
# the varying parameter is drawn from its domain with @domain. --max-trials
# is the campaign's lifetime budget:
$ ./exp.sh ax prepare --campaign tune --objective score --maximize \
      --max-trials 20 --seed 1 -- submit -- quad --x @domain
# drive the loop (re-run to resume); --max-inflight sets this call's concurrency:
$ ./exp.sh ax optimize --campaign tune --max-inflight 4
$ ./exp.sh ax list                  # one line per campaign: status, trials, best
$ ./exp.sh ax show --campaign tune  # one campaign's detail and current optimum
# reached the budget but want to keep searching? raise it, then optimize again:
$ ./exp.sh ax extend --campaign tune --max-trials 40
$ ./exp.sh ax optimize --campaign tune

Each trial is a real knit submit into the campaign’s group, so the trials are ordinary tracked jobs: on ax optimize, --max-inflight keeps that many in flight (per call, default 1) and --trials K runs at most K more this call (default: run to the campaign’s --max-trials); every trial records a call edge back to the campaign’s optimizations row, so a query can walk a campaign to its children. ax reads each trial’s objective from the declared output column, feeds it to the Ax model, and records the best trial on the campaign row and in campaigns/<name>/result.json. --minimize (the default) or --maximize sets the sense; --sobol-trials sets how many quasi-random trials explore before the model engages; --on-failure penalize tells the model a fixed penalty for a failed trial instead of dropping it.

The same campaign optimizes an app inside one allocation: swap -- submit for -- run and give the launch fields, as in -- run --procs 8 -- bench --msg-size @domain. App trials share the allocation, so they run sequentially (--max-inflight is job-mode only), and the objective is read from the app’s own table. Declare a parameter’s domain with --domain (see Declare a parameter’s search domain); ax and knit sweep read the very same domains, so a space written for one proposer serves the other unchanged.

Model the objective instead of optimizing it

Sometimes the goal is not the best point but a model of how the objective behaves across the whole space. Prepare with --model in place of --minimize / --maximize: the campaign then fits a surrogate by active learning, sending each trial where the model is least certain rather than toward an optimum. A model campaign records no best, so ax show lists its varying axes instead:

$ ./exp.sh ax prepare --campaign surf --objective score --model \
      --max-trials 20 -- submit -- quad --x @domain
$ ./exp.sh ax optimize --campaign surf   # spends the budget on active learning
$ ./exp.sh ax show --campaign surf
...
axes:       quad.x

Ask the fitted model for the objective at a point with ax predict. Write the point as the prepare command with a concrete value in each varying position; the value must be in its domain (a member of a {...} set, or inside a [lo..hi] range). It prints the predicted value by default; --variance, --std, --interval (the 95% confidence interval), and --json add detail:

$ ./exp.sh ax predict --campaign surf -- submit -- quad --x 6
93.8
$ ./exp.sh ax predict --campaign surf --std --interval -- submit -- quad --x 6
93.8  (±6.7, 95% CI 80.7–106.9)

ax predict works on any campaign that has run trials — a --model campaign or a --minimize / --maximize one — and reads the model only, recording nothing. ax show --diagnostics cross-validates the fitted surrogate and reports its fit quality (R² / RMSE); it refits the model, so it is opt-in.