AI

AI#

Configuring an AI provider and asking natural-language questions about the experiment.

Configure the AI provider#

Point knit at an OpenAI-compatible AI provider with bootstrap –ai- — storing only env-var names, never the key.*

APIs: bootstrap

The ai commands talk to an OpenAI-compatible chat API. Before using them you tell knit which env vars hold your provider’s credentials and settings. You configure this at bootstrap with the --ai-* options, which write the ai.* metadata:

$ ./exp.sh bootstrap --ai-api-key-env OPENAI_API_KEY --ai-model gpt-4o

Only env-var names and non-secret defaults are stored — the API key itself never reaches the database. At call time knit reads the value of the env var you named, so the secret lives only in your shell:

$ export OPENAI_API_KEY=sk-...
$ ./exp.sh ai ask --question "how many jobs completed?"

--ai-api-key-env is the only required setting. The rest tune where the request goes and which model answers, each with an env-var form (read at call time) and a literal fallback (baked into metadata):

  • --ai-base-url-env / --ai-base-url — the endpoint. The env var wins if set, then the literal, then https://api.openai.com/v1. Point these at any OpenAI-compatible gateway (Azure, a local server, a proxy).

  • --ai-model-env / --ai-model — the default model. A per-call --model on ai ask / ai query overrides both.

Re-running bootstrap updates only the ai.* options you type, so one field (say the model) can change without clearing the rest:

$ ./exp.sh bootstrap --ai-model gpt-4o-mini

If the named key env var is empty or unset when you run an ai command, knit fails with a clear message rather than calling out with no credentials. Like the rest of the ai group, configuring the provider needs a bootstrapped experiment. Once configured, ask a question (Ask a natural-language question) or generate a query (Answer with generated SQL).

Ask a natural-language question#

Ask ai ask a plain-English question; the model answers by calling knit’s read-only introspection tools — it can inspect, never mutate.

APIs: ai:ask

Once a provider is configured (Configure the AI provider), ai ask answers a plain-English question about this experiment. The model runs an agentic loop, answering by calling a fixed set of read-only knit commands exposed to it as tools — it can describe the command tree, read a command’s --help, run a read-only SQL query over the recorded run/job tables, read a job’s captured output (job show), and show metadata (metadata show):

$ export OPENAI_API_KEY=sk-...
$ ./exp.sh ai ask --question "which job used the most nodes, and did it finish?"

The tools are real knit commands — a recordable one records exactly as if you had run it — but the set is deliberately limited to read-only commands: there is no arbitrary-command tool, so ai ask can inspect the experiment but cannot run a command that mutates state. Its answer is grounded in what those tools return, not invented — the system prompt seeds it with a summary of your commands and tells it to call a tool when unsure.

A few options tune the call:

  • --model overrides the configured default model for this one question.

  • --max-iterations caps the agentic tool-call rounds (default 8).

  • --system replaces the built-in system prompt wholesale — use it to change the assistant’s persona or house rules.

  • --verbose streams each tool call and result to stderr, so you can watch the model’s reasoning; --raw prints the raw final-message JSON instead of just the answer text.

ai ask reasons over several tools and replies in prose. When you want tabular data back from a single query instead, use Answer with generated SQL.

Answer with a generated query#

Turn a question into one read-only SQL or Cypher query with ai query — the model writes it, knit runs it read-only and self-corrects on error.

APIs: ai:query

ai query answers a question by translating it into exactly one read-only query, running it against the experiment, and printing the result. It works over any experiment that records runs — a command with an output and a table:

@command "montecarlo" "Estimate pi and record the run."
@with_required "samples:integer" "How many samples to draw."
@with_optional "seed:integer" "1" "Random seed."
@with_output "pi:real" "0" "The estimate produced by this run."
@with_table
_montecarlo() {
    local samples seed estimate
    samples="$(knit_get_parameter "samples" "$@")"
    seed="$(knit_get_parameter "seed" "$@")"
    # A deterministic stand-in for a real estimate: enough to populate the
    # `montecarlo` table (samples, seed, pi) that `ai query` writes SQL against.
    # seed 1 gives exactly 3.14159; other seeds nudge it so runs differ.
    estimate="$(awk -v d="${seed}" \
        'BEGIN { printf "%.5f", 3.14159 + (d - 1) / 100000 }')"
    knit_output "pi" "${estimate}"
    printf 'samples=%s pi=%s\n' "${samples}" "${estimate}"
}
@done

knit_register "montecarlo" _montecarlo "Estimate pi and record the run."
knit_with_required "samples:integer" "How many samples to draw."
knit_with_optional "seed:integer" "1" "Random seed."
knit_with_output "pi:real" "0" "The estimate produced by this run."
knit_with_table
_montecarlo() {
    local samples seed estimate
    samples="$(knit_get_parameter "samples" "$@")"
    seed="$(knit_get_parameter "seed" "$@")"
    # A deterministic stand-in for a real estimate: enough to populate the
    # `montecarlo` table (samples, seed, pi) that `ai query` writes SQL against.
    # seed 1 gives exactly 3.14159; other seeds nudge it so runs differ.
    estimate="$(awk -v d="${seed}" \
        'BEGIN { printf "%.5f", 3.14159 + (d - 1) / 100000 }')"
    knit_output "pi" "${estimate}"
    printf 'samples=%s pi=%s\n' "${samples}" "${estimate}"
}
knit_done

The model is seeded with the live schema and the provenance name map (see Discover the schema), so it writes against your actual tables, columns, and edges:

$ export OPENAI_API_KEY=sk-...
$ ./exp.sh ai query --question "average pi per seed"

SQL or Cypher — the model chooses. By default (--lang auto) the model picks the language that fits the question: SQL for filtering, aggregation, and sorting within a table; Cypher (transpiled to SQL by the bundled knit-cypher-to-sql) for relationships across commands — which command called which, or which setup a job used:

$ ./exp.sh ai query --question "which setup did the montecarlo job use"

Pin the language with --lang sql or --lang cypher when you already know which you want; that also narrows the system prompt to a single language so the model is not tempted by the other. Whichever language is used, the generated statement passes through the same read-only guard as query sql / query graph — it must start with a read clause (SELECT/WITH/EXPLAIN/ PRAGMA for SQL; MATCH/OPTIONAL MATCH for Cypher) and contain no write keyword. If a statement is rejected or the backend reports an error, knit feeds the error back to the model and it tries again, up to --max-iterations rounds (default 3).

To review the query before trusting it, ask for it without running it — --query-only prints the generated statement and its detected language:

$ ./exp.sh ai query --query-only --question "average pi per seed"
SELECT seed, avg(pi) FROM montecarlo GROUP BY seed

The output is shaped by the shared query options:

  • --format picks the mode from the query_format set: box (the default here), column, csv, json, line, list, markdown, table, html, ascii, tabs — and it applies identically whether the answer came back as SQL or Cypher.

  • --no-header omits the header row (headers are on by default here, unlike query sql); --separator sets the column separator for csv/list.

  • --model overrides the model for this call and --verbose streams the chosen language, each generated statement, and any backend error to stderr.

Use ai query when you want data back and would rather not hand-write the query; reach for Run raw SQL or Query the provenance graph when you already know it, and Ask a natural-language question when you want a reasoned prose answer over several read-only tools.