Resources

Fetching input artifacts (datasets, source code) and declaring them as command dependencies.

Register a resource type

Declare how to acquire an input artifact with knit_register_resource and a download decorator.

APIs: knit_register_resource, knit_with_local, knit_with_git, knit_with_url

A resource is a named, downloadable input artifact — a dataset or a piece of third-party source code. knit_register_resource declares a resource type: what the artifact is and, through exactly one download decorator, how to acquire it. There is no body to write; knit supplies the download itself.

Shorthand

# A resource TYPE declares HOW to acquire an artifact --- exactly one download
# decorator selects the backend. The local backend links (or, with --copy, snapshots)
# a path already on disk; swap in @with_git <url> <ref> or @with_url <url>
# to download instead. There is no body: knit supplies the download itself.
@resource "dataset" "An input dataset staged from a local path."
@with_local "./data"
@done

Long form

# A resource TYPE declares HOW to acquire an artifact --- exactly one download
# decorator selects the backend. The local backend links (or, with --copy, snapshots)
# a path already on disk; swap in @with_git <url> <ref> or @with_url <url>
# to download instead. There is no body: knit supplies the download itself.
knit_register_resource "dataset" "An input dataset staged from a local path."
knit_with_local "./data"
knit_done

Pick the decorator that matches the source: knit_with_local <path> links or copies a path already on disk, knit_with_git <url> <ref> clones a repository at a ref, and knit_with_url <url> downloads (and optionally uncompresses) an archive. Add knit_with_checksum <sha256> to pin the artifact’s integrity.

Fetch a resource instance

Acquire a named instance of a resource type with the knit fetch dispatcher.

APIs: fetch

knit fetch acquires a named instance of a registered resource type. Like knit setup, it is a dispatcher: options before -- configure the fetch, and everything after -- selects the resource type and its own arguments.

$ ./exp.sh fetch --name sample -- dataset --path ./data
/path/to/experiment/resources/sample

The instance lands at resources/<name> and is recorded in the type’s table for provenance. Fetching is idempotent by name: re-fetching sample from the same source does nothing, while a different source under the same name is refused. A downloaded or copied instance is made read-only so a shared input cannot be mutated. knit fetch prints the instance path on stdout (logging is on stderr), but commands normally consume a resource by name (see Consume a resource in a command).

Consume a resource in a command

Declare a resource dependency with knit_with_resource and resolve it with knit_resource_path.

APIs: knit_with_resource, knit_resource_path

A command declares the resources it needs with knit_with_resource "<param>:<type>". The value the user passes is the instance name; knit checks that the named instance exists and is of the declared type before the body runs, then records a used_by provenance edge from the resource to the command. Inside the body, knit_resource_path turns the name into an on-disk path.

Shorthand

# A command declares the resources it needs with @with_resource
# "<param>:<type>". The value the user passes is the instance NAME, not a path;
# knit validates that the named instance exists and is of the right type before the
# body runs, then knit_resource_path turns the name into an on-disk directory.
@command "summarize" "Summarize a fetched dataset."
@with_resource "data:dataset" "Name of the dataset instance to read."
@with_table
@with_output "lines:integer" "0" "Number of lines in the dataset."
_summarize() {
    local dir
    dir="$(knit_resource_path "$(knit_get_parameter data "$@")")"

    local n
    n=$(wc -l < "${dir}/values.txt")
    knit_output "lines" "${n}"
    printf 'dataset %s has %s line(s)\n' "${dir}" "${n}"
}
@done

Long form

# A command declares the resources it needs with @with_resource
# "<param>:<type>". The value the user passes is the instance NAME, not a path;
# knit validates that the named instance exists and is of the right type before the
# body runs, then knit_resource_path turns the name into an on-disk directory.
knit_register "summarize" _summarize "Summarize a fetched dataset."
knit_with_resource "data:dataset" "Name of the dataset instance to read."
knit_with_table
knit_with_output "lines:integer" "0" "Number of lines in the dataset."
_summarize() {
    local dir
    dir="$(knit_resource_path "$(knit_get_parameter data "$@")")"

    local n
    n=$(wc -l < "${dir}/values.txt")
    knit_output "lines" "${n}"
    printf 'dataset %s has %s line(s)\n' "${dir}" "${n}"
}
knit_done

Any command can depend on a resource — including a setup, which is the usual place to process a fetched artifact (build fetched source, index a dataset). The user supplies the instance name on the command line, e.g. ./exp.sh summarize --data sample.