Using the Scheduler

Introduction

The scheduler is the Fastpath service that manages access to SUT classes and dispatches queued work onto concrete SUTs as they become available. It supports two main workflows:

  • Plan execution: submit a plan that targets a SUT class and let the scheduler split and run it across all available SUTs.

  • Manual leasing: acquire a SUT from a SUT class for interactive use, then release it when finished.

This guide walks through setting up a small demo scheduler instance, saving the scheduler URL and user token in Fastpath preferences, and then using the scheduler CLI verbs to submit and manage plans and leases.

Note

For low-level integration details and the REST interface itself, see Scheduler REST API.

Start a Demo Scheduler

For a simple demo, use the built-in static SUT provider. Create a file called sutclasses.yaml:

- sutclass: demo
  type: static
  params:
    suts:
      - name: demo-01
        connection:
          method: SSH
          params:
            host: demo-01
            user: fpuser
            keyfile: /path/to/id_ed25519
      - name: demo-02
        connection:
          method: SSH
          params:
            host: demo-02
            user: fpuser
            keyfile: /path/to/id_ed25519

This defines a SUT class called demo containing two static SUTs.

Set FASTPATH_SCHED_ARCHIVE_STORAGE_URL in the dispatcher’s environment to an fsspec URL for automatic archives. For example, use file:///var/lib/fastpath/archive-store for local storage, s3://fastpath-archives for S3, or webdav:// for WebDAV. Backend options use the standard FSSPEC_* environment variables. Set the non-secret FASTPATH_SCHED_ARCHIVE_BASE_URL in the scheduler’s environment to the URL clients use to access stored objects. For example, Artifactory can use:

FASTPATH_SCHED_ARCHIVE_STORAGE_URL=webdav://
FASTPATH_SCHED_ARCHIVE_BASE_URL=https://artifactory.example.com/artifactory/generic/fastpath
FSSPEC_WEBDAV='{"base_url":"https://artifactory.example.com/artifactory/generic/fastpath","headers":{"X-JFrog-Art-Api":"<token>"}}'

S3 uses its standard credential chain; S3-compatible storage can set FSSPEC_S3_ENDPOINT_URL.

Start the scheduler:

fastpath sched run \
  --host 127.0.0.1 \
  --port 5000 \
  --database sqlite:///scheduler.db \
  --storage ./sched_storage \
  --sutclasses ./sutclasses.yaml

If you want scheduled plans to automatically publish into a resultstore, also provide --resultstore:

fastpath sched run \
  --host 127.0.0.1 \
  --port 5000 \
  --database sqlite:///scheduler.db \
  --storage ./sched_storage \
  --sutclasses ./sutclasses.yaml \
  --resultstore sqlite:///scheduler-results.db

On first start the scheduler creates a default admin user and prints its token. The admin user has all capabilities, including admin:

****
Created default admin user 'admin' with token: <token>
****

Keep that token. It is needed to create additional scheduler users.

Save the Scheduler URL and Token

The scheduler CLI verbs take --url and --token options, but it is much more convenient to store them as Fastpath preferences:

fastpath preference set --category sched --name url http://127.0.0.1:5000
fastpath preference set --category sched --name token <admin-token>

Now scheduler commands can be run without repeating either argument; if –url or –token is omitted, they default to the values saved as preferences. For example:

fastpath sched sutclass-list

You can inspect the saved values with:

fastpath preference get --category sched --name url
fastpath preference get --category sched --name token

Create Scheduler Users

Create users with the sched user-create verb. Capabilities are passed as a comma-separated list of lower-case names.

For example, create a user that can execute plans and query scheduler state:

fastpath sched user-create alice --capabilities exec,query

Or create a user that can manually lease SUTs and query scheduler state:

fastpath sched user-create bob --capabilities lease,query

Each command prints the created user’s token. If you want to work as that user, update the saved sched.token preference to the new token (or pass it with –token).

Inspect the Scheduler State

List known SUT classes:

fastpath sched sutclass-list

This shows the SUT class id, name, whether it is enabled, current usage, and creation time.

To report usage over a specific time window, provide --from and/or --to:

fastpath sched sutclass-list --from 2026-03-01
fastpath sched sutclass-list --from 2026-03-01 --to 2026-04-01

Submit a Plan

Plans submitted to the scheduler must target a SUT class rather than a specific SUT. For example:

sut:
  sutclass: demo
swprofiles:
  - name: baseline
    kernel: /path/to/Image
    modules: /path/to/modules.tar.xz
benchmarks:
  - include: speedometer/v2.1.yaml
  - include: mmtests/kernbench.yaml
defaults:
  benchmark:
    warmups: 1
    repeats: 3
    sessions: 3
    timeout: 1h

Submit it to the scheduler:

fastpath sched plan-exec demo-plan.yaml

Optionally set a priority:

fastpath sched plan-exec --priority 8 demo-plan.yaml

If the scheduler was started with --resultstore, scheduled plan results are published there by default. To queue a plan without publishing its results:

fastpath sched plan-exec --no-publish demo-plan.yaml

If the plan references local files (e.g. kernel or modules), the scheduler client automatically uploads them along with the plan payload.

Plans and results are owned by the authenticated scheduler user by default. With the delegate capability, use --delegate to submit a plan as an existing, enabled scheduler user:

fastpath sched plan-exec artifacts/plan.yaml \
  --delegate johdoe01 \
  --url "$FASTPATH_SCHED_URL" \
  --token "$FASTPATH_SCHED_TOKEN"

Plans submitted to the scheduler must not contain a top-level user field; submissions that include one are rejected as invalid.

Track and Manage Plans

List all plans:

fastpath sched plan-list --all-users

Inspect specific plan by id:

fastpath sched plan-list 12

The plan listing shows:

  • the owning user

  • the target SUT class

  • plan priority

  • swprofile names, including nested profiles, in both single-plan and list views

  • progress

  • plan status

  • queued, started, and completed timestamps

  • the configured archive URL

Filter plan listings:

fastpath sched plan-list --states running
fastpath sched plan-list --users alice
fastpath sched plan-list --all-users --all-states

By default, plan-list filters to the authenticated user and to plans in the queued or running states. Use --all-users and --all-states to remove those default filters.

Update the priority of a queued or active plan:

fastpath sched plan-update 12 --priority 3

Cancel a plan:

fastpath sched plan-cancel 12

Cancellation is best-effort. Queued child jobs are cancelled immediately. Already-running child jobs are allowed to finish naturally. The final ZIP job still runs after cancellation.

Automatic Log Archives

Each plan ends with a ZIP job, including plans with failed or canceled benchmarks. It runs after result publication and SUT release.

The ZIP contains the plan directory as-is and is uploaded through fsspec as scheduler-logs/<plan-id>.zip. Local, S3, and WebDAV storage are selected at deployment time. The plan API reports the configured archive URL, but the object may no longer exist if it has been removed to save space.

Download 100.zip using your scheduler token, or choose a filename:

fastpath sched plan-logs 100
fastpath sched plan-logs 100 --output logs.zip

The command fetches the configured archive URL from the scheduler and downloads the stored file through fsspec. Configure the matching FSSPEC_* options in the client’s environment when the archive backend requires authentication.

The plan remains running until its ZIP job finishes. If the ZIP job fails, the plan fails. After upload, the ZIP job deletes the local plan directory and then its local ZIP. A restart during directory deletion reuses the complete local ZIP rather than rebuilding an archive from partially deleted logs.

Until cleanup, job logs remain under <storage>/plans/<plan-id>/jobs/ and the local ZIP remains at <storage>/plans/<plan-id>.zip, where <storage> is the scheduler’s --storage directory. These files are retained if upload fails. Dispatcher diagnostics go to standard error and are separate from the archived job logs; consult the service or container logs for dispatcher errors.

Acquire a Lease

To request a manual lease on a SUT from a SUT class:

fastpath sched lease-acquire demo

The command creates an ephemeral Ed25519 key locally and sends only its public key to the scheduler. The scheduler installs the key on every node assigned to the lease. The private key is stored beneath ~/.fastpath/suts/keys and never leaves the client.

To wait until the lease becomes active and a concrete SUT has been assigned:

fastpath sched lease-acquire demo --wait

The command prints the lease details, including the assigned SUT once it is available.

For automation and CI workflows, use --json to print the lease response in a machine-readable format:

fastpath sched lease-acquire demo --json

The JSON response is an array containing the lease object, including its lease id and current state. Combine it with --wait to wait until the lease leaves the queued state:

fastpath sched lease-acquire demo --wait --json

The command continues to create and manage the ephemeral SSH key and synchronize the local leased-SUT SSH configuration when JSON output is selected.

Manage Leases

List all leases:

fastpath sched lease-list --all-users

Inspect a specific lease by id:

fastpath sched lease-list 7

Filter lease listings:

fastpath sched lease-list --states acquired
fastpath sched lease-list --users bob
fastpath sched lease-list --all-users --all-states

By default, lease-list filters to the authenticated user and to leases in the queued or acquired states. Use --all-users and --all-states to remove those default filters.

Lease information can also be returned as JSON:

fastpath sched lease-list 7 --json
fastpath sched lease-list --states acquired --json

JSON output is always an array of lease objects. When a lease id is provided, the array contains one entry. When leases are listed without an id, the array may contain zero or more entries. Assigned SUT information is included when available.

JSON is written to standard output so that it can be consumed by scripts. Lease synchronization warnings are written to standard error and do not corrupt the JSON output. Avoid using the global --verbose option when parsing JSON, because verbose diagnostic output is also written to standard output.

Release a lease when finished:

fastpath sched lease-release 7

The release command removes the local private key. When lease synchronization removes the leased SUT, it also removes any descendants created through its Fastpath SSH alias from ~/.fastpath/suts/nested.json and the generated SSH configuration, along with their local private keys. Children of other active leases and direct hosts are preserved.