Plan Schema

Top-Level plan object

A plan describes the set of benchmarks which should be run against a set of swprofiles (which includes a specific kernel) on a root SUT. A swprofile may optionally create nested SUTs as KVM virtual machines. The following is the set of top-level keys that may be defined in a plan:

key	type	required	default	description
user	string	false	None	Optional user tag to attach to generated results.
sut	dict	true	N/A	Encapsulates all required information about the system under test (SUT). See “sut object”.
swprofiles	list	true	N/A	A list of swprofile objects, each of which describes a software profile that the SUT will be configured into, and for which the benchmarks will be run.
benchmarks	list	true	N/A	A list of benchmark objects, each describing a benchmark to run for each required software profile.
defaults	dict	false	default defaults object	Default values for various parameters. See “defaults object”.

Plan validation enforces the resultstore’s string length limits. SUT class, SUT, node, and software profile names, benchmark suite, name, and type, benchmark parameter names, and role names may contain at most 64 characters. The top-level user tag and benchmark image reference may contain at most 255 characters. A supplied Git SHA must contain exactly 40 hexadecimal characters. Benchmark limits apply after includes and overrides have been merged. Values that exceed these limits are rejected with the field path and allowed length.

sut object

The sut dictionary describes The system under test (SUT):

key	type	required	default	description
sutclass	string	false	None	Name of sutclass to which the SUT belongs.
name	string	false	None	User-supplied friendly name to identify the system under test.
connection	dict	1 from 2	N/A	For a single-node SUT, describes how to connect to the node. See “node.connection object”. Exactly one of “connection” and “nodes” is required.
nodes	list	1 from 2	N/A	A list of nodes objects, each of which describes a node (machine/device) from which the SUT is composed. Exactly one of “connection” and “nodes” is required.

node object

A node is a machine/device that runs a single instance of Linux. A SUT is composed of one or mode nodes. It contains the following keys:

key	type	required	default	description
name	string	false	None	User-supplied friendly name to identify the node. Must be unique amongst nodes.
connection	dict	true	N/A	Describes how to connect to the node. See “node.connection object”.

node.connection object

The connection dictionary describes how to connect to the node that the benchmarks will run on. It contains the following keys:

key	type	required	default	description
method	enum	true	N/A	Method used to connect to the node. For now, only “SSH” is supported. In future, “LAVA” will be added.
params	dict	true	N/A	Method-specific dictionary of parameters. See “SSH-params” below.

node.connection.SSH-params object

The SSH-params dictionary describes how to connect to a node using the “SSH” method. It contains the following keys:

key	type	required	default	description
host	string	true	N/A	Host name or IP address of the connection, or name of Host in SSH config file.
user	string	false	None	Login user for the remote connection. When None, SSH uses its default configured user, which may be specified in the SSH config file if host is the name of a Host in the SSH config file.
port	int	false	None	Remote port to connect to. When None, SSH uses its default configured port, which may be specified in the SSH config file if host is the name of a Host in the SSH config file.
keyfile	string	false	None	Path of private key to use for connection. When None, SSH uses its default configured private key(s).

swprofile object

The top-level swprofiles key maps a list of swprofile dictionaries. Each swprofile represents a software profile that the benchmarks are run against. It contains the following keys:

key	type	required	default	description
name	string	false	None	User-supplied friendly name to identify the software profile.
pkgtype	enum	false	Inferred	Describes what “kernel” parameter points to. Either “RAW” (if a raw kernel Image) or “DEB” (if a Debian package). If omitted, value is inferred; “DEB” if kernel string ends with “.deb”, or “RAW” otherwise.
kernel	string	true	N/A	The kernel package to be used by the SUT. Installation, reboot and uninstallation is managed by the automatically. May be a filesystem path or a URL.
modules	string	false	None	When pkgtype is RAW, modules to install, provided as tarball. Must be None for other pkgtypes. May be a filesystem path or a URL.
cmdline	list	false	[]	List of additional command line options to be appended to the kernel’s command line. The list is joined with a space (” “) separator.
sysctl	list	false	[]	List of sysctls to be applied persistently prior to reboot. Then reverted after reboot to remove persistence.
bootscript	list	false	[]	Bash commands to be executed after reboot with the correct kernel.
gitsha	string	false	None	Full, 40 character Git SHA for the revision of the source code used to build the provided kernel, or None if not known.
suts	list	false	[]	Nested SUTs to create after configuring this profile. Benchmarks are run concurrently only on leaf SUTs.

nested sut object

Each entry in swprofile.suts creates one single-node VM on the SUT running that profile. Nested SUT declarations are recursive and contain the following keys:

key	type	required	default	description
sutclass	string	false	None	Name of sutclass to which the SUT belongs.
image	string	false	registry.gitlab.arm.com/tooling/fastpath/containers/vm:latest	Docker image containing QEMU and all assets required to create the VM.
params	dict	false	{}	QEMU parameters used to create the VM.
swprofile	dict	true	N/A	The single software profile configured in the child. It may declare further nested SUTs.

A SUT which hosts nested SUTs must have exactly one node. Nested children are also single-node SUTs. Connection details, generated names, forwarded ports, and SSH credentials are managed by plan exec and cannot be specified in a nested SUT declaration.

image is the Docker image in which Fastpath launches QEMU. The current QemuVm implementation expects the image to provide:

qemu-system-aarch64 with Arm KVM virtual EL2 support and cloud-localds on PATH, together with the standard shell tools used to prepare cloud-init data and monitor QEMU;
/vm/AAVMF_CODE.fd and /vm/AAVMF_VARS.fd as the guest EFI firmware;
/vm/noble-server-cloudimg-arm64.img as the writable guest root filesystem image; and
a writable /vm directory in which Fastpath creates the cloud-init seed, PID file, and QEMU log.

Fastpath overrides the container entry point and runs the image as a privileged container with host networking. This contract is specific to QemuVm and may evolve as its QEMU command line gains features.

nested sut.params object

The nested SUT’s params dictionary is interpreted as a set of key/value pairs used to construct the QEMU command line. Currently, the following are supported:

key	type	required	default	description
memory	string	false	4G	Guest memory with an `M` or `G` suffix. Overcommit is permitted.
cpus	int	false	1	Guest vCPU count. Overcommit is permitted.
vel2	boolean	false	false	Run virtual machine at virtual EL2.

For example:

swprofiles:
  - name: host
    kernel: /path/to/host/Image.gz
    suts:
      - sutclass: guest
        params:
          memory: 8G
          cpus: 16
        swprofile:
          name: guest
          kernel: /path/to/guest/Image.gz

benchmark object

The top-level benchmarks key maps a list of benchmark dictionaries. Each benchmark represents a benchmark that is run for each software profile on the sut. It contains the following keys:

key	type	required	default	description
include	string	false	N/A	Path to benchmark fragment yaml, within a benchmark library. See Benchmark Library.
suite	string	false	unknown	Benchmark suite name. Passed when invoking container image.
name	string	true	N/A	Benchmark name. Passed when invoking container image.
type	string	false	unknown	Descriptive type label for the benchmark. E.g. cpu, memory, io, system, etc.
params	dict	false	{}	Dictionary of parameters specific to the benchmark. All keys and values must be strings. Passed when invoking container image.
image	string	true	N/A	Container image to pull to the SUT and invoke in order to run the benchmark. See Container Interface.
repeats	int	false	defaults.benchmark.repeats	Number of times to repeat the benchmark per boot session.
sessions	int	false	defaults.benchmark.sessions	Number of times to reboot the SUT to repeat the benchmark.
warmups	int	false	defaults.benchmark.warmups	Number of times to run the benchmark at the start of a boot session to warm up the system before the real repeats are executed.
timeout	string	false	defaults.benchmark.timeout	Timeout after which to assume the benchmark has hung. Provided as a string with format “<integer><suffix>” where the suffix is ‘s’ (seconds), ‘m’ (minutes), ‘h’ (hours) or ‘d’ days.
roles	list	false	[executer]	List of roles that the benchmark implements. When multiple roles are defined, each role is executed in parallel, possibly on different nodes. If not specified, the benchmark is assumed to have a single role.
rolemap	dict	false	{r: 0 for r in roles}	Dictionary mapping roles to nodes, where key is the role and value represents the node, either an integer index into sut.nodes or a node name if a string. Multiple roles may be mapped to the same node. Any unmapped roles default to sut.nodes[0].

defaults object

A dictionary that holds various default values. It contains the following keys:

key	type	required	default	description
benchmark	dict	false	default defaults.benchmark	Default values relating to the benchmark object.

defaults.benchmark object

A dictionary that holds various default values. It contains the following keys:

key	type	required	default	description
repeats	int	false	3	Default number of times to repeat a benchmark per boot session.
sessions	int	false	1	Number of times to reboot the SUT to repeat a benchmark.
warmups	int	false	1	Number of times to run the benchmark at the start of a boot session to warm up the system before the real repeats are executed.
timeout	string	false	1h	Timeout after which to assume the benchmark has hung. Provided as a string with format “<integer><suffix>” where the suffix is ‘s’ (seconds), ‘m’ (minutes), ‘h’ (hours) or ‘d’ days.

Benchmark Library

While the only required keys for a benchmark object are name and image, in practice a benchmark will usually have a number of parameters which can be set to vary its behaviour. It would be cumbersome to have to enter all these details for every benchmark in every plan, so to simplify this, a benchmark can be specified in its own yaml file fragment. And those yaml file fragments can be stored in a benchmark library, a well-known directory on the filesystem. Let’s say we have this simple plan, with a single benchmark:

sut:
  connection:
    method: SSH
    params:
      host: my-server
swprofiles:
  - kernel: /path/to/Image.gz
    modules: /path/to/modules.tar.xz
benchmarks:
  - suite: my-suite
    name: my-benchmark
    type: system
    params:
      cpus: 4
      ram: 4G
    image: docker.io/fastpath/benchmarks/my-benchmark:latest

Instead of defining the benchmark directly, we could define it in a benchmark fragment, let’s assume its called my-suite/my-benchmark-default.yaml and lives in the benchmark library:

suite: my-suite
name: my-benchmark
type: system
params:
  cpus: 4
  ram: 4G
image: docker.io/fastpath/benchmarks/my-benchmark:latest

Now the plan do simply include the benchmark:

sut:
  connection:
    method: SSH
    params:
      host: my-server
swprofiles:
  - kernel: /path/to/Image.gz
    modules: /path/to/modules.tar.xz
benchmarks:
  - include: my-suite/my-benchmark-default.yaml

We could even define a variant benchmark that inherits from the default one and overrides some parameters. Let’s assume this is called my-suite/my-benchmark-single.yaml:

include: my-suite/my-benchmark-default.yaml
params:
  cpus: 1

Now the plan can include both benchmarks:

sut:
  connection:
    method: SSH
    params:
      host: my-server
swprofiles:
  - kernel: /path/to/Image.gz
    modules: /path/to/modules.tar.xz
benchmarks:
  - include: my-suite/my-benchmark-default.yaml
  - include: my-suite/my-benchmark-single.yaml

For the generated catalog of benchmark fragments available in the Fastpath benchmark library, see Fastpath Benchmark Catalog.