Plan Schema

Top-Level plan object

A plan describes the set of benchmarks which should be run against a set of swprofiles (which includes a specific kernel) on a single SUT. The following is the set of top-level keys that may be defined in a plan:

key	type	required	default	description
sut	dict	true	N/A	Encapsulates all required information about the system under test (SUT). See “sut object”.
swprofiles	list	true	N/A	A list of swprofile objects, each of which describes a software profile that the SUT will be configured into, and for which the benchmarks will be run.
benchmarks	list	true	N/A	A list of benchmark objects, each describing a benchmark to run for each required software profile.
defaults	dict	false	default defaults object	Default values for various parameters. See “defaults object”.

sut object

The sut dictionary describes The system under test (SUT):

key	type	required	default	description
name	string	false	None	User-supplied friendly name to identify the system under test.
connection	dict	true	N/A	Describes how to connect to SUT. See “connection object”.

sut.connection object

The connection dictionary describes how to connect to the SUT that the benchmarks will run on. It contains the following keys:

key	type	required	default	description
method	enum	true	N/A	Method used to connect to the SUT. For now, only “SSH” is supported. In future, “LAVA” will be added.
params	dict	true	N/A	Method-specific dictionary of parameters. See “SSH-params” below.

sut.connection.SSH-params object

The SSH-params dictionary describes how to connect to a SUT using the “SSH” method. It contains the following keys:

key	type	required	default	description
host	string	true	N/A	Host name or IP address of the connection, or name of Host in SSH config file.
user	string	false	None	Login user for the remote connection. When None, SSH uses its default configured user, which may be specified in the SSH config file if host is the name of a Host in the SSH config file.
port	int	false	None	Remote port to connect to. When None, SSH uses its default configured port, which may be specified in the SSH config file if host is the name of a Host in the SSH config file.
keyfile	string	false	None	Path of private key to use for connection. When None, SSH uses its default configured private key(s).

swprofile object

The top-level swprofiles key maps a list of swprofile dictionaries. Each swprofile represents a software profile that the benchmarks are run against. It contains the following keys:

key	type	required	default	description
name	string	false	None	User-supplied friendly name to identify the software profile.
pkgtype	enum	false	Inferred	Describes what “kernel” parameter points to. Either “RAW” (if a raw kernel Image) or “DEB” (if a Debian package). If omitted, value is inferred; “DEB” if kernel string ends with “.deb”, or “RAW” otherwise.
kernel	string	true	N/A	The kernel package to be used by the SUT. Installation, reboot and uninstallation is managed by the automatically. May be a filesystem path or a URL.
modules	string	false	None	When pkgtype is RAW, modules to install, provided as tarball. Must be None for other pkgtypes. May be a filesystem path or a URL.
cmdline	list	false	[]	List of additional command line options to be appended to the kernel’s command line. The list is joined with a space (” “) separator.
sysctl	list	false	[]	List of sysctls to be applied persistently prior to reboot. Then reverted after reboot to remove persistence.
bootscript	list	false	[]	Bash commands to be executed after reboot with the correct kernel.
gitsha	string	false	None	Full, 40 character Git SHA for the revision of the source code used to build the provided kernel, or None if not known.

benchmark object

The top-level benchmarks key maps a list of benchmark dictionaries. Each benchmark represents a benchmark that is run for each software profile on the sut. It contains the following keys:

key	type	required	default	description
include	string	false	N/A	Path to benchmark fragment yaml, within a benchmark library. See Benchmark Library.
suite	string	false	unknown	Benchmark suite name. Passed when invoking container image.
name	string	true	N/A	Benchmark name. Passed when invoking container image.
type	string	false	unknown	Descriptive type label for the benchmark. E.g. cpu, memory, io, system, etc.
params	dict	false	{}	Dictionary of parameters specific to the benchmark. All keys and values must be strings. Passed when invoking container image.
image	string	true	N/A	Container image to pull to the SUT and invoke in order to run the benchmark. See Container Interface.
repeats	int	false	defaults.benchmark.repeats	Number of times to repeat the benchmark per boot session.
sessions	int	false	defaults.benchmark.sessions	Number of times to reboot the SUT to repeat the benchmark.
warmups	int	false	defaults.benchmark.warmups	Number of times to run the benchmark at the start of a boot session to warm up the system before the real repeats are executed.
timeout	string	false	defaults.benchmark.timeout	Timeout after which to assume the benchmark has hung. Provided as a string with format “<integer><suffix>” where the suffix is ‘s’ (seconds), ‘m’ (minutes), ‘h’ (hours) or ‘d’ days.

defaults object

A dictionary that holds various default values. It contains the following keys:

key	type	required	default	description
benchmark	dict	false	default defaults.benchmark	Default values relating to the benchmark object.

defaults.benchmark object

A dictionary that holds various default values. It contains the following keys:

key	type	required	default	description
repeats	int	false	3	Default number of times to repeat a benchmark per boot session.
sessions	int	false	1	Number of times to reboot the SUT to repeat a benchmark.
warmups	int	false	1	Number of times to run the benchmark at the start of a boot session to warm up the system before the real repeats are executed.
timeout	string	false	1h	Timeout after which to assume the benchmark has hung. Provided as a string with format “<integer><suffix>” where the suffix is ‘s’ (seconds), ‘m’ (minutes), ‘h’ (hours) or ‘d’ days.

Benchmark Library

While the only required keys for a benchmark object are name and image, in practice a benchmark will usually have a number of parameters which can be set to vary its behaviour. It would be cumbersome to have to enter all these details for every benchmark in every plan, so to simplify this, a benchmark can be specified in its own yaml file fragment. And those yaml file fragments can be stored in a benchmark library, a well-known directory on the filesystem. Let’s say we have this simple plan, with a single benchmark:

sut:
  connection:
    method: SSH
    params:
      host: my-server
swprofiles:
  - kernel: /path/to/Image.gz
    modules: /path/to/modules.tar.xz
benchmarks:
  - suite: my-suite
    name: my-benchmark
    type: system
    params:
      cpus: 4
      ram: 4G
    image: docker.io/fastpath/benchmarks/my-benchmark:latest

Instead of defining the benchmark directly, we could define it in a benchmark fragment, let’s assume its called my-suite/my-benchmark-default.yaml and lives in the benchmark library:

suite: my-suite
name: my-benchmark
type: system
params:
  cpus: 4
  ram: 4G
image: docker.io/fastpath/benchmarks/my-benchmark:latest

Now the plan do simply include the benchmark:

sut:
  connection:
    method: SSH
    params:
      host: my-server
swprofiles:
  - kernel: /path/to/Image.gz
    modules: /path/to/modules.tar.xz
benchmarks:
  - include: my-suite/my-benchmark-default.yaml

We could even define a variant benchmark that inherits from the default one and overrides some parameters. Let’s assume this is called my-suite/my-benchmark-single.yaml:

include: my-suite/my-benchmark-default.yaml
params:
  cpus: 1

Now the plan can include both benchmarks:

sut:
  connection:
    method: SSH
    params:
      host: my-server
swprofiles:
  - kernel: /path/to/Image.gz
    modules: /path/to/modules.tar.xz
benchmarks:
  - include: my-suite/my-benchmark-default.yaml
  - include: my-suite/my-benchmark-single.yaml