.. # Copyright (c) 2025, Arm Limited. # # SPDX-License-Identifier: MIT ########### Plan Schema ########### ********************* Top-Level plan object ********************* A plan describes the set of benchmarks which should be run against a set of swprofiles (which includes a specific kernel) on a single SUT. The following is the set of top-level keys that may be defined in a plan: ========== ====== ======== =========================== =========== key type required default description ========== ====== ======== =========================== =========== sut dict true N/A Encapsulates all required information about the system under test (SUT). See "sut object". swprofiles list true N/A A list of swprofile objects, each of which describes a software profile that the SUT will be configured into, and for which the benchmarks will be run. benchmarks list true N/A A list of benchmark objects, each describing a benchmark to run for each required software profile. defaults dict false default defaults object Default values for various parameters. See "defaults object". ========== ====== ======== =========================== =========== ********** sut object ********** The sut dictionary describes The system under test (SUT): ========== ====== ======== =========================== =========== key type required default description ========== ====== ======== =========================== =========== name string false None User-supplied friendly name to identify the system under test. connection dict true N/A Describes how to connect to SUT. See "connection object". ========== ====== ======== =========================== =========== ********************* sut.connection object ********************* The connection dictionary describes how to connect to the SUT that the benchmarks will run on. It contains the following keys: ========== ====== ======== =========================== =========== key type required default description ========== ====== ======== =========================== =========== method enum true N/A Method used to connect to the SUT. For now, only "SSH" is supported. In future, "LAVA" will be added. params dict true N/A Method-specific dictionary of parameters. See "SSH-params" below. ========== ====== ======== =========================== =========== ******************************** sut.connection.SSH-params object ******************************** The SSH-params dictionary describes how to connect to a SUT using the "SSH" method. It contains the following keys: ========== ====== ======== =========================== =========== key type required default description ========== ====== ======== =========================== =========== host string true N/A Host name or IP address of the connection, or name of Host in SSH config file. user string false None Login user for the remote connection. When None, SSH uses its default configured user, which may be specified in the SSH config file if host is the name of a Host in the SSH config file. port int false None Remote port to connect to. When None, SSH uses its default configured port, which may be specified in the SSH config file if host is the name of a Host in the SSH config file. keyfile string false None Path of private key to use for connection. When None, SSH uses its default configured private key(s). ========== ====== ======== =========================== =========== **************** swprofile object **************** The top-level swprofiles key maps a list of swprofile dictionaries. Each swprofile represents a software profile that the benchmarks are run against. It contains the following keys: ========== ====== ======== =========================== =========== key type required default description ========== ====== ======== =========================== =========== name string false None User-supplied friendly name to identify the software profile. pkgtype enum false Inferred Describes what "kernel" parameter points to. Either "RAW" (if a raw kernel Image) or "DEB" (if a Debian package). If omitted, value is inferred; "DEB" if kernel string ends with ".deb", or "RAW" otherwise. kernel string true N/A The kernel package to be used by the SUT. Installation, reboot and uninstallation is managed by the automatically. May be a filesystem path or a URL. modules string false None When pkgtype is RAW, modules to install, provided as tarball. Must be None for other pkgtypes. May be a filesystem path or a URL. cmdline list false [] List of additional command line options to be appended to the kernel's command line. The list is joined with a space (" ") separator. sysctl list false [] List of sysctls to be applied persistently prior to reboot. Then reverted after reboot to remove persistence. bootscript list false [] Bash commands to be executed after reboot with the correct kernel. gitsha string false None Full, 40 character Git SHA for the revision of the source code used to build the provided kernel, or None if not known. ========== ====== ======== =========================== =========== **************** benchmark object **************** The top-level benchmarks key maps a list of benchmark dictionaries. Each benchmark represents a benchmark that is run for each software profile on the sut. It contains the following keys: ========== ====== ======== =========================== =========== key type required default description ========== ====== ======== =========================== =========== include string false N/A Path to benchmark fragment yaml, within a benchmark library. See Benchmark Library. suite string false unknown Benchmark suite name. Passed when invoking container image. name string true N/A Benchmark name. Passed when invoking container image. type string false unknown Descriptive type label for the benchmark. E.g. cpu, memory, io, system, etc. params dict false {} Dictionary of parameters specific to the benchmark. All keys and values must be strings. Passed when invoking container image. image string true N/A Container image to pull to the SUT and invoke in order to run the benchmark. See Container Interface. repeats int false defaults.benchmark.repeats Number of times to repeat the benchmark per boot session. sessions int false defaults.benchmark.sessions Number of times to reboot the SUT to repeat the benchmark. warmups int false defaults.benchmark.warmups Number of times to run the benchmark at the start of a boot session to warm up the system before the real repeats are executed. timeout string false defaults.benchmark.timeout Timeout after which to assume the benchmark has hung. Provided as a string with format "" where the suffix is 's' (seconds), 'm' (minutes), 'h' (hours) or 'd' days. ========== ====== ======== =========================== =========== *************** defaults object *************** A dictionary that holds various default values. It contains the following keys: ========== ====== ======== =========================== =========== key type required default description ========== ====== ======== =========================== =========== benchmark dict false default defaults.benchmark Default values relating to the benchmark object. ========== ====== ======== =========================== =========== ************************* defaults.benchmark object ************************* A dictionary that holds various default values. It contains the following keys: ========== ====== ======== =========================== =========== key type required default description ========== ====== ======== =========================== =========== repeats int false 3 Default number of times to repeat a benchmark per boot session. sessions int false 1 Number of times to reboot the SUT to repeat a benchmark. warmups int false 1 Number of times to run the benchmark at the start of a boot session to warm up the system before the real repeats are executed. timeout string false 1h Timeout after which to assume the benchmark has hung. Provided as a string with format "" where the suffix is 's' (seconds), 'm' (minutes), 'h' (hours) or 'd' days. ========== ====== ======== =========================== =========== ################# Benchmark Library ################# While the only required keys for a benchmark object are name and image, in practice a benchmark will usually have a number of parameters which can be set to vary its behaviour. It would be cumbersome to have to enter all these details for every benchmark in every plan, so to simplify this, a benchmark can be specified in its own yaml file fragment. And those yaml file fragments can be stored in a benchmark library, a well-known directory on the filesystem. Let's say we have this simple plan, with a single benchmark: .. code-block:: yaml sut: connection: method: SSH params: host: my-server swprofiles: - kernel: /path/to/Image.gz modules: /path/to/modules.tar.xz benchmarks: - suite: my-suite name: my-benchmark type: system params: cpus: 4 ram: 4G image: docker.io/fastpath/benchmarks/my-benchmark:latest Instead of defining the benchmark directly, we could define it in a benchmark fragment, let's assume its called `my-suite/my-benchmark-default.yaml` and lives in the benchmark library: .. code-block:: yaml suite: my-suite name: my-benchmark type: system params: cpus: 4 ram: 4G image: docker.io/fastpath/benchmarks/my-benchmark:latest Now the plan do simply include the benchmark: .. code-block:: yaml sut: connection: method: SSH params: host: my-server swprofiles: - kernel: /path/to/Image.gz modules: /path/to/modules.tar.xz benchmarks: - include: my-suite/my-benchmark-default.yaml We could even define a variant benchmark that inherits from the default one and overrides some parameters. Let's assume this is called `my-suite/my-benchmark-single.yaml`: .. code-block:: yaml include: my-suite/my-benchmark-default.yaml params: cpus: 1 Now the plan can include both benchmarks: .. code-block:: yaml sut: connection: method: SSH params: host: my-server swprofiles: - kernel: /path/to/Image.gz modules: /path/to/modules.tar.xz benchmarks: - include: my-suite/my-benchmark-default.yaml - include: my-suite/my-benchmark-single.yaml