.. # Copyright (c) 2025, Arm Limited. # # SPDX-License-Identifier: MIT ####################### Define & Execute a Plan ####################### ************ Introduction ************ A plan is a yaml file that defines: - The connection to the SUT that will run all the benchmarks - A list of software profiles for which the benchmarks will be run - A list of benchmark to run The user creates a plan and passes it to Fastpath for execution. Fastpath configures the SUT for each software profile in turn, by installing the specified kernel and booting it with the specified command line parameters, etc, and runs the specified benchmarks, collating the results and logs into an output "resultstore". .. image:: ../images/fastpath-plan-exec.png The plan may refer to supplementary benchmark yaml files which are part of a Benchmark Library. Fastpath ships with a default `Benchmark Library `_. Users may additionally set up their own, private Benchamrk Libraries. See :doc:`planschema` for a full description of the plan and benchmark library. ******************* Build a Simple Plan ******************* 1. Define the ``sut`` section: .. code-block:: yaml sut: name: my-fastpath-sut connection: method: SSH params: user: fpuser host: port: keyfile: This section defines the parameters to connect to the SUT over SSH. Here we assume you're using the ``fpuser`` account as per the guide at :doc:`configuresut`. The port will usually be 22, and in that case there is no need to specify it since that's the default. The keyfile is the private key associated with the user account which will allow logging in to the SUT via SSH as described at :doc:`configuresut`. Alternatively, you could define all of this as part of your ~/.ssh/config then refer to the config name using the only host field: .. code-block:: yaml sut: name: my-fastpath-sut connection: method: SSH params: host: my-fastpath-sut .. warning:: The private key must be kept secure as it can be used to access the SUT with root privileges. 2. Define the ``swprofiles`` under which the benchmarks will execute: .. code-block:: yaml swprofiles: - name: without-mthp kernel: /path/to/Image.gz modules: /path/to/modules.tar.xz cmdline: - thp_anon=64K:never;2M:madvise - name: with-mthp kernel: /path/to/Image.gz modules: /path/to/modules.tar.xz cmdline: - thp_anon=64K:always;2M:madvise Each swprofile must contain the path to the kernel image, and if modules are needed then the path to the modules tarball must be provided as well. See :doc:`buildkernel` for a guide to building an appropriate kernel. This example defines 2 swprofiles, both using the same kernel but one has 64K multi-size THP disabled and the other has it enabled. This demonstrates how to configure the kernel command line. We give each swprofile a convenient name so that we can easily refer to them when doing analysis on the results. 3. Define the ``benchmarks`` to execute: .. code-block:: yaml benchmarks: - include: speedometer/v2.1.yaml - include: mmtests/kernbench.yaml Here we will run 2 benchmarks, Speedometer v2.1 (Javascript benchmark) and kernbench (Time to compile the linux kernel). The actual benchmark definitions are in the yaml files within the Benchmark Library. 4. Add some miscellaneous parameters: .. code-block:: yaml defaults: benchmark: warmups: 1 repeats: 3 sessions: 2 timeout: 1h Each benchmark may specify some optional parameters, including how many times to run each benchmark and a per-iteration timeout. In cases where the values are not explicitly provided for a benchmark, Fastpath will fallback to default values, which can be specified globally here. In this case, we request 2 sessions meaning the system will be booted twice. Then in each session we will run 1 warmup run, where the results are discarded, and 3 normal repeats where the results are kept. See :doc:`planschema` for more info. Putting it all together, we have the following which should be saved as demo.yaml: .. code-block:: yaml sut: name: my-fastpath-sut connection: method: SSH params: host: my-fastpath-sut swprofiles: - name: without-mthp kernel: /path/to/Image.gz modules: /path/to/modules.tar.xz cmdline: - thp_anon=64K:never;2M:madvise - name: with-mthp modules: /path/to/modules.tar.xz cmdline: - thp_anon=64K:always;2M:madvise benchmarks: - include: speedometer/v2.1.yaml - include: mmtests/kernbench.yaml defaults: benchmark: warmups: 1 repeats: 3 sessions: 2 timeout: 1h Let's validate the plan with the following command: .. code-block:: shell fastpath plan show demo.yaml This will parse, normalise, flatten, and validate the plan. If your plan is buggy, this command will tell you why. If your plan is well-formed, it will output the plan in its final form: .. code-block:: yaml %YAML 1.2 --- defaults: benchmark: repeats: 3 sessions: 2 warmups: 1 timeout: 1h sut: name: my-fastpath-sut connection: method: SSH params: host: my-fastpath-sut user: null port: null keyfile: null swprofiles: - name: without-mthp pkgtype: null kernel: /data_nvme0n1/ryarob01/fastpath/demo/assets/Image.gz modules: /data_nvme0n1/ryarob01/fastpath/demo/assets/modules.tar.xz cmdline: - thp_anon=64K:never;2M:madvise sysctl: [] bootscript: [] gitsha: null - name: with-mthp pkgtype: null kernel: /data_nvme0n1/ryarob01/fastpath/demo/assets/Image.gz modules: /data_nvme0n1/ryarob01/fastpath/demo/assets/modules.tar.xz cmdline: - thp_anon=64K:always;2M:madvise sysctl: [] bootscript: [] gitsha: null benchmarks: - suite: speedometer name: v2.1 type: browser params: {} image: registry.gitlab.geo.arm.com/software/linux-arm/fastpath/containers/speedometer:v1.0 repeats: 3 sessions: 2 warmups: 1 timeout: 10m - suite: mmtests name: kernbench type: system params: config: |- export KERNBENCH_ITERATIONS=1 export KERNBENCH_VERSION=6.12 config-base: configs/config-workload-kernbench-max performance-governor: 'False' image: registry.gitlab.geo.arm.com/software/linux-arm/fastpath/containers/mmtests:v1.0 repeats: 3 sessions: 2 warmups: 1 timeout: 1h **************** Execute the Plan **************** Execute the plan, placing the results in the results directory: .. code-block:: shell fastpath plan exec --output results/ demo.yaml The plan will execute, showing a progress as benchmarks are completed: .. code-block:: shell Executing demo.yaml... 3%|██▍ | 1/32 [05:01<2:35:48, 301.57s/it] Upon completion, the results can be managed and analysed using various Fastpath commands. See :doc:`resultshow` for more information.