Define & Execute a Plan

Introduction

A plan is a yaml file that defines:

The connection to the SUT that will run all the benchmarks
A list of software profiles for which the benchmarks will be run
A list of benchmark to run

The user creates a plan and passes it to Fastpath for execution. Fastpath configures the SUT for each software profile in turn, by installing the specified kernel and booting it with the specified command line parameters, etc, and runs the specified benchmarks, collating the results and logs into an output “resultstore”.

The plan may refer to supplementary benchmark yaml files which are part of a Benchmark Library. Fastpath ships with a default Benchmark Library. Users may additionally set up their own, private Benchamrk Libraries.

See Plan Schema for a full description of the plan and benchmark library.

Build a Simple Plan

Define the sut section:

sut:
  name: my-fastpath-sut
  connection:
    method: SSH
    params:
      user: fpuser
      host: <sut hostname>
      port: <ssh port>
      keyfile: <private key>

This section defines the parameters to connect to the SUT over SSH. Here we assume you’re using the fpuser account as per the guide at Setup & Configure a SUT. The port will usually be 22, and in that case there is no need to specify it since that’s the default. The keyfile is the private key associated with the user account which will allow logging in to the SUT via SSH as described at Setup & Configure a SUT.

Alternatively, you could define all of this as part of your ~/.ssh/config then refer to the config name using the only host field:

sut:
  name: my-fastpath-sut
  connection:
    method: SSH
    params:
      host: my-fastpath-sut

Warning

The private key must be kept secure as it can be used to access the SUT with root privileges.

Define the swprofiles under which the benchmarks will execute:

swprofiles:
  - name: without-mthp
    kernel: /path/to/Image.gz
    modules: /path/to/modules.tar.xz
    cmdline:
      - thp_anon=64K:never;2M:madvise
  - name: with-mthp
    kernel: /path/to/Image.gz
    modules: /path/to/modules.tar.xz
    cmdline:
      - thp_anon=64K:always;2M:madvise

Each swprofile must contain the path to the kernel image, and if modules are needed then the path to the modules tarball must be provided as well. See Build a Suitable Kernel for a guide to building an appropriate kernel.

This example defines 2 swprofiles, both using the same kernel but one has 64K multi-size THP disabled and the other has it enabled. This demonstrates how to configure the kernel command line. We give each swprofile a convenient name so that we can easily refer to them when doing analysis on the results.

Define the benchmarks to execute:

benchmarks:
  - include: speedometer/v2.1.yaml
  - include: mmtests/kernbench.yaml

Here we will run 2 benchmarks, Speedometer v2.1 (Javascript benchmark) and kernbench (Time to compile the linux kernel). The actual benchmark definitions are in the yaml files within the Benchmark Library.

Add some miscellaneous parameters:

defaults:
  benchmark:
    warmups: 1
    repeats: 3
    sessions: 2
    timeout: 1h

Each benchmark may specify some optional parameters, including how many times to run each benchmark and a per-iteration timeout. In cases where the values are not explicitly provided for a benchmark, Fastpath will fallback to default values, which can be specified globally here. In this case, we request 2 sessions meaning the system will be booted twice. Then in each session we will run 1 warmup run, where the results are discarded, and 3 normal repeats where the results are kept. See Plan Schema for more info.

Putting it all together, we have the following which should be saved as demo.yaml:

sut:
  name: my-fastpath-sut
  connection:
    method: SSH
    params:
      host: my-fastpath-sut
swprofiles:
  - name: without-mthp
    kernel: /path/to/Image.gz
    modules: /path/to/modules.tar.xz
    cmdline:
      - thp_anon=64K:never;2M:madvise
  - name: with-mthp
    modules: /path/to/modules.tar.xz
    cmdline:
      - thp_anon=64K:always;2M:madvise
benchmarks:
  - include: speedometer/v2.1.yaml
  - include: mmtests/kernbench.yaml
defaults:
  benchmark:
    warmups: 1
    repeats: 3
    sessions: 2
    timeout: 1h

Let’s validate the plan with the following command:

fastpath plan show demo.yaml

This will parse, normalise, flatten, and validate the plan. If your plan is buggy, this command will tell you why. If your plan is well-formed, it will output the plan in its final form:

%YAML 1.2
---
defaults:
  benchmark:
    repeats: 3
    sessions: 2
    warmups: 1
    timeout: 1h
sut:
  name: my-fastpath-sut
  connection:
    method: SSH
    params:
      host: my-fastpath-sut
      user: null
      port: null
      keyfile: null
swprofiles:
- name: without-mthp
  pkgtype: null
  kernel: /data_nvme0n1/ryarob01/fastpath/demo/assets/Image.gz
  modules: /data_nvme0n1/ryarob01/fastpath/demo/assets/modules.tar.xz
  cmdline:
  - thp_anon=64K:never;2M:madvise
  sysctl: []
  bootscript: []
  gitsha: null
- name: with-mthp
  pkgtype: null
  kernel: /data_nvme0n1/ryarob01/fastpath/demo/assets/Image.gz
  modules: /data_nvme0n1/ryarob01/fastpath/demo/assets/modules.tar.xz
  cmdline:
  - thp_anon=64K:always;2M:madvise
  sysctl: []
  bootscript: []
  gitsha: null
benchmarks:
- suite: speedometer
  name: v2.1
  type: browser
  params: {}
  image: registry.gitlab.geo.arm.com/software/linux-arm/fastpath/containers/speedometer:v1.0
  repeats: 3
  sessions: 2
  warmups: 1
  timeout: 10m
- suite: mmtests
  name: kernbench
  type: system
  params:
    config: |-
      export KERNBENCH_ITERATIONS=1
      export KERNBENCH_VERSION=6.12
    config-base: configs/config-workload-kernbench-max
    performance-governor: 'False'
  image: registry.gitlab.geo.arm.com/software/linux-arm/fastpath/containers/mmtests:v1.0
  repeats: 3
  sessions: 2
  warmups: 1
  timeout: 1h

Execute the Plan

Execute the plan, placing the results in the results directory:

fastpath plan exec --output results/ demo.yaml

The plan will execute, showing a progress as benchmarks are completed:

Executing demo.yaml...
  3%|██▍                                                                         | 1/32 [05:01<2:35:48, 301.57s/it]

Upon completion, the results can be managed and analysed using various Fastpath commands. See Store & Analyse Results for more information.