Nested SUT Execution Design

Status

Prototype implementation design.

Summary

Allow a software profile in a plan to declare child SUTs. fastpath plan exec configures the profile on the parent SUT, creates each child as a KVM virtual machine on that parent, and repeats the process recursively. Benchmarks run only on leaf SUTs. For a given benchmark, all leaves execute each warmup and measured repeat concurrently.

The initial implementation supports one-node child SUTs and requires every SUT which hosts children to have one node. The data model represents the hosting relationship between nodes so that a later implementation can support a multi-node child without another resultstore schema change.

Goals

Express an arbitrary-depth tree of SUTs in a plan.
Specify each child VM configuration directly in the plan.
Require exactly one software profile for each nested SUT.
Configure parent software before creating its children.
Run the same benchmark concurrently on every leaf SUT.
Record enough hierarchy in the resultstore to reconstruct VM placement.
Reuse the existing VM container and invoke QEMU in it directly.
Preserve existing behavior for plans without nested SUTs.

Non-goals for the first version

Hosting child VMs on a multi-node parent SUT.
Creating a multi-node child SUT.
Supporting nested SUT backends other than KVM.
Dynamically balancing children between parent nodes.
Continuing with a subset of leaves after provisioning or execution failure.
Running benchmarks on non-leaf SUTs.
Running different benchmark definitions on different leaves.

Terminology

root SUT: The concrete SUT in the top-level sut section. It is acquired by the scheduler or supplied directly to plan exec as it is today.
nested SUT: A SUT declared in a profile’s suts list and created by plan exec.
hosting node: The single node of a parent SUT on which a nested VM is created.
leaf SUT: A SUT whose software profile has no suts entries. Only leaf SUTs execute benchmarks.

Plan Schema

Each software profile gains an optional suts list, defaulting to an empty list. Each list item accepts the following fields:

sutclass: Optional logical SUT class name recorded in the resultstore, defaulting to None. It classifies the created child but does not select or allocate a scheduler provider.
image: Container-image reference used to launch QEMU. Every nested SUT has an image, but the field may be omitted because normalization supplies the default registry.gitlab.arm.com/tooling/fastpath/containers/vm:latest. The value cannot be None. This is a Docker image containing the VMM and all VM assets, not just the guest root filesystem image.
params: Optional VM launch-parameter mapping, defaulting to an empty mapping. It describes how plan exec creates the child on the exact parent SUT represented by the enclosing profile.
swprofile: Required single software-profile mapping. This is deliberately singular; nested SUTs cannot declare a list of alternative profiles.

For example:

sut:
  sutclass: aws-m7g.metal
swprofiles:
  - name: host-profile
    kernel: host-image
    suts:
      - sutclass: vm-small
        image: registry.example.com/fastpath/vm:latest
        params:
          memory: 4G
          cpus: 4
        swprofile:
          name: leaf-a
          kernel: guest-image-a
      - sutclass: vm-large
        image: registry.example.com/fastpath/vm:latest
        params:
          memory: 8G
          cpus: 8
        swprofile:
          name: level-one
          kernel: guest-image-b
          suts:
            - sutclass: vm-small
              image: registry.example.com/fastpath/vm:latest
              params:
                memory: 4G
                cpus: 4
              swprofile:
                name: leaf-b
                kernel: guest-image-c
benchmarks:
  - include: micromm/fork.yaml

Schema and semantic rules are:

suts is valid on top-level and nested software profiles.
A nested entry permits only sutclass, image, params, and swprofile. Only swprofile is required.
A nested entry cannot provide a name, node list, or connection details.
sutclass is metadata for the created SUT; changing it does not change VM resources. Hardware fingerprinting records the resources actually observed in the guest.
image must normalize to a container-image reference. If omitted, it defaults to registry.gitlab.arm.com/tooling/fastpath/containers/vm:latest; explicitly setting it to None is invalid.
params.memory is an optional size with an M or G suffix, defaulting to 4G.
params.cpus is an optional positive integer, defaulting to 1.
params.vel2 is an optional boolean, defaulting to false. When true, the guest receives virtual EL2 and can host nested SUTs.
SSH keys, forwarded ports, generated names, and runtime connection details are executor-owned state and are not valid params fields.
A SUT with children must contain exactly one node. This rejects multi-node parents before any machine is modified.
The VM factory always creates one-node child SUTs.
Missing suts lists normalize to [] and appear in schema order in plan show output.
Benchmark role-map node indexes remain local to each leaf SUT. Since nested children are one-node SUTs, only node index 0 is valid on nested leaves.

The schema is recursive. Validation should therefore use a shared profile schema registered with Cerberus, or a small recursive profile validator, rather than copying the schema to a fixed maximum depth.

VM Parameters and Credentials

Nested VM parameters are part of the plan because plan exec already owns the exact parent SUT. They describe a host-bound creation operation, not a request for the scheduler to find or allocate another resource. This keeps direct and scheduled execution self-contained and avoids introducing a second provider-resolution step inside plan exec.

plan exec generates an ephemeral SSH keypair for the topology. It injects the public key into each VM through cloud-init, uses the private key for child and descendant connections, and removes the private key during final cleanup. The key file is created with owner-only permissions and is never written to the normalized plan, logs, or resultstore. One keypair per plan exec invocation is sufficient because all descendant connections are scoped to that invocation.

The scheduler upload and plan-splitting paths must walk profiles recursively when validating, uploading, and rewriting kernel and modules paths. Only the top-level profile list is split into jobs; nested profiles stay attached to their parent profile.

VM Container Image

The nested SUT image is a Docker image used as the complete VM-launch environment. plan exec starts a privileged container from it with host networking, overrides its entry point, prepares cloud-init data inside the container, and invokes QEMU there. It is therefore distinct from the guest root filesystem image, which is one asset contained within the Docker image.

The image contract belongs to the VMM implementation rather than to nested SUTs in general. The initial QemuVm implementation requires:

qemu-system-aarch64 with Arm KVM virtual EL2 support and cloud-localds on PATH;
the normal shell utilities used by its generated commands, including base64, cat, grep, kill, printf, sh, and sleep;
EFI firmware at /vm/AAVMF_CODE.fd and /vm/AAVMF_VARS.fd;
a writable guest root filesystem at /vm/noble-server-cloudimg-arm64.img; and
a writable /vm directory for generated cloud-init data, seed.iso, qemu.pid, and qemu.log.

When params.vel2 is true, QemuVm enables the virt machine’s virtualization property. The physical host, host kernel, and container’s QEMU/KVM stack must support Arm nested virtualization when this option is used.

These paths and tools form the current QemuVm image interface. They are not intended as a generic interface shared by every VMM. A future VMM component, such as CrossVm, may define different plan parameters and a different image contract so that VMM-specific features do not have to fit a lowest-common- denominator abstraction. Changes to the commands or assets used by QemuVm must be reflected in both its default image and this documented interface.

VM Provisioning Architecture

Nested execution already owns the exact parent SUT, so plan exec performs host-bound VM operations directly through a reusable component, for example QemuVm. Each instance owns the configuration and lifecycle of one VM. Construction has no remote side effects; callers may invoke start() and stop() explicitly or use the instance as a context manager:

with QemuVm(parent, container, hostname, public_key, image,
           user, memory, cpus, ssh_port) as vm:
    run_workload(vm)

Validate a hosting node and normalized VM parameters.
Allocate a sibling-local VM number and SSH port.
Start the VM container on a supplied parent connection.
Return the generated child SUT description and runtime connection data.
Stop and remove the VM container idempotently.
Roll back a partially created container when VM startup fails.

The component provides one implementation of quoting, naming, key handling, image selection, and QEMU command construction without coupling the plan executor to scheduler allocation state. plan exec owns the credentials: it passes its generated public key to the VM and uses the corresponding ephemeral private key for runtime connections.

Use a deterministic VM name based on the execution topology and sibling index. Sibling ports start at an executor-owned base (currently 8022). Ports are not a plan parameter. The same port range can be reused under different parents because each parent has its own network namespace.

Nested Connectivity

The current QEMU command forwards guest port 22 onto the immediate parent’s network namespace. That is directly reachable for a first-level VM but not for an arbitrary-depth descendant. Runtime child connections must therefore use the parent SSH connection as a gateway.

Extend machine.SSHMachine so a connection can be opened through another SSHMachine using Fabric/Paramiko gateway or direct-tcpip support. A child connects to localhost:<allocated-port> as seen from its parent. Its own children use that child connection as the next gateway, forming a chain at any depth. Gateway objects are runtime state and must not be serialized into the normalized plan or resultstore.

The existing subconnection() behavior must preserve the gateway chain so parallel benchmark roles still receive independent SSH transports.

Host Validation

Before starting any child on a parent, validate the constraints Fastpath can determine without launching QEMU:

The parent SUT has exactly one node.
Docker is usable by Fastpath, as required by the existing VM launcher.
/dev/kvm exists as a character device on the parent.
The requested VM image supports the parent architecture. The current image and QEMU configuration are AArch64-only, so reject other architectures clearly.

Fastpath deliberately does not compare requested guest memory or vCPU totals with the parent’s discovered resources. CPU and memory overcommit are valid VM parameters and may be used to stress KVM. If the host or QEMU cannot satisfy a request, VM startup fails through the normal provisioning and cleanup path.

Preflight that /dev/kvm exists, but do not check CPU virtualization flags or the SSH user’s access to the device. The VM container is privileged, and QEMU startup remains the authoritative capability check when the device exists but is unusable or the complete machine configuration is unsupported. Preserve QEMU’s stderr in the Fastpath log and wrap startup failure with the child’s topology path. Any siblings which already started are removed through normal rollback.

Execution Model

For each top-level software profile, retain serial profile execution and use the following lifecycle:

Configure and fingerprint the root SUT with the top-level profile.
Normalize and validate all of that profile’s immediate child definitions.
Start sibling VMs concurrently.
Connect to each child concurrently.
Configure each child’s single profile concurrently.
Fingerprint each child’s hardware and active software concurrently.
Repeat steps 2–6 recursively for children which are not leaves.
Set up the benchmark containers and logs on all leaves.
For each benchmark session, benchmark, warmup, and repeat, execute all leaves concurrently and wait for all leaves before advancing.
Tear down the VM tree in post-order, concurrently between siblings.

The executor adds runtime-only parent, children, active profile, VM, and log directory state to its existing Sut model. These fields preserve the existing do_one_sut() execution flow and are not serialized into normalized plan dictionaries or resultstores. The same Benchmark objects are shared by all leaves; each SUT owns its benchmark-specific role maps and directories.

Concurrency Semantics

“Concurrently” means lockstep at repeat granularity:

Every leaf starts a given warmup or measured repeat before Fastpath waits for any leaf’s completion.
The next repeat does not begin until all leaves finish the current repeat.
Roles within each leaf continue to execute concurrently as they do today.
The next benchmark does not begin until every leaf finishes the current one.

Generate one session UUID for each concurrent leaf cohort. Each result remains owned by its leaf SUT, while the shared UUID identifies results produced during the same coordinated session.

SQLAlchemy sessions, the global logger, progress reporting, and resultstore merges are not currently thread-safe. Worker functions should return parsed result/error objects and log data to the coordinator. The coordinator performs resultstore merges serially. Logging should include the topology path and use separate per-leaf directories. Progress totals must be multiplied by the statically known leaf count.

Only leaves reboot between benchmark sessions. Rebooting an internal parent would destroy all descendants. The VM tree is destroyed before moving to the next top-level software profile.

Log Layout

The log directory mirrors the runtime SUT hierarchy. Each child uses a sut-<index> directory inside its parent’s software-profile directory, where index is its position in the enclosing profile’s suts list. The child directory contains the same software-profile content that logs/ contains for a root SUT. For example:

logs/
`-- swprofile-root-<hash>/
    |-- swprofile.yaml
    |-- session-<uuid>-node-0.kmsg
    |-- sut-0/
    |   `-- swprofile-middle-<hash>/
    |       |-- swprofile.yaml
    |       |-- session-<uuid>-node-0.kmsg
    |       `-- sut-0/
    |           `-- swprofile-deep-leaf-<hash>/
    |               |-- swprofile.yaml
    |               |-- session-<uuid>-node-0.kmsg
    |               `-- benchmark-<suite>-<name>-<hash>/
    `-- sut-1/
        `-- swprofile-leaf-<hash>/
            |-- swprofile.yaml
            |-- session-<uuid>-node-0.kmsg
            `-- benchmark-<suite>-<name>-<hash>/

Every software-profile directory contains a kernel log for each session and node. Only leaf software-profile directories contain benchmark, repeat, and role output. Internal profile directories also contain their child SUT directories. A plan with no nested SUTs keeps the existing layout unchanged: benchmark directories remain directly below the root profile.

Failure and Cleanup Policy

The first version is fail-fast for the whole topology:

A validation failure occurs before child launch where possible.
A child launch, connection, configuration, or benchmark orchestration failure cancels the remaining work for that topology.
Every successfully started VM is registered immediately for cleanup.
Cleanup always runs in finally blocks, in post-order, so descendants are stopped before their hosting parent.
Sibling cleanup may run concurrently and stop operations are idempotent.
Cleanup errors are logged without replacing the primary exception; if there is no primary exception, cleanup failure makes the command fail.

Benchmark failures already represented by ERROR rows remain benchmark outcomes rather than orchestration exceptions. All leaves complete the current repeat so the concurrent cohort remains meaningful.

Resultstore Model

Add two nullable hosting references to NODE:

NODE.parent_node_id -> NODE.id
NODE.parent_swprofile_id -> SWPROFILE.id

NULL identifies a physical/root node. A non-NULL value identifies the immediate node which hosts this node’s VM and the software profile running on that parent when the VM is hosted. In the initial implementation every nested SUT has one node and that node references the parent’s sole node and single active profile.

The profile reference is required because internal parent SUTs run no benchmarks and therefore have no RESULT.swprofile_id of their own. Without it, the resultstore would retain physical placement but lose the host kernels and settings which enabled each level of nested virtualization. Starting with a leaf result, its own RESULT.swprofile_id plus the hosting references on each ancestor edge reconstruct the complete software stack.

This is preferred to SUT.parent_node_id because it avoids a circular foreign-key dependency between SUT and NODE and naturally extends to a future multi-node child whose nodes may be hosted by different parent nodes. The SUT hierarchy can be reconstructed through each node’s owning SUT.

Update ORM relationships, CSV schema documentation, exports, imports, merge, and deduplication. A node’s parent node and parent profile must participate in merge identity so otherwise identical VMs under different host configurations do not collapse. Gathering and merging must include ancestor nodes, SUTs, and hosting profiles even when those internal SUTs have no results of their own. Nodes must be merged in root-to-leaf order so each mapped parent exists before its child.

Existing resultstores require an idempotent conversion utility, following the convert_rs_add_sutclass.py pattern:

Add nullable parent_node_id and parent_swprofile_id to NODE.csv, SQLite, and MySQL stores.
Leave all existing values NULL.
Add the database foreign key where the backend’s supported migration path permits it; application-level integrity checks remain required for CSV.

The hierarchy must not change the meaning of RESULT.sut_id, ERROR.sut_id, or RMDESC.node_id. Results and role maps point to the leaf SUT and leaf nodes on which the benchmark actually ran.

Scheduler Behavior

The scheduler continues to acquire only the root SUT. Nested capacity is owned by that job and is not represented as independently schedulable resources. The scheduler does not resolve nested sutclass values or allocate nested providers; plan exec validates the explicit VM parameters and the concrete parent’s runtime capabilities.

The current scheduler splits jobs by top-level profile, benchmark, and session. That behavior may remain for the first version, with the consequence that the VM tree is recreated for each split job. Avoiding that cost would require a larger scheduler lifecycle change and is outside this feature.

Security

Generate ephemeral SSH credentials locally with owner-only private-key permissions and remove them during cleanup.
Continue shell-quoting every value passed to Docker and QEMU.
Do not serialize private keys, generated ports, or gateway objects into job plans or resultstores.
Validate generated container names, hostnames, and ports before use.
Treat the VM image as trusted infrastructure because it runs privileged with host networking and access to KVM.

Implementation Plan

Phase 1: Recursive plan model

Add recursive suts validation and normalization.
Add and normalize the explicit params schema.
Add helpers to walk all profiles, nested entries, and leaves.
Update sorting and plan show output for recursive profiles.
Update scheduler file validation, upload rewriting, and job splitting.
Add valid, invalid, normalization, and arbitrary-depth plan tests.

Phase 2: Resultstore hierarchy

Add both NODE hosting foreign keys and ORM relationships.
Make merge gather ancestors and insert nodes in topology order.
Update CSV fixtures, schema documentation, filtering/import/export tests.
Add an idempotent existing-resultstore conversion utility.
Test two identical child shapes under different parents remain distinct.

Phase 3: Reusable VM factory

Implement host-bound launch/stop logic in a reusable VM factory.
Represent each VM with a side-effect-free QemuVm constructor, explicit start() and stop() methods, and context-manager cleanup.
Add host and VM-parameter validation without KVM or capacity preflight.
Add ephemeral key generation, injection, permissions, and cleanup.
Unit-test command construction, rollback, quoting, and VM-parameter checks.

Phase 4: Hierarchical SSH

Add gateway-aware machine connections and subconnections.
Connect child VMs through their immediate parent.
Test direct, one-gateway, and multi-gateway command execution with mocks.
Verify reconnect and reboot preserve the gateway chain.

Phase 5: Topology lifecycle

Extend each runtime Sut with its parent, children, active profile, VM, and log-directory state.
Provision/configure preorder and clean up post-order.
Reject multi-node hosting and unsupported parent architectures before launch; leave KVM capability validation to QEMU.
Add topology-qualified logging and leaf-aware progress totals.
Test partial startup/configuration failures clean every started descendant.

Phase 6: Concurrent leaf execution

Share benchmark models while creating per-SUT role maps and log directories.
Refactor repeats to return objects for coordinator-owned resultstore merges.
Run leaves concurrently with a barrier between repeats and benchmarks.
Reboot leaves only and keep internal parents running.
Add deterministic tests proving overlap and barrier ordering.

Phase 7: End-to-end coverage and documentation

Extend the plan-exec harness for one-level, sibling, and two-level trees.
Verify result/error ownership, role maps, hierarchy, logs, and cleanup.
Add an opt-in KVM integration test gated on writable /dev/kvm.
Document plan syntax, VM parameters, restrictions, and failure messages.
Confirm plans without suts produce unchanged execution and results.

Acceptance Criteria

Existing non-nested plan tests remain unchanged and pass.
A one-level plan creates all declared VMs, configures them, runs benchmarks only on them, records parent nodes, and removes every VM.
A two-level plan reaches the grandchild through chained SSH gateways and runs the benchmark on the grandchild only.
Sibling leaves overlap during each repeat and synchronize before the next.
A QEMU/KVM startup failure retains QEMU diagnostics, identifies the child by topology path, and removes any siblings which already started.
Multi-node parents, invalid VM parameters, and unsupported parent architectures fail with a topology path identifying the invalid declaration.
Failure at any lifecycle stage leaves no Fastpath VM containers running.
CSV, SQLite, and MySQL resultstores can represent and merge the hierarchy.

Review Decisions

The following choices should be confirmed before implementation:

Put host-bound VM parameters directly in each nested SUT declaration while retaining sutclass only as logical/resultstore classification.
Generate one ephemeral SSH keypair per plan exec invocation rather than requiring configured child credentials.
Represent placement and host software with NODE.parent_node_id and NODE.parent_swprofile_id rather than a SUT-level parent foreign key.
Define concurrency at repeat granularity with barriers between repeats.
Fail the whole topology on orchestration failure rather than retaining a partial set of leaves.
Recreate nested VMs for every scheduler-split job in the first version.
Permit deliberate CPU and memory overcommit while rejecting non-AArch64 VM hosts in the first version.