Leased and Nested SUT SSH Access Design

Status

Proposed design for staged implementation.

Summary

Fastpath maintains ~/.fastpath/suts/fastpath-suts.conf with OpenSSH aliases for active scheduler leases and nested SUTs. A one-time Include in ~/.ssh/config makes them available to the normal ssh command.

For a manual lease, the client generates one ephemeral key pair and sends only the public key to the scheduler. The scheduler installs it on every node before reporting the lease as acquired and attempts to revoke it from every node during release. The private key never leaves the client.

Nested SUT entries use their existing client-generated keys and contain the complete ProxyJump chain. The same aliases can be passed to commands such as fastpath sut install, provided Fastpath explicitly loads the generated file for Fabric.

Goals

  • Support ssh <name> for active leased and nested SUTs.

  • Accept the same names wherever Fastpath accepts an SSH host.

  • Limit lease-owner access to the lease lifetime when revocation succeeds.

  • Keep private lease keys out of the scheduler and its API.

  • Update generated files safely under concurrent CLI processes.

  • Preserve scheduler-owned credentials used for automation.

Non-goals

  • Replacing scheduler credentials used for plans and recovery.

  • Sharing or recovering a lease private key on another client.

  • Replacing QEMU user networking with TAP or routed networking.

  • Providing direct access to non-SSH SUT services.

  • Introducing SSH certificates in the first version.

Design

Lease credentials

fastpath sched lease-acquire generates one Ed25519 key pair for the lease. The private key is stored beneath ~/.fastpath/suts/keys with mode 0600. The lease request contains the public key, which the scheduler validates and stores with the lease. The public key and lease id are sufficient for installation, removal, and diagnostics.

The scheduler uses its existing SUT credentials to add a marked entry to the existing connection user’s ~/.ssh/authorized_keys on every node in the SUT. It verifies every installation before changing the lease to acquired. On release, it attempts to remove and verify removal of the entry on every node before invoking the provider’s existing release operation.

The credential lifecycle belongs to the scheduler rather than individual SUT providers. A provider hook may be added later for environments using cloud metadata, AuthorizedKeysCommand, an SSH certificate authority, or another access-control service.

Generated SSH configuration

Fastpath owns:

~/.fastpath/                         mode 0700
~/.fastpath/suts/                    mode 0700
~/.fastpath/suts/fastpath-suts.conf  mode 0600
~/.fastpath/suts/keys/               mode 0700

The user opts in by placing this near the start of ~/.ssh/config, before broad Host * defaults:

Include ~/.fastpath/suts/fastpath-suts.conf

An install command may offer to add the line, but must not modify the user’s SSH configuration without consent.

The generated file is rebuilt from local active-SUT state, then atomically renamed into place. A single-node leased SUT uses its SUT name as the alias. A multi-node leased SUT appends -node<n> using a one-based node index. A nested SUT uses the VM name printed by fastpath sut create. Fastpath reports an error instead of overwriting an active entry if two aliases collide.

Entries use IdentityFile and IdentitiesOnly yes. Nested entries also use HostKeyAlias with the VM name so guests reached through common addresses such as localhost:8022 do not collide. The first version uses the user’s normal OpenSSH known-hosts handling rather than maintaining a separate file; there is therefore no additional known-hosts plumbing.

Nested SUTs

The parent of a nested SUT may be:

  • a scheduler-leased SUT name from the generated configuration;

  • another nested SUT name; or

  • a host, user, port, and key file supplied directly to fastpath sut create.

An existing alias is used directly as the jump host. For direct connection arguments, Fastpath creates an internal parent entry so that user, port, and key file settings remain available after sut create exits. Each nested entry contains all ancestor aliases in root-to-parent order.

For example, if lab-sut is the scheduler-provided SUT name and fastpath-sut-a1b2c3d4e5f6 is the name assigned by sut create:

Host lab-sut
    HostName sut.example.com
    User ubuntu
    IdentityFile ~/.fastpath/suts/keys/lease-42
    IdentitiesOnly yes

Host fastpath-sut-a1b2c3d4e5f6
    HostName localhost
    User ubuntu
    Port 23456
    IdentityFile ~/.fastpath/suts/fastpath-sut-a1b2c3d4e5f6/id_ed25519
    IdentitiesOnly yes
    HostKeyAlias fastpath-sut-a1b2c3d4e5f6
    ProxyJump lab-sut

Host fastpath-sut-123456789abc
    HostName localhost
    User ubuntu
    Port 34567
    IdentityFile ~/.fastpath/suts/fastpath-sut-123456789abc/id_ed25519
    IdentitiesOnly yes
    HostKeyAlias fastpath-sut-123456789abc
    ProxyJump lab-sut,fastpath-sut-a1b2c3d4e5f6

Destroying a nested SUT removes its entry, key, and all descendant entries.

Fastpath and Fabric

Fabric supports the required HostName, User, Port, IdentityFile, and multi-hop ProxyJump settings. Paramiko treats an Include directive as an ordinary option and does not read the referenced file. This has been verified against the version used by Fastpath.

Fabric can be pointed at one runtime SSH configuration file, but doing so skips the normal user and system files. Fastpath should instead construct a Paramiko SSHConfig object, parse the generated, user, and system files into it in the required precedence order, and pass that object to fabric.Config. This keeps existing user configuration while making generated names available to SSHMachine.

Lifecycle

Acquire

  1. The client generates and stores a lease key, then submits its public key.

  2. The scheduler acquires the SUT and completes provider initialization.

  3. The scheduler installs and verifies the key on every leased node.

  4. The scheduler reports the existing lease and SUT metadata as acquired.

  5. The client uses the returned SUT name and connection details to regenerate its SSH configuration.

No additional fields are required in lease-list output. The generated alias is derived from the SUT name, and the client already owns the corresponding private key.

If installation fails, the scheduler attempts to remove the key from every node already modified, releases the SUT through the existing path, and marks the lease failed. The CLI reports that credential installation failed, while the scheduler log identifies the lease and nodes on which installation or rollback failed.

Release

  1. The scheduler marks the lease as releasing.

  2. It attempts to remove and verify removal of the lease key from every node.

  3. It reports any revocation errors clearly in the scheduler log.

  4. It releases the SUT through the provider’s existing release path regardless of revocation success, then completes the lease normally.

  5. The client removes its local key, state, and alias after the scheduler accepts its lease release request.

This best-effort revocation keeps the existing scheduler state machine and SUT release behavior. It accepts the small risk that a failed revocation leaves the lease key installed; the error must include the lease, SUT, and affected node so an operator can remediate it.

The public key is durable lease data. On scheduler restart, an active manual lease retains its already installed key. Recovery of a releasing lease attempts revocation before following the existing provider release path. No separate credential states or retry state machine are required.

If the client loses its private key, the first version requires releasing and reacquiring the SUT. The scheduler never returns or reconstructs the key.

API and Data Model

POST /leases requires a top-level public_key field:

{
  "sutclass": "myclass",
  "priority": 5,
  "public_key": "ssh-ed25519 AAAAC3... fastpath-lease"
}

LeaseJob gains only the validated public key. Existing job and lease states represent acquisition, failure, release, and recovery. Responses never include the private key or the scheduler’s keyfile path, and lease-list response shape is unchanged.

Security

  • Accept only supported key algorithms, initially Ed25519.

  • Parse public keys and reject client-supplied authorized_keys options.

  • Do not write public keys, bearer tokens, or private data to logs.

  • Require TLS for scheduler traffic because bearer tokens and SUT details are sensitive even though private keys are not transmitted.

  • Update authorized_keys atomically and report failed revocation clearly.

  • Treat scheduler administrators as privileged because scheduler credentials already permit SUT access.

Implementation Plan

The implementation starts by limiting scheduler-leased SUT access to the lease lifetime. It then builds the generated configuration and CLI integration around those credentials. Nested SUTs remain deferred until the leased-SUT path is complete.

Phase 1: ephemeral lease credentials

  • Require a top-level public_key in the lease API and LeaseJob.

  • Generate one Ed25519 key per lease on the client, store the private key under ~/.fastpath/suts, and store only its public key on the scheduler.

  • Install and verify the key on every leased node before acquisition completes.

  • Attempt removal on every node before the existing provider release path.

  • Remove the local private key after confirmed release, failure, or cancellation.

  • Test malformed keys, multi-node partial failures, restart recovery, API redaction, local file permissions, and clear installation or revocation errors.

This phase changes lease authentication but does not generate SSH configuration or alter Fabric behavior. The existing lease API response remains unchanged.

Phase 2: leased SUT configuration renderer

  • Add a small utility that renders OpenSSH Host entries from acquired lease metadata already returned by the scheduler.

  • Use the SUT name as the alias for a single node and append -node<n> for a multi-node SUT, and emit each node’s HostName, User, and Port when present.

  • Emit the lease key’s IdentityFile and IdentitiesOnly yes.

  • Write ~/.fastpath/suts/fastpath-suts.conf atomically with mode 0600; create parent directories with mode 0700.

  • Reject duplicate or invalid aliases instead of silently overwriting an entry.

  • Unit test rendering, deterministic ordering, permissions, empty lease sets, multi-node leases, and collisions.

This phase has no SSH Include, Fabric, scheduler, or nested-SUT changes. It only provides a tested renderer and file-update operation.

Phase 3: leased SUT synchronization

  • Query acquired leases owned by the current user and rewrite fastpath-suts.conf from that authoritative snapshot after every successful lease-list, lease-acquire, and lease-release operation.

  • Always use a separate states=acquired&users=__current__ request so the displayed lease-list filters cannot produce an incomplete configuration.

  • Remove entries that are no longer present in the scheduler response, so an interrupted acquire or release does not leave permanent stale aliases.

  • Associate each acquired lease with the local private key created in phase 1; report a clear error if that key is missing.

  • Report the generated path, number of nodes, and any name collision clearly.

  • Keep the existing lease API and lease-list response unchanged.

  • Test queued, acquired, released, failed, and multi-node lease responses.

At the end of this phase, normal lease operations maintain a file containing the host names of all currently acquired leased SUTs. The file is not yet loaded by OpenSSH or Fabric automatically.

Phase 4: Fastpath command integration

  • Make SSHMachine load fastpath-suts.conf in addition to normal user and system SSH configuration, accounting for Paramiko’s lack of Include support.

  • Verify aliases with fastpath sut fingerprint command.

  • Keep this change limited to direct leased-SUT entries; do not add jump-host or nested-SUT behavior yet.

Phase 5: nested SUT aliases

  • Register the VM name, connection data, key, and parent relationship created by fastpath sut create.

  • Support parents supplied as leased aliases, other nested aliases, or direct connection arguments.

  • Add unique HostKeyAlias values and arbitrary-depth ProxyJump chains.

  • Remove descendants and keys on destroy and rollback.

  • Keep nested-SUT state separate from the authoritative leased-SUT snapshot so lease synchronization preserves entries for active leases and direct hosts. When a leased SUT is removed from the snapshot or its alias is assigned to a new lease, remove its nested descendants and their local keys.