Regression Tracker

Introduction

Fastpath monitors the Linux kernel across a range of hardware platforms and workloads. When a statistically significant performance change is detected between two kernel versions, it is raised as a tracked regression entry and investigated. This page records all known findings together with their current status and, where available, links to upstream reports and fix commits.

Column descriptions:

  • ID – Tracking identifier (e.g. FP-619-02).

  • Branch / Good / Bad – The kernel tree and the two versions that bracket the regression.

  • SUT(s) – System(s) Under Test on which the regression was observed.

  • Benchmark / Resultclass – The specific workload and measured quantity.

  • Delta – Measured change between the good and bad kernel versions.

  • Regressing Commit – The first bad commit identified by bisection.

  • Status – One of four states:

    • Detected – Regression observed; queued for investigation.

    • Investigating – Currently under active investigation.

    • Reported – Reported upstream; awaiting a fix.

    • Fixed – Fix merged upstream.

  • Report – Link to the upstream mailing-list report or patch.

  • Fix Commit(s) – Commit(s) that resolved the regression.

Regression History

ID

Branch

Good Kernel

Bad Kernel

SUT(s)

Benchmark / Resultclass

Delta

Regressing Commit

Status

Report

Fix Commit(s)

FP-614-01

mainline

6-13-0

6-14-0-rc4

aws-m7g.metal-01

mmtests/hackbench

(process-pipes-30)

-10.69%

aaec5a95d596

“pipe_read: don’t wake up the writer if the pipe is still full”

Fixed

Upstream report

FP-615-01

mainline

6-15-0-rc1

6-15-0-rc2

aws-m7g.metal-01

pts/pgbench

Scale: 100 Clients: 1000, Read Only (TPS)

-98.81%

7ab4f0e37a0f

“ACPI PPTT: Fix coding mistakes in a couple of sizeof() calls”

Fixed

Upstream report

adfab6b39202

“ACPI: PPTT: Fix processor subtable walk”

FP-619-01

mainline

6-18-0

6-19-0-rc1

aws-m7g.metal-01

micromm/vmalloc

fix_align_alloc_test: p:1, h:0, l:500000 (usec)

-40.57%

a06157804399

“mm/vmalloc: request large order pages from buddy allocator”

Reported

Upstream report

FP-619-02

mainline

6-18-0

6-19-0-rc1

aws-m7g.metal-01

cesw-aarch64-ampereone-1s-a192-32x-02

schbench/thread-contention

(-m 64 -t 4 -r 10 -s 1000), avg_rps (req/sec)

-17.41%

e837456fdca8

“sched/fair: Reimplement NEXT_BUDDY to align with EEVDF goals”

Fixed

Upstream report

4f70f106bca1

“sched/fair: Disable scheduler feature NEXT_BUDDY”

15257cc2f905

“sched/fair: Revert force wakeup preemption”

(merged in 6-19-0-rc7)

FP-619-03

mainline

6-18-0

6-19-0-rc2

aws-m7g.metal-01

micromm/munmap

p:1, d:10 (seconds)

-23.74%

7e44d00a13ca

“memcg: use mod_node_page_state to update stats”

Reported

Upstream report

FP-700-01

mainline

6-19-0

7-0-0-rc1

aws-m7i.24xlarge-01

aws-m7g.metal-01

cesw-aarch64-ampereone-1s-a192-32x-02

micromm/vmalloc

kvfree_rcu_2_arg_vmalloc_test: p:1, h:0, l:500000 (usec)

-9.26%

e47c897a2949

“slab: add sheaves to most caches”

Investigating