Full RMS-scaled softmax results

Both frozen studies are included. Every row has 131,072 values. Values are microseconds per invocation.

Within each study, A1 and A2 identify two separately provisioned GPU measurement sessions, not CUDA memory allocations or individual timing samples. Each session has 21 warm rounds per variant. These tables keep the sessions separate; the main article averages the two session medians from the final study, with whiskers spanning those medians.

Warm results: 21 shuffled rounds, 32 invocations per timed batch. Host completion times one invocation plus stream synchronization; event intervals time batches and can contain host supply gaps. Host enqueue excludes completion; host batch completion amortizes the host wait across the batch. Allocations, transfers, compilation, graph setup and profiling are outside these warm measurements.

The baseline is a deliberately explicit nine-kernel educational decomposition. It is not a stock-framework baseline. Local and retained paths also use graph replay unless labeled ordinary launches. Local3 means three kernels; Local2/Local1 reduce launches by repeating work. The cuDNN pipeline includes the benchmark RMS preprocessing and is not exhaustively tuned.

Every timing sample · Every measured p10/median/p90

The saved studies retain setup/capture/instantiate/first-replay records, device records, compiled resources, pilot candidates, traces and profiler counters. This website includes selected evidence; complete records remain in the research archives. The CSVs here contain all steady-state and external-eviction timing samples, including empty synchronization calls. No new GPU run was performed.

row_scope_followup: 20261020T023055Z

Methods and selected evidence

1 row(s): all warm variants

Scroll horizontally to see the full table.

Scroll horizontally to see the full table.

16 row(s): all warm variants

Scroll horizontally to see the full table.

Scroll horizontally to see the full table.

External-eviction diagnostic: a separate buffer of at least 256 MiB is touched before each invocation, outside the measured interval. This is a cache-eviction experiment, not a guarantee that every cache or metadata entry is cold. Fifteen rounds per variant.

Scroll horizontally to see the full table.

32 row(s): all warm variants

Scroll horizontally to see the full table.

Scroll horizontally to see the full table.

64 row(s): all warm variants

Scroll horizontally to see the full table.

Scroll horizontally to see the full table.

External-eviction diagnostic: a separate buffer of at least 256 MiB is touched before each invocation, outside the measured interval. This is a cache-eviction experiment, not a guarantee that every cache or metadata entry is cold. Fifteen rounds per variant.

Scroll horizontally to see the full table.

initial_full_grid: 20261020T022150Z

Methods and selected evidence

1 row(s): all warm variants

Scroll horizontally to see the full table.

Scroll horizontally to see the full table.

16 row(s): all warm variants

Scroll horizontally to see the full table.

Scroll horizontally to see the full table.

External-eviction diagnostic: a separate buffer of at least 256 MiB is touched before each invocation, outside the measured interval. This is a cache-eviction experiment, not a guarantee that every cache or metadata entry is cold. Fifteen rounds per variant.

Scroll horizontally to see the full table.

32 row(s): all warm variants

Scroll horizontally to see the full table.

Scroll horizontally to see the full table.

64 row(s): all warm variants

Scroll horizontally to see the full table.

Scroll horizontally to see the full table.

External-eviction diagnostic: a separate buffer of at least 256 MiB is touched before each invocation, outside the measured interval. This is a cache-eviction experiment, not a guarantee that every cache or metadata entry is cold. Fifteen rounds per variant.

Scroll horizontally to see the full table.

Method and interpretation

The semantic operator shared by every measured implementation is:

import numpy as np

def rms_scaled_softmax(x, eps=1e-5):
    inv_rms = 1 / np.sqrt(
        np.mean(x * x, axis=1, keepdims=True) + eps
    )
    exp_scores = np.exp(
        (x - np.max(x, axis=1, keepdims=True)) * inv_rms
    )
    return exp_scores / np.sum(
        exp_scores, axis=1, keepdims=True
    )

RMS scaling changes the operator relative to ordinary softmax. This is a constructed teaching workload, not a claim that production attention normally contains this exact sequence. Tiny cross-block statistics still use global memory; the retained path keeps the large tile buffers in shared memory.

The latest study selects the full-grid retained kernel for one row and row-scoped retention for 16/32/64 rows using the pilot. Both variants and ordinary-launch controls remain visible above. At 64 rows both retained implementations use two cooperative launches. A graph and a single kernel are distinct concepts; a single-kernel graph need not beat its ordinary launch.

The predeclared 1.25× primary-case event speedup was not achieved. The fastest tested local family also had the lowest held-out event medians among the tested local alternatives. The one-/two-kernel alternatives remain valuable controls because fewer kernels can require more repeated work.

Resource queries constrain the interpretation at 32 rows: the row-scoped binary reports 32 registers/thread and an occupancy limit of eight active blocks/SM; the full-grid binary reports 34 and a limit of six. These are capacity limits, not observed occupancy. Their selected tile widths differ, so the comparison does not isolate barrier cost.

The initial study used only full-grid synchronization. Its measurements are preserved separately rather than pooled with the row-scoped follow-up. The latest reported GPU ledger includes both studies; all GPU work was already torn down and archived.