2026.10.14Pilot edition13 min read

Noise, Direction, and the Cost of Experiments

Muon's batch-size effects · Active experimental design for scaling laws

From direction-dependent gradient noise to the compute cost of discovering scaling laws: two main readings, two related updates, and one classic—S4.

MuonScaling laws

Daily classic · 2022S4

Main reading 01Relevant sections and author commentary read

Batch size and direction-dependent optimization dynamics

The Best Optimizer Depends on Batch Size

Xingyu Dang · Kaiyue Wen · Sadhika Malladi

First posted Oct 6 · v2 Oct 12

With a fixed sample budget $T$ , a larger batch size $B$ reduces gradient noise but also reduces the number of updates. The paper separates these effects using a noisy quadratic model.[1]

Model

f(w)=\frac12\sum_i h_iw_i^2,\qquad \widehat g_i=h_iw_i+\sqrt{\frac{c_i}{B}}\,z_i,\qquad K=\frac{T}{B}.

$h_i>0$ : curvature; $c_i>0$ : per-sample gradient-noise variance; $z_i\sim\mathcal N(0,1)$ . The direction-dependent curvature-to-noise ratio is $\mathrm{CNR}_i=h_i^2/c_i$ .

Ideal Muon reduces to a sign operation on a $1\times1$ matrix. Start with SignSGD with momentum:

Update

m_{t+1}=\mu m_t+(1-\mu)\widehat g_t,\qquad w_{t+1}=w_t-\eta\,\operatorname{sign}(m_{t+1}),\quad 0\le\mu<1.

Local mechanism. Hold $w_i$ fixed and let momentum reach stationarity. Under independent Gaussian noise,

Response

\mathbb E[\operatorname{sign}(m_i)]=\operatorname{erf}(q_i),\qquad q_i=w_i\sqrt{\frac{B(1+\mu)h_i^2}{2(1-\mu)c_i}}.

The mean displacement per processed sample is proportional to $\eta\,\operatorname{erf}(q_i)/B$ . Preserving this local displacement gives:

Two regimes

\eta(B)\propto \begin{cases}\sqrt B,& |q_i|\ll1 \quad \text{noise-dominated},\\[2pt] B,& |q_i|\gg1 \quad \text{signal-dominated}.\end{cases}

Directional intervention. Starting from the same checkpoint, increase the batch from $128\mathrm K$ to $2\mathrm M$ , retaining small-batch updates only in selected directions. The recovered fraction of the local loss gap is

Metric

R=\frac{L_{\rm large}-L_{\rm mixed}}{L_{\rm large}-L_{\rm small}}.

Directions retaining small-batch updates	Local gap recovered
The 768 sharpest directions, about $0.001\%$	Up to 59.5%
Random subspace of the same dimension	Change of at most about 3%

Local curvature diagnostic on the same data stream. Sharp directions come from a Muon-preconditioned curvature approximation.[2]

Related updateAbstract only

Noise can smooth the sign—and change the order of feature learning

From Parallel to Sequential: Small-Batch Noise Restores Simplicity Bias of Muon

Tom Jacobs · Rebekka Burkholz

arXiv Oct 12 · OpenReview record dated Oct 1

The authors report that small-batch noise restores sequential feature learning absent from full-batch spectral descent, and formalize the mechanism in deep linear models.[3][4]

The following illustrative scalar derivation explains noise smoothing. For $z\sim\mathcal N(0,1)$ and $\sigma>0$ ,

Scalar intuition

\begin{aligned}\mathbb E[\operatorname{sign}(g+\sigma z)]&=2\Phi(g/\sigma)-1\\&=\sqrt{\frac2\pi}\frac g\sigma+O\!\left((g/\sigma)^3\right),\qquad |g|\ll\sigma.\end{aligned}

Main reading 02Relevant sections read

Discovering a scaling law has its own compute budget

SLDBench v2: Active Scaling Law Discovery

Lingbei Meng et al.

First posted Oct 12

Background

Chinchilla (2022) used small-scale training runs to study compute allocation between model size and data. SLDBench (2025; ICLR 2026) then asked agents to discover predictive laws automatically. The new question: obtaining the fitting data itself costs compute.[6][7]

Candidate law

L(N,D)=E+aN^{-\alpha}+bD^{-\beta}.

The expression above is only a candidate form. The agent neither knows the correct functional form nor gets all observations for free; it receives a catalog of priced experiments.[5]

Experiment catalog

\mathcal U=\{(x_i,c_i)\}_{i=1}^{n},\qquad c_i>0.

$x_i$ is a training configuration; $c_i$ is usually its training FLOPs. The outcome $y_i$ is revealed only after purchase. Omitting group indices, the agent buys $A_t$ each round and stops at time $\tau$ :

Spending

S_{t+1}=S_t\cup A_t,\qquad \rho=\frac{\sum_{i\in S_\tau}c_i}{\sum_{i\in\mathcal U}c_i}.

Evaluation records $(\text{extrapolation error at larger scales},\,\rho)$ . The agent must jointly decide which experiments to buy, when to stop, and which law to fit. Test configurations are larger than candidate experiments and cannot be purchased directly.[5]

Setting	Range of median $R^2$ across systems	Experiment cost
Passive · all observations	0.723–0.753	100%
Active · chosen purchases	0.235–0.570	About 8%–44%, averaged within each system

30 selected difficult tasks and 8 agent systems. The R² range is across systems, not a confidence interval.[5]

BriefAbstract only

Monarch-Muon: break a large orthogonalization into smaller blocks

Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining

Alexander Miasnikov et al.

arXiv Oct 12 · OpenReview record dated Oct 1

Monarch factors impose structure on the weights, reducing weight, gradient, and optimizer-state storage while producing smaller update subproblems. The abstract reports these measurements for a 6.89B system:[8][9]

Measurement	Original setup	Monarch-Muon
Peak memory	68.9 GB	52.9 GB
Muon optimizer step	1288 ms	304 ms

Daily classic · 001Relevant method, theory, and experiment sections read

S4: one state space, three ways to compute

Efficiently Modeling Long Sequences with Structured State Spaces

Albert Gu · Karan Goel · Christopher Ré

First posted Oct 2021 · ICLR 2022 · reading v3 (Aug 2022)

ICLR 2022 Oral · Outstanding Paper Honorable Mention ↗

Where it comes from

Long sequences require both memory and efficient training. HiPPO (2020) designed continuous memory through polynomial projections; S4 (2022) sought a state-space parameterization that is fast to compute and numerically stable.[10][13]

Consider one channel with scalar input and output, and hidden state $x(t)\in\mathbb R^N$ . $A$ governs memory evolution, $B$ writes the input, and $C$ reads the state. Omit the separately computable skip term.[12]

Continuous system

\dot x(t)=Ax(t)+Bu(t),\qquad y(t)=Cx(t).

Apply a bilinear discretization with step size $\Delta>0$ , assuming the inverse exists. $\Delta$ is the sampling timescale and can be learned as a model parameter.[12]

Discretization

\bar A=\left(I-\frac{\Delta}{2}A\right)^{-1}\left(I+\frac{\Delta}{2}A\right),\qquad \bar B=\left(I-\frac{\Delta}{2}A\right)^{-1}\Delta B.

With zero initial state $x_{-1}=0$ and fixed parameters, stepwise recurrence and full-sequence convolution are exactly equivalent:[12]

RNN ↔ convolution

\begin{aligned}x_k&=\bar A x_{k-1}+\bar B u_k,\\ y_k&=\sum_{j=0}^{k}K_j u_{k-j},\qquad K_j=C\bar A^j\bar B.\end{aligned}

Computing the kernel is the real bottleneck. Direct eigendecomposition of a HiPPO matrix can be severely ill-conditioned. S4 uses a normal matrix plus a low-rank correction, then a unitary change of basis gives DPLR:[10]

Structured parameterization

A=V(\Lambda-PQ^*)V^*,\qquad V^*V=I,\qquad P,Q\in\mathbb C^{N\times r},\quad r\ll N.

$\Lambda$ is diagonal. Write $\widehat A=\Lambda-PQ^*$ and $R(\omega)=(\omega I-\Lambda)^{-1}$ . When the relevant inverses exist, Woodbury’s identity gives:[10]

Low-rank correction

(\omega I-\widehat A)^{-1}=R-RP(I+Q^*RP)^{-1}Q^*R.

A high-dimensional inverse becomes a diagonal inverse plus an $r\times r$ correction. The paper reduces the finite-length kernel’s generating function to Cauchy computations and an inverse FFT. For fixed low rank, theoretical kernel construction costs $\widetilde O(N+L)$ , and each recurrent step costs $O(N)$ . These are per-SSM costs and assume a fast Cauchy algorithm.[10]

Path-X: connectivity classification over length 16,384	Accuracy
Random guessing	50.00%
S4 original	88.10%

The “original” row in v3 Appendix Table 10; later improvements are listed separately as “updated.” Original Path-X setup: 6 layers, 256 channels, state dimension 64 per channel, one A100 40GB.[10]

READ THE ORIGINALS

Original sources

Search window

2026-10-12 — 2026-10-14 · America/New_York

READ · THINK · DISCUSS

Discussion & likes

Discuss a derivation, add a reference, or suggest the next classic.

Comment, reply, or like below. Sign in with GitHub to participate; comments are public.

Loading discussion…

Open discussion on GitHub ↗

← Back to all issues