ZHZixun Huang
Research DigestNo. 008
Daily research notes22 min read

Step Size, Scale, and Pareto Frontiers

MuPACE · Constant-step Muon · Pareto frontier · InstructGPT

Three main readings: learning Muon's global step size, classifying constant-step spectral optimizers by gain and length, and fixed-confidence identification of a multi-objective LLM Pareto frontier. Twelve scaling, data, and test-time-compute briefs follow. The classic reading derives the RLHF objective in InstructGPT.

MuonScaling lawsPost-trainingStatistical learning
Daily classic · 2022InstructGPT
Main reading · New preprintRead §§3–5 and appendix ablations; meta-training not reproduced

MuPACE: Muon chooses the direction; the trajectory chooses the step

MuPACE: Learning to Pace Muon for LoRA Fine-Tuning
Jan Kobiolka · Christian Frey · Josif Grabocka
First posted 2026-10-20 07:52 EDT · v1; entered the public list on Oct 21

A normalized spectral update separates direction from scale: Muon approximately supplies the polar direction, while the learning rate controls almost all of the length. MuPACE does not learn a new optimizer; a roughly 20K-parameter LSTM adjusts an analytic schedule from the observed trajectory. [1]

Muon determines the matrix direction; the policy outputs only the global scalar ηₜ
st=UtΣtVt⊤,dt≈UtVt⊤,θt+1=θt−ηtdts_t=U_t\Sigma_tV_t^\top,\qquad d_t\approx U_tV_t^\top,\qquad \theta_{t+1}=\theta_t-\eta_t d_t
RMS normalization separates scale λ₁ from shape λ₂:₅; exp–tanh bounds each deviation
η(x)=λ1∥s0∥2∥s∥2s(x),λi,t=λi0exp⁡ ⁣(citanh⁡hi,t),x=t/T\eta(x)=\lambda_1\frac{\lVert s_0\rVert_2}{\lVert s\rVert_2}s(x),\qquad \lambda_{i,t}=\lambda_i^0\exp\!\big(c_i\tanh h_{i,t}\big),\qquad x=t/T
Depth extrapolation follows Depth-μP; validation loss enters the policy state and meta-objective, not the LoRA update gradient
ηt←(LrefLd)1/2ηt,min⁡ϕ  ET,τ∼πϕ1T−T0∑t>T0log⁡ℓ~tval\eta_t\leftarrow\left(\frac{L_{\rm ref}}{L_d}\right)^{1/2}\eta_t,\qquad \min_\phi\;\mathbb E_{\mathcal T,\tau\sim\pi_\phi}\frac{1}{T-T_0}\sum_{t>T_0}\log \widetilde\ell_t^{\rm val}
ComparisonResultBudget meaning
64 in-distribution problemsBeats cosine / WSD / constant defaults on 78% / 66% / 64%One MuPACE run
Versus per-task sweepsMedian recovery of tuning gain: 80% / 83% / 78%Several baseline trials per task
Qwen3-32B, Qwen3.8-27BMedian test-loss gap to 5-trial tuned baselines: 1.13–3.02%One versus five trials; eight tasks per model
Reported by the paper; test loss is measured at each arm's best-validation checkpoint. [1]
Main reading · New preprintRead Theorems 1 and 3, Lemma 4, and numerical experiments; results are restricted to strongly convex matrix quadratics

Constant-step Muon: gain and length decide convergence

Constant Step Sizes for Muon-Type Spectral Optimizers on Strongly Convex Quadratics
Pratik Jawanpuria · Bhabatosh Garai · Bamdev Mishra
First posted 2026-10-20 03:07 EDT · v1; entered the public list on Oct 21

The paper removes nonconvexity, noise, and schedules to ask whether a fixed-step spectral update reaches the optimum. For G=UΣVᵀ, the update preserves singular vectors and maps σᵢ to gᵢ. The key quantities are gain rᵢ=gᵢ/σᵢ and update length. [2]

H≻0; this is an exactly analyzable matrix quadratic, not a theorem for general neural networks
f(X)=12tr⁡[(X−X⋆)⊤H(X−X⋆)],Gt=H(Xt−X⋆),Xt+1=Xt−αΦ(Gt)f(X)=\tfrac12\operatorname{tr}[(X-X_\star)^\top H(X-X_\star)],\quad G_t=H(X_t-X_\star),\quad X_{t+1}=X_t-\alpha\Phi(G_t)
Gain controls local amplification; whether length vanishes decides whether a fixed-step method can actually stop
Φ(G)=Udiag⁡(gi)V⊤=P(G)G,ri=giσi,∥Φ(G)∥F=(∑igi2)1/2\Phi(G)=U\operatorname{diag}(g_i)V^\top=P(G)G,\qquad r_i=\frac{g_i}{\sigma_i},\qquad \lVert\Phi(G)\rVert_F=\Big(\sum_i g_i^2\Big)^{1/2}
RegimeRepresentative updatePaper result
c≤rᵢ≤CFinite-step NS normalized with ε>0Linear convergence if αλmaxC<2; no better local optimal rate than GD
Unbounded gain, length→0‖G‖* sign(G)Linear convergence if α tr(H)<2
Non-vanishing lengthPolar map; current-norm NS with ε=0No convergence for any α>0 from almost every initialization
The same NS polynomial can land in different regimes solely through normalization and scaling. [2]
Theorem 1 in the paper, for the bounded-gain regime
f(Xt+1)≤[1−u(2−u)cλmin⁡Cλmax⁡]f(Xt),u=αCλmax⁡∈(0,2)f(X_{t+1})\le\left[1-u(2-u)\frac{c\lambda_{\min}}{C\lambda_{\max}}\right]f(X_t),\qquad u=\alpha C\lambda_{\max}\in(0,2)
Main reading · Subscribed-author preprintRead the setup, Theorems 3.1–3.2, and real-Arena experiments; experiments not reproduced

Do not scalarize first: identify the LLM Pareto frontier at fixed confidence

Beyond Scalar Leaderboards: Adaptive Sampling for Pareto Frontier Identification in Multi-Objective LLM Arenas
Liang Twist Shan · Jiahao Ai · Zhimei Ren
First posted 2026-10-19 20:50 EDT · v1; entered the public list on Oct 21

Collapsing math, code, and writing into one leaderboard lets the weights choose values on the reader's behalf. VB-EGE treats each human comparison as a bandit query and aims to identify the strict Pareto frontier with confidence 1−δ, rather than maximize online cumulative reward. [3]

Multi-objective Bradley–Terry model; each query returns one binary preference on one objective
Pr⁡(Yt=1∣Qt=(i,j,r))=σ(θi,r−θj,r),P(θ)={i:∄j, θi,r<θj,r ∀r}\Pr(Y_t=1\mid Q_t=(i,j,r))=\sigma(\theta_{i,r}-\theta_{j,r}),\qquad \mathcal P(\theta)=\{i:\nexists j,\ \theta_{i,r}<\theta_{j,r}\ \forall r\}
Theorem 3.1; all classification gaps Δᵢ must be positive, and κ_B is the minimum BT-link derivative on a bounded score range
Pr⁡ ⁣(P^=P(θ), τ≤dK+128dκB2∑i=1KLiθ(δ)(Δiθ)2)≥1−δ\Pr\!\left(\widehat{\mathcal P}=\mathcal P(\theta),\ \tau\le dK+\frac{128d}{\kappa_B^2}\sum_{i=1}^K\frac{L_i^\theta(\delta)}{(\Delta_i^\theta)^2}\right)\ge1-\delta
Theorem 3.2: minimax lower bound on a gap-separated class; upper and lower bounds match in K, d, and Δ⁻² up to logarithms
QB,d(Δ,δ)≥d(K−1)Δ2kl⁡(1−2δ,δ)Q_{B,d}(\Delta,\delta)\ge\frac{d(K-1)}{\Delta^2}\operatorname{kl}(1-2\delta,\delta)
Brief · New preprintRead the main experiments and hyperparameter tables; training not reproduced

DPO hyperparameters do not transfer cleanly from small models

The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It
Hannan Mahadik · David Salinas · Jacek Golebiowski · Aaron Klein
First posted 2026-10-20 05:14 EDT · v1

A shared 3×3 grid over Qwen3 0.6B–14B gives negative cross-scale rank correlations in several pairs. Downstream ranks after roughly 30% of training still correlate 0.63–0.90 with final ranks. Validation loss is weakly aligned with the final downstream optimum; a short run on the target model is more reliable than transfer from a neighboring scale. [4]

Brief · New preprintRead the compute-allocation and test-time-scaling sections; a technical note, not one unified new theorem

A scaling-law note from geometric approximation to compute allocation

Foundations of the Scaling Law
Tianyu Zhao · Zhixiang Ren
First posted 2026-10-20 03:52 EDT · v1

The note connects geometric approximation, spectral visibility, N–D allocation, and test-time coverage. For L−L∞=AN^{-αN}+BD^{-αD}, C=κND, stationarity gives N*∝C^{αD/(αN+αD)} and D*∝C^{αN/(αN+αD)}. The authors explicitly warn that upper risk bounds cannot be treated as exact loss rates without another argument. [5]

Brief · New preprintBased on the abstract; full proofs not checked

Synthetic transfer: Wasserstein closeness can still fail completely

When Does Training on Synthetic Data Provably Work?
Leroy Odunlami · Tobias Riedlinger · Hanno Gottschalk
First posted 2026-10-20 07:23 EDT · v1

With shared conditional labels, a controlled density ratio gives a finite-sample excess-error bound. If the real distribution has mass outside synthetic support, an expressive classifier can approach worst-case real error even for Wasserstein-close distributions. Under Lipschitz assumptions, the guarantee is instead controlled by the probability that real data lie outside the synthetic reach. [6]

Brief · New preprintRead the shrinkage argument, loss decomposition, and main experiments

Under data repetition, random-token replacement is shrinkage

Benefit and Cost of Random-Token Replacement under Repeated Data
Wonsup Shin · Jingu Kim
First posted 2026-10-19 23:57 EDT · v1

For a first-order source, Q_c=α_cP_c+(1−α_c)π: replacement shrinks the conditional predictor toward the corpus marginal. With known entropy rate H, the authors decompose L_final=H+APPROX+EST+MEM and find that the benefit mainly suppresses post-minimum MEM, rather than improving minimum validation loss. [7]

Brief · New preprintBased on the abstract; full experiment tables not checked

Test-time compute: keep revising or resample?

Adaptive Replay Refinement: Allocating Test-Time Compute Between Revision and Resampling
Ahsan Bilal · Muhammad Ahmed Mohsin · Muhammad Umer · Ayesha Mohsin · Sarmad Bilal · Dean F. Hougen
First posted 2026-10-20 10:41 EDT · v1

ARR uses replay decoding to detect a revision fixed point, then reallocates released budget to an independent sample. Across seven models and seven benchmarks, the abstract reports 84.6% on MATH500 using 41% of majority voting's serial passes. Serial passes are not FLOPs or end-to-end latency. [8]

Brief · New preprintBased on the abstract; full cost tables not checked

TEX: scale testing agents with a K×K execution matrix

TEX: Test-Time Scaling Testing Agents via Execution-based Cross-Validation
Akash Gokul · Srijan Bansal · Yingbo Zhou · Semih Yavuz
First posted 2026-10-19 17:06 EDT · v1

K agents generate tests and patches in parallel, then cross-execute them to obtain a K×K feedback matrix. The abstract reports 86.4% on SWT-Bench Verified, while random selection among TEX outputs reaches 84.4%. The result supports candidate-quality gains from collaborative feedback, but K² execution cost needs a matched accounting. [9]

Brief · New preprintBased on the abstract; coefficients and all hardware settings not checked

Energy per token from active bytes and realized decode time

The Token-Energy Law of LLM Decode
Chenxu Niu · Batuhan Sencer · Wei Zhang · Yong Chen
First posted 2026-10-20 13:32 EDT · v1

An affine model combines shared dense-weight reads, request-private KV-history traffic, and realized decode time. Under BF16, three models, and Hopper/Blackwell settings, held-out condition-mean energy errors are at most 3.73%. This is a deployment-side empirical law, not a training-loss scaling law. [10]

Brief · New preprintBased on the abstract; eleven rewards not checked budget by budget

One policy for arbitrary test-time budgets

Rewards for Budget-Aware Multi-Turn Agents That Adapt to Arbitrary Test-Time Constraints
Tom A. Lamb · Adam Golinski · Eleonora Gualdoni · Sinead Williamson · Michael Kirchhof
First posted 2026-10-20 11:49 EDT · v1

Qwen3.5 4B/9B policies are trained across dynamic budgets in tokens, API calls, environment steps, and turns. Overspending penalties improve within-budget success and natural compliance, but sigmoid and fixed-negative penalties respectively favor success and compliance: reward design defines a Pareto trade-off. [11]

Brief · New preprintBased on the abstract; per-question statistical tests not checked

Subtask-level inverse scaling in VLMs

On the Limits of Scale: Inverse Scaling in Vision-Language Models
Rohit Saxena · Alessandro Suglia · Marek Rei · Pasquale Minervini
First posted 2026-10-19 18:39 EDT · v1

Nineteen models from four families across eight benchmarks produce 558,334 model-question evaluations. The authors report 375 subtask–model-pair inverse-scaling combinations and 13 subtask–dataset pairs replicated across families. Improving the aggregate with scale does not imply monotonicity on every capability slice. [12]

Brief · New preprintBased on the abstract; full size-generalization curves not checked

Scale graph reasoning by taking adjacency out of the prompt

Scaling Graph Reasoning in Language Models with Learnable Graph Positional Encodings
Javier Porras-Valenzuela · Arush Arora · Alejandro Ribeiro
First posted 2026-10-19 20:39 EDT · v1

A GNN plus Graph Transformer produces permutation-equivariant positional encodings that modulate attention logits, reducing context burden from serialized edges to O(N). Training uses small graphs and the abstract reports extrapolation to 1,000 nodes. This is structural size scaling, not a parameter–data power law. [13]

Brief · New preprintBased on the abstract; preregistered study, full trajectories not checked

Negative result: multi-agent collaboration fails a compute-matched test

Does Multi-Agent Collaboration Pay When Agents Serve Users?
Ethan Yang · Xiangcheng Shi
First posted 2026-10-20 01:13 EDT · v1

Across about 7,950 τ²-bench conversations, a same-model centralized MAS loses in 12 of 13 valid cells and ties one. Without a budget ceiling, it at best ties using 5.5× the tokens. The paper also discloses that its planner did not see tool schemas, so this is strong negative evidence about the tested implementation, not an impossibility theorem for all MAS. [14]

Brief · New preprintBased on the abstract; matched-cost details not checked

LaLa-World: mix heterogeneous world-model data through language

LaLa-World: Scaling Latent World Models with Natural Language Actions
Junha Song · Geonmo Gu · Jeongeun Park · Heejeong Nam · Lucas Maes · Byeongho Heo · Dongyoon Han · Sangdoo Yun
First posted 2026-10-20 04:32 EDT · v1

Numerical actions and visual changes are expressed in language so that one JEPA-style predictor can train across simulation, robot, image, and video data. Against a matched RAEv2 baseline, the abstract reports comparable planning at 1/6 the training cost and 13.9× faster planning. Here, scaling means cross-domain data composition, not a model-size power law. [15]

Daily classic · Historical paperRead §3, §4, and the training appendix; human evaluation not reproduced

InstructGPT: preference, KL, and pretraining mix

Training Language Models to Follow Instructions with Human Feedback
Long Ouyang · Jeff Wu · Xu Jiang · et al.
First posted 2022-03-04; this reading used v1 and the NeurIPS 2022 version
NeurIPS 2022; ACM Digital Library displayed 33,248 citations on 2026-10-21 ↗

Earlier RLHF work covered Atari and robotics (2017) and summarization (2020). InstructGPT moved SFT→reward model→PPO to a broad instruction-following distribution. The mathematical hinge is how pairwise preferences become scalar reward and how policy movement away from supervised support is constrained. This is a historical paper, not today's news. [16][17]

Bradley–Terry pairwise loss; reward is identified only through differences, so a shift normalization is applied afterward
LRM(θ)=−E(x,yw,yl)log⁡σ ⁣(rθ(x,yw)−rθ(x,yl))\mathcal L_{\rm RM}(\theta)=-\mathbb E_{(x,y_w,y_l)}\log\sigma\!\left(r_\theta(x,y_w)-r_\theta(x,y_l)\right)
The PPO-ptx objective: KL limits policy shift, while pretraining likelihood directly anchors the original capability distribution
max⁡ϕ Ex,y∼πϕ ⁣[rθ(x,y)−βlog⁡πϕ(y∣x)πSFT(y∣x)]+γ Ex∼Dprelog⁡πϕ(x)\max_\phi\ \mathbb E_{x,y\sim\pi_\phi}\!\left[r_\theta(x,y)-\beta\log\frac{\pi_\phi(y\mid x)}{\pi_{\rm SFT}(y\mid x)}\right]+\gamma\,\mathbb E_{x\sim D_{\rm pre}}\log\pi_\phi(x)
Simplified derivation: with γ omitted and the sequence treated as a one-step bandit, the KL-regularized optimum exponentially tilts the reference; this is not a PPO convergence theorem
π∗(y∣x)∝πSFT(y∣x)exp⁡ ⁣(rθ(x,y)β)\pi^*(y\mid x)\propto\pi_{\rm SFT}(y\mid x)\exp\!\left(\frac{r_\theta(x,y)}{\beta}\right)
EvidencePaper resultDoes not imply
API-prompt human evaluation1.3B PPO-ptx beats 175B GPT-3; 175B InstructGPT beats 175B GPT-3 85±3%A 100× smaller model preserves every capability
Public NLPPPO shows an alignment tax; pretraining gradients reduce several regressionsMore KL is always better
Training compute175B SFT: 4.9; PPO-ptx: 60; GPT-3 pretraining: 3640 PF-daysData collection and the full experimental program are negligible
All results use the paper's 2022 distributions, models, and evaluations; win rate is not a universal measure of present-day chat capability. [16]

READ THE ORIGINALS

Original sources

  1. MuPACE · original paper; methods, experiments, and ablations read
  2. Constant Step Sizes for Muon-Type Spectral Optimizers · original paper; theorems and experiments read
  3. Beyond Scalar Leaderboards · original paper; theorems and Arena experiments read
  4. The Shape of the DPO Hyperparameter Landscape · original paper; main experiments and hyperparameter tables read
  5. Foundations of the Scaling Law · original paper; relevant sections read
  6. When Does Training on Synthetic Data Provably Work? · original abstract
  7. Benefit and Cost of Random-Token Replacement under Repeated Data · original paper
  8. Adaptive Replay Refinement · original abstract
  9. TEX · original abstract
  10. The Token-Energy Law of LLM Decode · original abstract
  11. Rewards for Budget-Aware Multi-Turn Agents · original abstract
  12. On the Limits of Scale · original abstract
  13. Scaling Graph Reasoning · original abstract
  14. Does Multi-Agent Collaboration Pay? · original abstract
  15. LaLa-World · original abstract
  16. Training Language Models to Follow Instructions with Human Feedback · original paper; methods, results, and training appendix read
  17. InstructGPT · ACM Digital Library citation record; checked 2026-10-21
Search window

2026-10-18 08:13 — 2026-10-21 08:13 · America/New_York

READ · THINK · DISCUSS

Discussion & likes

Discuss a derivation, add a reference, or suggest the next classic.

Comment, reply, or like below. Sign in with GitHub to participate; comments are public.

Loading discussion…

Open discussion on GitHub ↗

← Back to all issues