Two main readings and nine briefs, including one version watch: update geometry near zero and divergent extrapolations with nearly equal fitting loss. The classic separates sparse structure, initialization, and discovery cost.
MuonScaling lawsWharton facultyDeep learning theory
Let F be lower bounded with globally L-Lipschitz gradient in Frobenius norm. The paper studies finite-step NS in exact arithmetic, without weight decay, rather than replacing it by polar.[1]
Deterministic input; fixed beta in [0,1) and theta in [0,1] [1]
Mt=βMt−1+(1−β)gt,Nt=θMt+(1−θ)gt,gt=∇F(Wt)
Fixed J at least one; (a,b,c)=(3.4445,-4.7750,2.0315) [1]
At epsilon=0, scalar normalization removes nonzero input magnitude; the paper constructs a two-cycle on a convex quadratic. Positive epsilon shrinks small-input updates and provides the sector inequality for descent–momentum tracking.[1]
Theorem 3.2; u_J is the global output-norm bound for the finite NS map in Lemma 2.2 [1]
0<η≤4LuJε(1−β)⟹t=1∑∞∥gt∥F2<∞,∥gt∥F→0
Main readingRead Sections 2–4, Algorithm 1 and limitations; code not run
Chinchilla (2022) fitting commonly uses multi-start local optimization. This paper separates global optimization accuracy from agreement among nearly optimal extrapolations.[2]
x denotes resource axes, y observed loss; h_delta is Huber loss with threshold delta [2]
A monotone reparameterization bounds residuals over parameter boxes. Branch-and-bound splits promising boxes and discards those whose lower bounds cannot beat the incumbent. Stopping certifies its gap to a global lower bound. Four real scaling ladders demonstrate divergent extrapolation with nearly equal fit error.[2]
BriefBased on abstract and version history; not reproduced
Anda Tang · Xuanlin Yang · Junjie Wang · Zhouchen Lin
2026-10-26 · v1
A unitary transform mixes layers before slice-wise polarization and inverse transformation. Authors characterize steepest descent in a transformed tensor spectral norm. Theorems and matched budgets were not checked; no speedup magnitude is reported here.[3]
BriefBased on abstract and version history; not reproduced
First posted 2026-10-23 23:29 New York time · v1; delayed-index catch-up
In Starpilot and Craftax actor–critic experiments, authors find a single-agent Elo variant more predictable than mean return and report extrapolation across two orders of magnitude in compute. Craftax compute-optimal model size can be non-monotone. Fit/extrapolation splits were not checked.[4]
BriefBased on abstract and version history; not reproduced
An information scale for memorization and sample quality
Druv Pai · Dejan Stancevic · Yi Ma · Luca Ambrogioni
First posted 2026-10-23 18:21 New York time · v1; delayed-index catch-up
Model information measures sample dependence on the particular training dataset. Authors posit two scaling regimes and derive induced laws for memorization and quality. This is a proposed, empirically fitted law; the abstract does not establish universality.[5]
BriefBased on abstract and version history; not reproduced
In deep linear networks, depth improves the training-time exponent for difficult tasks but not easy ones; predictor-norm time reparameterization yields a common law. Abstract only: initialization, spectral conditions and nonlinear numerical evidence were not checked in detail.[6]
BriefBased on abstract and version history; not reproduced
Yonghoon Lee · Edgar Dobriban · Eric Tchetgen Tchetgen
2026-10-26 · v1
Wharton author subscription: an identification index separates impossibility from informative prediction with MNAR outcomes and instrumental variables. Efficient influence functions support doubly robust coverage. Abstract only; this does not remove the need for identifying assumptions.[7]
BriefBased on abstract and version history; not reproduced
Ruixuan Zhao · Jingxin Yan · Xiaochuan Shi · Torben Martinussen · Eric Tchetgen Tchetgen · Linbo Wang
2026-10-25 New York time · v1
Wharton author subscription: survival and cumulative-incidence ratios are targeted with variation-independent nuisance parameters and valid monotone curves, supporting likelihood and doubly robust estimation. Abstract only; descriptive selected-cohort results are not causal effects.[8]
BriefBased on abstract and version history; not reproduced
Version watch: width, connectivity and unit complexity
First posted 2026-05-12; v2 dated 2026-10-26; substantive version differences not verified
The current abstract studies allocation among unit count, effective per-unit complexity and connectivity at fixed parameter budget, observing an interior tradeoff on SHD-Adding and Enwik8. Version watch only: changes from v1 were not established, so these are not presented as new results.[9]
BriefBased on abstract and version history; not reproduced
Cheng Gui · Yihua Shao · Lei Li · Jingcai Guo · Maolin Wang
2026-10-25 · v1
Included under inference scaling: maximum proxy-reward selection increasingly probes score tails. Authors express BoN through a Beta-weighted integral of quantile-conditional true reward and propose tail calibration audits, explicitly complementing ranking accuracy. Continuity/tie assumptions were not checked.[10]
BriefBased on abstract and version history; not reproduced
Catch-up: fixed exposure, different learning weights
Jiazheng Li · Huaqing Zhang · Jing Xu · Tianle Cai · Jingzhao Zhang
First posted 2026-10-23 00:09 New York time · v1; delayed catch-up outside the main window
Data-mixture catch-up: LARE preserves sampling exposure while adapting domain weights from running loss statistics. The abstract reports gains over a matched direct mixture at 1.5B/240B tokens. Full text not read; reweighting is distinct from changing exposure, and numerical gains are not repeated here.[11]
Daily classic · historical workRead Sections 1–4, experimental setup and reinitialization controls; an empirical hypothesis, not a general existence theorem
Sparse structure is not enough: which initialization survives?
Pruning (LeCun et al., 1990; Han et al., 2015) showed that trained networks can be compressed. The lottery hypothesis asks whether the subnetwork trains independently from its original initialization. Let d count parameters and m be a fixed binary mask.[12][13]
Weights outside the mask remain zero [12]
θ0∼D,m∈{0,1}d,fm(x;θ)=f(x;m⊙θ)
Original hypothesis: j is the minimum-validation-loss iteration and a its test accuracy [12]
∃m:∥m∥0≪d,jm≤jdense,am≥adense
Algorithm restatement: train, magnitude-prune, reset each round; trained values are not retained [12]
Key control fixes the mask and resamples initialization, separating structure from initialization [12]
f(x;m⊙θ0)vs.f(x;m⊙θ0′),θ0′∼D
MNIST fully connected and CIFAR10 convolutional experiments support the hypothesis. Original initialization generally beats reinitializing the same mask. Deep networks are learning-rate sensitive: VGG-19 at 0.1 did not yield winning tickets without warmup, but did with it. Iteration savings exclude repeated mask discovery.[12]