Timing, Geometry, and the Limits of Gaussianity

When good data enters depends on when the learning rate decays

Let γ(t) be the mid-training (MT) fraction and η(t) the learning rate. With compute and MT tokens fixed, introducing MT earlier can win under decay and lose under constant LR. [1]

The explanation uses block-diagonal quadratics and additive gradient noise. Extract one MT-specific direction with curvature λ>0, initial value z₀, variance ν, and discrete-step interval h. [1]

With constant η, spending the budget fixes R(0); placing MT last maximizes every R(s), minimizing variance. Under decay, moving MT earlier also raises R(0), creating a bias–variance trade-off. [1]

Muon's initialization geometry: a diagonal mean is not a diagonal trajectory

The object is E[GGᵀ] over random initialization. It motivates compatibility with known row-block Hessian structure, without estimating Hessian values. [2]

For W∈Rᵈˣᵏ, G=(WWᵀ−M)W, fixed M⪰0, and independent N(0,1) weights, Wishart moments give the exact identity: [2]

When the third moment detects Gaussianity better than the fourth

Graph two-sample tests count between-group edges in a pooled similarity graph. Under the Poissonized null, conditioning on locations gives independent Bernoulli(p) labels. Balanced and unbalanced proportions require different Gaussian criteria. [3]

A star graph explains the exception: conditioning on its center leaves a two-point limit. This is a simplified derivation of the paper's star example. [3]

HyMUON combines long-history averaging with a fast EMA

HyM blends long-history averaging with fast momentum; HyMUON then applies NS. LLaMA-1.3B/C4 at 10k steps gives validation loss 2.777 versus 2.903. HyMUON is tuned and evaluates averaged iterates; Muon uses previously selected settings and training iterates. The v2 note concerns author order/affiliations; no substantive new result is confirmed. [4]

Kinetic Muon: the step map changes period-two stability

Kinetic maps wrap update directions, including Muon. In the reduced model, period-two linear stability requires step-length elasticity in (0,1). Many ResNet-18 gains can be reproduced by matching effective LR, so default-LR comparisons are insufficient. [5]

Long-context continuation separates short- and long-context budgets

Five model scales and over 100 budget configurations support an additive power law. Constant-Last4 averages the latest four constant-LR checkpoints. A 1.22B trajectory at about 100B LCCP tokens shows no clear saturation within that range, rather than indefinitely. [6]

Harness diversity as another RL scaling axis

A factorized configuration space and balanced factorial coverage create a compact cross-harness curriculum. The authors report transfer to unseen white-box configurations and external black-box harnesses; the abstract does not establish a universal power-law exponent. [7]

Pixel LMs: languages, scripts, and capacity scale differently

At fixed per-language compute, three axes are evaluated on five tasks and 83 languages. Capacity mitigates multilingual interference; language/script expansion helps locally, while unseen-script transfer does not keep improving with scale. [8]

Virtual-world CPT: diversity has limits for distant transfer

120 configurations span 0.8B–27B, 50–1,600 hours, and five mixtures. Diversity helps related-world transfer but has limited distant-world gains; larger single-world models can transfer worse. The authors distinguish early interface learning from later world-specific learning. [9]

Fewer biased answers may mean more Unknown responses

On BBQ, larger Qwen3/Gemma-3/SmolLM2 models improve accuracy without a simple bias-scaling trend. Of 26 configurations with fewer biased responses, 20 also increase over-abstention, making refusal and forced-choice controls essential. [10]

After failure: more compute, information, or feedback?

BRA fixes the failed state and revision protocol: six models, 194 function-level and 126 repository-level failures. Specifications help consistently; compute's value declines with capability. Three of four frontier models overwhelmingly choose feedback yet gain less than random resource selection. [11]

Low-temperature synthetic-data sharpening and high-temperature dreaming

In a controlled Markov model, recursive Tₛ<1 sampling moves subdominant eigenvalues toward the unit circle, even with data accumulation. A T_d>1 self-sampling regularizer mitigates drift over 5,000 generations and in LSTM/GPT-2 124M tests, without a general LLM guarantee. [12]

BRACE forecasts within-group reward contrast before generation

Historical evidence, success-rate uncertainty, and correlated rewards predict mixed-reward groups before allocation. At three Qwen3-VL scales, fewer new rollouts exceed baseline peaks. Historical-bank collection is accounted for separately, so this is not the same reduction in total training cost. [13]

MemoryTax: a stronger reader cannot recover what a writer discarded

One LoCoMo conversation, 152 questions, one run per cell compare memory methods across three model capabilities. Writer quality limits designed memory. Photo-caption access explains part of the agent advantage; on 88 photo-free questions, Pi and full context differ by one answer. [14]

GRAFT routes gradients with monitor flags, then removes FORGET

Flagged trajectories update only FORGET; unflagged ones update both adapters, with RETAIN-only anchor training. Removing FORGET gives near-zero hacking and nearly +20pp correctness over base in Qwen3-8B/Countdown-Code. RL uses handcrafted monitors and single turns; constructing anchors remains a limitation. [15]

An agent-training roadmap: supervision granularity and rollout bottlenecks

The synthesis connects SFT, distillation, single-turn RL, stateful environments, and full harnesses. Supervision shifts from token probabilities to trajectory outcomes, while long-episode generation becomes the bottleneck. It links corresponding TRL/OpenEnv/AsyncGRPO implementations. [16]

ALTK-Evolve distills training-task outcomes into guidelines

GPT-4.1 extracts rules from training trajectories, then freezes them for held-out prompts without weight updates. Retail Pass^4 is 67.5%—all four trials succeed, not pass@4. The leaderboard snapshot is October 12, and longer guidelines increase token use. [17]