Perturbations, state, and sampling paths

Stale perturbations: what convergence survives?

SAM (2021) usually needs two gradient evaluations per step. SOMA transforms a stale gradient residual after subtracting a momentum-based EMA. Indices below follow Algorithm 1; NS denotes a finite-step spectral transform.[1]

Theorem 1 assumes a lower-bounded objective, operator/nuclear-norm Hölder smoothness and bounded p-th central noise moments with independent minibatches. It controls the average expected nuclear gradient norm, retaining a floor proportional to ρ raised to ν. Direction-insensitivity of this bound does not establish better generalization.[1]

When the scheduler changes the experiment

In this OLMo-core/Dion3 integration, lr and initial_lr referenced the same scalar tensor. An in-place scheduler update therefore changed its own baseline. Converting values during checkpoint serialization preserved an already corrupted baseline.[2][3]

The fix registers initial_lr using float(lr), separating the immutable baseline from the runtime tensor. The author reports matching learning-rate trajectories for 20 uninterrupted steps versus a cold restart after step 10, plus HSDP/FP8 interruption checks.[2]

Capability depends on the training mixture

The original blog reports internal xdev-bench-hard pass@3 falling from 90.5% to 81.0% after removing cyber data from mid-training and related RL environments. This is not pass@1. Both stages changed together, preventing separate attribution; no fitted compute–performance curve is provided. The technical comparison, rather than the launch itself, is included.[4]

Selection can dominate long-context training

HyTopK finds exact top-k indices online; selection replay reuses them during recomputation. The paper reports 2.07× end-to-end GLM-5.1 training speedup over Megatron-LM at 1M context on 512 H20 GPUs. Layer-level gains for other architectures are not whole-model gains. This studies context-dependent compute costs, not a new loss scaling law.[5]

Expand source diversity at a fixed trajectory budget

For Qwen3.5-35B-A3B, repository coverage rises from 20% to 100% at fixed trajectory count, average difficulty and task proportions: four benchmarks improve overall, with intermediate fluctuations. At 30,000 trajectories, raising mean difficulty from 2.4 to 3.2 gives no consistent benefit. These ablations distinguish source breadth from difficulty; they establish no universal power law.[6]