Budgets, truncation, and transfer

MAI-Base-2: scale unique data along with the model

Transferring a short scaling run to long training requires preserving repetition. Let $N$ be active parameters, $D$ training tokens and $U$ unique tokens. The report preserves TPP and each source’s repetition rate. A one-source rewrite is: [1]

With steps $s$, learning rate $\eta$ and weight decay $\lambda$, a quadratic fit in log-hyperparameter coordinates gives these empirical optima: [1]

The final model has 2.8T total/80B active parameters and trains on 50T tokens. Momentum resharding, batched all-to-all and specialized Newton–Schulz kernels put the Muon optimizer step at 2.1% of the average iteration (165 ms). [1]

On which budget axis does Muon win?

On one A100: 17.7M/33.6M LMs process 20.48M FineWeb-Edu-subset tokens over 5,000 steps; a 2.70M ViT trains for 100 epochs on balanced/100:1 long-tailed CIFAR-100. Each strategy searches 23 configurations with one development seed, then compares five paired seeds. [2]

STOPD: unbiased sums need not imply unbiased updates

On-policy distillation scores the student’s own trajectories; long suffixes are expensive. STOPD randomly truncates blocks and restores position coverage through inverse-probability weights. With cap $L$, block width $K$, full length $T_i$, cutoff $S_i$ and token gradient $h_{it}$: [3]

Cutoffs must be independent, with generation and teacher feedback unchanged at a shared prefix. Let $N=\sum_iT_i$ be the complete token count and $C$ the lagged normalizer from the previous step: [3]

Muon has a corresponding optimizer coefficient, but the bound fixes parameters and previous state; it does not equate training trajectories. For Qwen3-1.7B←4B mathematics, AdamW costs 941→513 seconds/step, with 250-step accuracy 30.1→29.4. Time includes generation, teacher scoring and updates. The Muon counterpart costs 1,029→542 seconds/step. [3]

Muon optimality: radial laws versus minimax geometry

For convex nondecreasing radial objectives, exact polar optimality is fragile under a fixed atomless radial law, while worst-case optimization over radial laws can recover the polar direction. Full-column-rank and other assumptions matter; this is not a general nonconvex training guarantee. [4]

Hassani: escalation from model disagreement

Approximates optimal weak-to-strong routing through response disagreement without correctness labels. The approximation improves as the strong model better approximates the ground-truth distribution; the strong model is a proxy, not truth. [5]

A depth/width compression law in deep linear networks

Expresses neural collapse through a feature kernel in deep linear models with exchangeable class structure. Accumulated compression depends on $L/N$ at large width, not a universal power law for nonlinear networks. [6]

Ensembling has two sample sizes

For fixed model/answer counts and margin exponent $\beta$, buffered ERM bounds clean excess risk by $\widetilde O(Q^{-1/2}+N^{-\beta/2})$: $Q$ labeled questions and $N$ generations per model/question. A two-model/two-answer submodel has an $N^{-\beta}$ identification floor. [7]

TIPS: learn when to reveal and remask

A 600K controller learns unmasking/remasking over frozen LLaDA-8B. At 32 decoder evaluations, two variants achieve 58.46%/57.97% micro-average on GSM8K/MATH-500; the controller incurs separate SFT/RL training costs. [8]

EMO: alternate context optimization and logit steering

An external judge revises context, followed by constrained steering from before/after logit differences. One-round target-KL improvement assumes context improvement and faithful steering; matched target-model generations need not match total judge cost. [9]

Where should a byte-patching budget go?

Compares entropy and novelty patching at about 11% matched patch rate across three seeds and 1M/12M/50M BLT-style models. Gains concentrate in mathematical text and shrink on prose/code; local surprisal alone need not identify valuable computation. [10]

LEASH: oversight after escalation

Uses weak-model hidden-state comparisons with successful/failed trajectories to trigger escalation and continue oversight afterward. OSWorld gains hold against random routing at matched escalation rates; cost savings depend on the specified backbones, prices and tasks. [11]

Three actions for deciding whether to think longer

Chooses between the current answer, a fallback and further expansion with action-specific confidence calibration. Online compute falls about 30%/16% versus fixed policies; utility changes against fitted Q are small and mixed. [12]

Select data responsive to additional compute

Prioritizes examples that fail with short reasoning but succeed with more compute. Qwen3-8B improves its 512–4,096-token frontier over matched non-CR data. The 45.3% figure is recovery of low-compute failures, not overall accuracy. [13]

Why scaling search may require forgetting

Across four automated-research tasks, accumulated history anchors search. Restarting sessions/history from the best candidate improves outcomes at matched budgets; this does not imply that all memory is harmful. [14]

LoC: the visual budget after correct retrieval

On SciEGQA oracle pages, raising visual tokens from 1K to 32K adds 15.22 accuracy points; ground-truth crops add 3.75 points at fixed 16K. These measure reading-budget effects and an oracle diagnostic, not improved retrieval. [15]

NPO: amortize inference search into training

Freezes a few-step generator and trains a flow policy over initial noise toward a reward-tilted distribution. Search moves into training; the abstract’s 16–549× compares inference costs, not total training-plus-inference expenditure. [16]

DMS: sharpen paths and answers separately

Mildly sharpens reasoning paths before aggregating and sharpening the answer marginal, with annealed SMC to mitigate path collapse. A base-model-likely answer need not be correct; no external ground truth is introduced. [17]

Priming: reconstruction before distributional training

A one-step generator first learns fixed-noise reconstruction, then feature-space distributional post-training. ImageNet-256 FID 1.19 at about one-tenth the training compute is the authors’ comparison with similar-FID methods, not an independent replication. [18]

AFCS: synthesize from facts, not just passages

From the same FineWiki source, extracts/verifies atomic facts before generation, comparing fact/paragraph/chunk/entity granularity for OLMo-3-7B and 32B continued pretraining. A data-quality study, not a universal scaling exponent. [19]

RLVR: progress and collapse have different clocks

In a local quadratic model, square-root learning-rate scaling preserves first-order drift per sample, not curvature-adjusted progress. An invariant collapse sample count comes from a separate drift-threshold hypothesis. Experiments span 16× batch sizes; these are not one theorem. [20]

SCRouter: jointly choose model and token budget

Separately predicts utility, security and completion-within-budget probability, then weights them by task to select model/tokens. The benchmark has 38,869 queries and 12 models × 12 budgets; security preferences are task-conditioned, not a single universal quality score. [21]

Manacá: nonmonotone data mixing and negative DPO results

Scanning 10%–60% general data in safety SFT gives nonmonotone over-refusal, with a knee around 40%–50%; three DPO attempts report negative results. Rising training reward margin need not improve free generation. Evidence is specific to this PT-BR model and protocol. [22]

Flow matching: coupling minibatch is a resource axis

An online entropic-OT approximation enables larger source/target couplings. The abstract claims strict reductions in full stochastic-gradient variance and integration error versus independent pairing, improving with coupling batch size. Proof conditions have not been checked. [23]

The low-compute frontier for protein language models

ProtModernBERT separates architecture, data and parameter count using controlled pretraining and cluster-based validation. Some tasks approach Amplify-120M/ProtBert at as little as 10%–20% of pretraining FLOPs: model-training efficiency, not a biological scaling law. [24]

FedStop: the marginal value of another local epoch

Models federated local training as finite-horizon stopping: continue while expected next-epoch loss reduction exceeds priced drift. One-step look-ahead optimality requires nested stopping decisions; closed forms treat geometric and power-law returns separately. [25]

Nemotron: multi-teacher on-policy distillation and quantization

Technical sections describe two-stage visual/language SFT, specialized teachers scoring student rollouts, and NVFP4 quantization-aware distillation. Multiple components change together; the blog does not isolate MOPD causally. Projected customer cost savings are not treated as research findings. [26]

Kakade: model averaging for replay-free continual learning

Comparing v1/v2 abstracts: Soup to go is retitled Forget About It, adding Lookahead and Qwen3-14B adapter continual-learning descriptions. Periodic parameter interpolation avoids replay; up to 36.3% lower task-training time is protocol-specific, not universal. [27]

Cai/Long/Su: watermark localization in mixed text

The authors note updated experiments and supplementary material; exact version differences remain unchecked. The current abstract casts token localization as multiple testing and separates detection, discovery and classification boundaries. Existing boundaries are not claimed as new in this revision. [28]