What to prune, what to preserve

2N: not simply the hardest examples

Sorscher et al. (2022) studied pruning-induced changes to learning curves. [3] Here the setting is Gaussian linear classification: dimension $N$, sample count $P$, perfectly aligned generator/test/pruner directions, and ridgeless squared loss. [1]

The paper reduces classification error to an angle; retention must exceed the dimension/sample ratio. [1]

A conditional-density bound yields a variance bound; optimization over retention gives the key chain from Appendix E: [1]

Freeze weights, learn one cross-budget ranking

Fixed-target pruning typically repeats work for each budget. Here only global MLP-neuron/head scores $z$ are learned; backbone $\theta$ stays frozen. Let $b$ denote removed FLOPs and $\kappa_b$ the cost-weighted ranking threshold. [2]

Fourteen targets (5%–70%), 300 Adam calibration steps; ViT-B uses batch 32 and 2,000 unique images by default. One ranking yields nested subnetworks, not optimization-free adaptation. [2]

Indic: corpus inventory is not training budget

Translate, judge, repair, transliterate: 75/100 is a rubric threshold, not accuracy. The 2.8T-token inventory includes shared-source language/script variants and already includes 438.4B recovered tokens. Nemotron CPT uses 10B Indic tokens, natural plus synthetic, so stage gains do not isolate synthetic data. This adds the data side to yesterday’s Nemotron coverage; the corpus is still marked forthcoming. [4]

Beam: language adherence, not reasoning accuracy

One epoch of Multilingual-Thinking LoRA SFT changes requested-language adherence from 3/16 to 16/16 held-out prompts: tiny sample, no reasoning-correctness evaluation. Separately, the GRPO recipe uses 32 prompts × 8 responses, group advantages and truncated importance sampling, without KL/entropy penalties. A runnable recipe is not demonstrated downstream improvement. [5]

Muse Spark: attention-only LoRA on 816B

Experts remain frozen; LoRA targets four attention projections. The reported run uses 64 H100s, 500 steps, length 2,048 and packing, taking about two hours. Training token accuracy rises 72%→84% and loss falls 1.26→0.57; these are training metrics, not held-out capability gains. [6]

Apertus: post-training can erode long-context ability

Weights were released July 24; this is a new technical write-up. Five stages: SFT → long-context SFT → RL → DPO → SDPO. Reasoning traces are added only where non-thinking fails; tool trajectories require execution verification. Long-context SFT restores ability eroded by short-context SFT; no isolated causal ablation is established here. [7]

Self-improvement: protect tests, not just instructions

This technical synthesis proposes separate tests for goals, role, memory and behavior, separating development, acceptance and final audits while keeping evaluators outside candidate mutation scope. LIP-Eval is a proposal without pilot data or calibrated thresholds, not a validated stability guarantee. [8]

MirrorMoE: more expert combinations need not be explored

Shared experts reduce representation variability while fine-grained segmentation enlarges routing space. Token-adaptive expert suppression shrinks exploration. The abstract reports gains across model scales, not an extrapolatable scaling exponent. [9]

MPT: expand the inference-time agent population

Jointly learns actions and participation bids, assigning local credit from collective outcomes. Larger inference populations improve performance and dropout robustness; this does not establish superiority at fixed total compute. [10]

Librarian Agents: amortize a curated workspace

An RL builder organizes documents into synthetic files and reference hierarchies before queries. Authors report +33 accuracy points on BrowseComp-Plus and +31 on BRIGHT versus strongest baselines. Repeated queries must amortize construction; retrieval is not free. [11]

Anvil: data repair and first-fault distillation

Uses 5.8M source-grounded records over 160K images, with 5.22M for training. A 9B student routes to specialist teachers at the earliest semantic fault and normalizes stages. Controlled studies examine coverage, selective transfer and feedback repair: a data-quality study. [12]

Data World: specify mechanisms before files

Explicit generative mechanisms produce environments, tasks and rubrics; independent audits check target recoverability. There are 2,135 training tasks from 1,037 worlds. More data and genuinely solvable tasks are distinct; no universal scaling law is claimed. [13]

MetaExp: cross-task structure as data design

Across 1,000 task families/4,956 trajectories, fixed tools or latent workflows coexist with changing tasks to train experience transfer; downstream weights stay frozen. Domain-excluded tests support transfer, not unlimited self-improvement or a power law in environment count. [14]

Density of Thought: a proxy can reverse across training regimes

Six model scales, two training paradigms, only 60 GSM8K problems: the density-variance/correctness relation weakens with scale and reverses after RL distillation. Direct steering changes accuracy 60.0%→8.3%; a small-sample negative result, not a universal law. [15]

GRADS: dependency-controlled synthetic difficulty

Maintains necessary evidence/inference dependencies instead of padding irrelevant context; independent solvers assess difficulty separation. At matched training budgets, SFT termination exceeds 95% and calls/turns fall by up to 75%, not total training cost. [16]

AURA: stop reasoning using retrieval utility

Intermediate embeddings separate relevant documents from hard distractors; a lightweight utility estimator controls depth. On BRIGHT, 4B achieves 34.7 nDCG@10 at 2.8 steps/96 tokens on average; matched-deployment latency drops 4.6→0.76 seconds within this retrieval protocol. [17]

Experimental design: amortize information-gain search

Defines information-gain rewards under a generative model independent of the LLM’s beliefs, training reasoning/action policies on Twenty Questions against training- and inference-time BED baselines. Transfer to Medi-Q is not evidence of clinical effectiveness. [18]

SWE-Stream: separate data quality from full-scale results

Constructs 80,874 executable tasks and 118,688 successful trajectories. SWE-bench Verified is 45.6% under matched training budgets versus 69.0% with full training: different comparisons. The 1.45× gain is accepted-task construction throughput, not model-training speed. [19]

KAN: favorable exponents from architecture or target structure?

For arbitrary-depth standard spline-KANs, the abstract states a worst-case exponential-in-dimension coefficient lower bound on isotropic Sobolev balls. Favorable exponents for low-arity composition are not KAN-exclusive. Full proof unchecked; target class and accuracy threshold matter. [20]

DropKV: kernel speed is not end-to-end speed

Decouples joint eviction into per-token scores; the constant-factor guarantee requires a cone condition. A 19.8× scoring-kernel gain translates to at most 9.9% matched-batch prefill acceleration, with improved RULER/LongBench at equal cache budgets. [21]

Short drafters: preserve speedup across context lengths

Freezes short-context drafters, caps their attention cache and rebases rotary indices; three model pairs/five corpora reach 2M context. The claim is flatness of speedup, not a fixed multiple; speculative cost per emitted token must remain below target-only cost. [22]

Latent Compaction: train the compressed cache

With weights frozen, fits full-cache attention/log-normalizers before end-to-end cache distillation, without downstream questions. Reported RULER accuracy at 32× compression is 96.7% versus KVZip+ 40.0%; matched compression budgets do not imply matched total test-time-training cost. [23]

Player-1: synthetic-environment coverage and transfer

Initializes from 6,044 demonstrations; 16 synthetic 2D/3D environments supply verifiable rewards and randomized control dynamics. Broader coverage improves transfer. MineDojo-300 gains over each model’s SFT baseline are 27.8/17.7 points for 9B/35B-A3B, not equal-budget architecture comparisons. [24]

Multiple developers: development budget as a scaling axis

On GAIA, specialists individually at at most 35% combine to reach 70%, 12 points above the median general-purpose system built at several times each specialist’s development budget. Aggregate development and inference costs still matter; incentive-theory conditions are unchecked. [25]

LeanGraphSearch: retrieval gains at matched reflection counts

Propagates relevance along Mathlib dependencies before reranking, comparing proof pass rates at 8/16/32 reflection turns. Retrieval costs must still enter total budgets; included through topic coverage, not as a Wharton-author update. [26]

PTS: screen variables, then choose observations

Uses correlation-aware SuSiE screening followed by IBOSS in the screened space. Correlated-Gaussian and biobank-scale synthetic-genotype studies report near-full-data prediction at lower training time: sample/feature budget design, not a biological scaling law. [27]

PLES v3: revision differences unverified

First posted September 1; v3 updated October 15. PLES selects scaling-law experiments by information gain per unit cost. Revision differences are unverified; the abstract’s below-one-tenth budget claim is not presented as a newly added result. [28]

LLMSurgeon v2: assumptions behind mixture inversion

First posted May 28; v2 updated October 15. Infers pretraining domain mixtures using label shift and a calibrated soft confusion matrix. Revision differences are unverified; generated-text domain frequencies do not unconditionally equal training frequencies. [29]