Meta Models · Limits of data curation · Reading LoRA
Two main readings, two briefs, and three version watches recovered in the overlapping window: predicting experiments from training records and separating acceleration from persistent curation gains. The classic is LoRA and its factorized gradients.
First submitted 2026-10-27 · v1 · Recovered in the overlapping window
Building on parametric scaling curves (Kaplan, 2020; Chinchilla, 2022), this paper supplies configurations q, progress t, and historical losses to frozen TabPFN. Mixture proportions are configuration fields.[1]
The prediction target; context updates do not train weights [1]
R={(qi,tij,Lij)}i,j,L(q,t)=fθ(q,t∣R)
Evaluation design
TabPFN
Baseline
Delphi: 3,737 runs; 24 held-out groups; mean Spearman ρ
0.967
XGBoost 0.875
Leaving out five-domain mixtures individually, MAE is about 20%–40% below the best evaluated parametric law at four model sizes; laws win on two-domain mixtures and some RedPajama settings.[1]
Main readingRelevant methods and experiments read; not reproduced
Reaching it faster, or reaching a better endpoint?
Jiaxing Wang · Zicheng Zhang · Deping Xiang · Jin Xu · Yong Li · Chao Liu · Pengzhang Liu · Qixia Jiang
First submitted 2026-10-27 · v1 · Recovered in the overlapping window
The paper analyzes growing sample budgets in teacher–student perceptrons and distinguishes the effects empirically in LLMs. If the population training objective equals target loss and the baseline is globally optimal, reallocation cannot beat that optimum; finite-budget acceleration remains possible.[2]
Population optimality without shift; not a nonconvex training guarantee [2]
θ0∈argθminLtar(θ)⟹Ltar(θ(w))≥Ltar(θ0)
Domain-level diagnostic; training loss is measured on seen examples [2]
First submitted 2026-10-27 · v1 · Recovered in the overlapping window
Based on the abstract: small GPT-2 models use BabyView home-recording transcripts and synthetic TinyDialogues, varying quantity, source, and diversity of child environments. Multiple children improve pooled held-out language prediction, without equally clear grammatical-benchmark gains; one loss does not represent every capability.[3]
Min Chen · Junqi Wang · Zeguan Wu · Wenhao Huang · Junyu Liu
First submitted 2026-10-26 · v1 · Recovered in the overlapping window
Based on the abstract: in two-layer modular-addition networks, a second-order loss expansion in per-neuron norms defines couplings, followed by Fiedler ordering and coarse-graining to estimate critical exponents. This concerns scaling of ML generalization transitions, rather than a universal model–data–compute budget power law.[4]
Version watchAbstract and version metadata; differences unverified
Exact Unlearning of Finetuning Data via Model Merging at Scale
Kevin Kuo · Amrith Setlur · Kartik Srinivas · Aditi Raghunathan · Virginia Smith
First submitted 2025-04-06; updated 2026-10-27 · v2
Aditi Raghunathan subscription: v2 is available. Based on its current abstract, SIFT-Masks combines a global sign constraint and local masks for model merging and exact unlearning. The v1→v2 content changes have not been checked; current claims are not treated as newly established results.[7]
Version watchAbstract and version metadata; differences unverified
Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency
Jia-Nan Wang · Zixun Huang · Kairui Li · Zilin Wang · Lei Wu
First submitted 2026-09-02; updated 2026-10-27 · v3
Lei Wu subscription and scaling: v3 is available. Current Comments mention revised proofs and added neural-network experiments. Without a v2→v3 comparison, these additions cannot all be assigned to v3; earlier critical-batch conclusions are not reported as new.[8]
Version watchAbstract and version metadata; differences unverified
Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
Jongwoo Ko · Sara Abdali · Young Jin Kim · Tianyi Chen · Pashmina Cameron
First submitted 2026-03-11; updated 2026-10-27 · v2
v2 is available. Its current abstract describes REOPOLD controlling where, how strongly, and when teacher feedback acts, with extensions to test-time scaling. Version differences remain unchecked; the abstract’s sample-efficiency maximum is not attributed to this update.[9]
Historical classicParameterization, initialization, and rank experiments read; gradient formulas are explanatory derivations, not paper theorems
Full fine-tuning stores all parameters per task; adapters (2019) add layers. Following low-intrinsic-dimension adaptation (2020), LoRA freezes pretrained weights and learns a low-rank increment that can be merged for deployment.[5][6]
Paper parameterization; W₀ is frozen [5]
W=W0+sBA,s=α/r,B∈Rd×r,A∈Rr×k,r≪min(d,k)
Per-matrix parameter count and representation constraint [5]
#(A,B)=r(d+k),rank(W−W0)≤r
Let G denote the loss gradient with respect to the effective weight. The following uses the chain rule and a first-order expansion of simultaneous SGD. The paper uses Adam, so this recurrence does not directly describe its experiments.
Explanatory derivation; not a paper theorem
∇BL=sGA⊤,∇AL=sB⊤G
Induced matrix update under simultaneous SGD; factors set left/right preconditioning