ZHZixun Huang
Research DigestNo. 012
Daily research notes12 min read

Which policy generated this trajectory?

Search-agent RL · Mixed-version rollouts · TRPO

One previously unlisted technical blog from the overlap window; historical reading derives the gap between surrogate improvement and policy improvement.

Post-trainingDeep learning theory
Daily classic · 2015TRPO
Main readingRead data strategy, synchronization, results and limitations; not reproduced

Filtering tasks; tracking the policy behind each token

Post-Training at the Frontier
Ching-Chia Wang · Eddy Nassif · Aastha Jhunjhunwala
Published October 22, 2026; newly included from overlap window

For search agents, filtering and rollout synchronization jointly determine the update data. This October 22 blog is newly included: eight probes per question, then SFT on 4,932 traces and GRPO on 1,480 tasks.[1]

Independent Bernoulli illustration: intermediate difficulty is more likely retained; not a paper theorem
K∣p∼Binomial⁡(8,p),Pr⁡(0<K<8)=1−p8−(1−p)8K\mid p\sim\operatorname{Binomial}(8,p),\qquad \Pr(0<K<8)=1-p^8-(1-p)^8

Eight failures do not establish unlearnability. Filtering depends on both policy and tools; the reported pipeline does not isolate their contributions.[1]

Setup / metricReported result
Nemotron 3.5 Lightning; BrowseComp36.97% → 45.45% accuracy
Mean tool calls / taskAbout 168 → 115
SFT / RL timeAbout 1h / 8h; SFT uses 4×B200
Blog report; both tools and training changed.[1]
Illustration: execution version v_t; exact trajectory IS requires actual behavior probabilities and support
q(τ∣x)=∏t=1Tqvt(at∣ht),w(τ)=∏t=1Tπθ(at∣ht)qvt(at∣ht)q(\tau\mid x)=\prod_{t=1}^T q_{v_t}(a_t\mid h_t),\qquad w(\tau)=\prod_{t=1}^T\frac{\pi_\theta(a_t\mid h_t)}{q_{v_t}(a_t\mid h_t)}

Hot-Load preserves requests and caches across weight changes. Strict on-policy operation can reset prompt caches and wait for all replicas. The equation is not an implemented correction; cached execution need not equal recomputation under new weights.[1]

Daily classic · historical workRead §§2–6, Appendix A coupling proof, §8.2 and Table 1; no reproduction

Why surrogate improvement needs a trust region

Trust Region Policy Optimization
John Schulman · Sergey Levine · Philipp Moritz · Michael I. Jordan · Pieter Abbeel
First posted 2015-02-19; ICML 2015; read v5 (2017-04-20)
ICML 2015 · foundational historical policy-optimization work ↗

Natural policy gradients and conservative policy iteration (2002) motivate TRPO (2015): improving on old data changes future state visitation.[2][3]

Discounted MDP; 0<γ<1; unnormalized occupancy; bounded advantage
J(π)=Eπ∑t≥0γtrt,ρπ(s)=∑t≥0γtPr⁡π(st=s),Aπ=Qπ−VπJ(\pi)=\mathbb E_\pi\sum_{t\ge0}\gamma^t r_t,\quad \rho_\pi(s)=\sum_{t\ge0}\gamma^t\Pr_\pi(s_t=s),\quad A_\pi=Q_\pi-V_\pi
Identity and surrogate: replacing new occupancy with old occupancy is the approximation [2]
J(π′)−J(π)=∑sρπ′(s)Aˉ(s),Lπ(π′)=J(π)+∑sρπ(s)Aˉ(s),Aˉ(s)=Ea∼π′Aπ(s,a)J(\pi\prime)-J(\pi)=\sum_s\rho_{\pi\prime}(s)\bar A(s),\quad L_\pi(\pi\prime)=J(\pi)+\sum_s\rho_\pi(s)\bar A(s),\quad \bar A(s)=\mathbb E_{a\sim\pi\prime}A_\pi(s,a)

Coupling bounds history divergence by tα; zero mean under the old policy bounds the new mean advantage by 2εα.[2]

Slightly looser form of Appendix A coupling bound; second-order error [2]
α=max⁡sDTV(π(⋅∣s),π′(⋅∣s)),ϵ=max⁡s,a∣Aπ(s,a)∣,∣J(π′)−Lπ(π′)∣≤∑t≥04ϵtγtα2=4ϵγ(1−γ)2α2\alpha=\max_s D_{\rm TV}(\pi(\cdot\mid s),\pi\prime(\cdot\mid s)),\quad \epsilon=\max_{s,a}|A_\pi(s,a)|,\quad |J(\pi\prime)-L_\pi(\pi\prime)|\le\sum_{t\ge0}4\epsilon t\gamma^t\alpha^2=\frac{4\epsilon\gamma}{(1-\gamma)^2}\alpha^2
Paper KL lower bound; exact improvement over the old-policy baseline guarantees nondecrease [2]
J(π′)≥Lπ(π′)−CDKLmax⁡(π,π′),C=4ϵγ(1−γ)2,DKLmax⁡=max⁡sDKL(π(⋅∣s)∥π′(⋅∣s))J(\pi\prime)\ge L_\pi(\pi\prime)-C D_{\rm KL}^{\max}(\pi,\pi\prime),\quad C=\frac{4\epsilon\gamma}{(1-\gamma)^2},\quad D_{\rm KL}^{\max}=\max_s D_{\rm KL}(\pi(\cdot\mid s)\Vert\pi\prime(\cdot\mid s))
Practical TRPO: average-KL constraint, approximately solved by conjugate gradients and line search [2]
max⁡θ Es∼dπ,a∼π ⁣[πθ(a∣s)π(a∣s)Aπ(s,a)]s.t.Es∼dπDKL(π∥πθ)≤δ,dπ=(1−γ)ρπ\max_\theta\ \mathbb E_{s\sim d_\pi,a\sim\pi}\!\left[\frac{\pi_\theta(a\mid s)}{\pi(a\mid s)}A_\pi(s,a)\right]\quad\mathrm{s.t.}\quad\mathbb E_{s\sim d_\pi}D_{\rm KL}(\pi\Vert\pi_\theta)\le\delta,\quad d_\pi=(1-\gamma)\rho_\pi

READ THE ORIGINALS

Original sources

  1. Original technical blog: filtering, Hot-Load, pipeline results
  2. Historical TRPO paper: theory, implementation and experiments
  3. Official ICML 2015 publication record
Search window

2026-10-22 08:49 — 2026-10-25 08:49 · America/New_York

READ · THINK · DISCUSS

Discussion & likes

Discuss a derivation, add a reference, or suggest the next classic.

Comment, reply, or like below. Sign in with GitHub to participate; comments are public.

Loading discussion…

Open discussion on GitHub ↗

← Back to all issues