SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning
- Published
- Source
- arXiv
- Paper number
- 1388
- Field
- Reasoning / RL
- arXiv ID
- 2610.38391
Key points
- In annotated WebShop trajectories, the authors found an outcome-step mismatch: 29.8% of steps in successful trajectories are erroneous and 42.4% of steps in failed trajectories are correct.
- Distilling trajectory-level skills across all steps produces off-target signals at unrelated steps, which account for 57.9% of nonzero sampled-token gaps.
- SCOPE first learns error localization and step-specific skill generation through a supervised warm-up, then during online RL analyzes its own trajectories and applies distillation only at selected error steps.
- A bounded signed distillation coefficient prevents extreme teacher-student gaps from destabilizing training, blocking the collapse that occurred when the training-stage gap reached 2715.5 nats.
- With Qwen2.5-3B, SCOPE records 97.7% success on ALFWorld and 85.2% on WebShop, outperforming SEED by 5.9 and 6.3 percentage points respectively.
- On unseen ALFWorld tasks it reaches 89.6% success, 23.2 points above GRPO with 46.7% fewer interactions, and scaling to Qwen3-32B yields 99.2% success with mean steps of 7.7.
Paper links
External research summaries. These are not HDATF publications or measured product results.