RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs
- Published
- Source
- arXiv
- Paper number
- 1395
- Field
- Reasoning / RL
- arXiv ID
- 2611.02455
Key points
- The authors propose RLiD, which trains a large decision model that picks an action from an executable menu in one forward pass, removing the credit noise that token-level RL inherits from how actions are worded.
- Proposition 1 shows that marginalising over alternative wordings of the same action preserves the expected update and removes the variance component tied to realisation.
- In an exact FrozenLake credit audit, a decision-plane batch gradient with exact advantages reaches cosine 0.88 to the true gradient, while 71% to 97% of a token policy's score second moment is realisation variance.
- On ARC the Qwen2.5-1.5B decision model reaches 78.6% in 0.12 GPU-h, ahead of token GRPO at 75.9% with 1.17 GPU-h, and in the same-runner Sokoban the decision plane reaches 44.5%, above every token variant.
- On ALFWorld the full distillation and student-RL route reaches 99.3% success in 12.7 H100 GPU-hours, whereas token-level RL of the same LLM reaches 86.6% in 17.3 GPU-hours.
- Without the admissible-action list the generative distilled student keeps 99.3% unseen success, and without memory it reaches 90.5%.
Paper links
External research summaries. These are not HDATF publications or measured product results.