When Fine-Grained Credit Assignment Helps and Hurts: Multimodal Agentic Reinforcement Learning through Video Editing

Key points

The authors built 797 editing tasks of four types, Content Selection, Narrative Sequencing, Rough Cut, and Fine Cut, from 49 real films and their released trailers.
Training Qwen3.5-9B with outcome-only GRPO raised the overall score from 0.284 to 0.413, so outcome rewards alone stayed competitive on the shorter-horizon tasks.
GRPO+PRM with a fixed process-reward weight scored only 0.334 overall with 44.1% truncated rollouts, showing reward hacking where process rewards kept rising while task completion rates and outcome scores declined.
Decaying the process-reward weight to zero between updates 20 and 30 gave the best overall score of 0.447 and cut the truncation rate to 6.2%, mitigating the hacking while preserving early gains.
On the longest Fine Cut tasks, decaying process rewards delivered a 45.4% relative gain over outcome-only GRPO, while on the three shorter-horizon task types the relative gain was at most 10.4%.
On-policy self-distillation (OPSD), a token-level signal, scored 0.277 with reference answers and collapsed to 0.044 with turn-level suggestions, showing that dense supervision alone does not guarantee effective learning.

External research summaries. These are not HDATF publications or measured product results.