PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment
- Published
- Source
- arXiv
- Paper number
- 911
- Field
- Robotics
- arXiv ID
- 2608.14284
Key points
- Success rate alone cannot distinguish a near-miss failure from one that is blocked from the beginning, and it also misses hesitation and unnecessary motion during successful runs.
- The method turns execution videos into progress curves and computes multiple metrics for reached level, progress efficiency, stagnation, regression, and recovery.
- The new metrics FNS, DRR, and SQS show how close a failure was to success, how much recovery occurred after a large regression, and how stable the successful process was.
- On RoboDojo, model rankings differ between success rate and process metrics, and the rank correlation between simulation and real-world evaluation stays only between 0.18 and 0.58.
- RoboPulse++ evaluates the scoring model with 700 execution trajectories and 2,244 human-annotated segments, and even the best F1 score for progress-decrease segments is only 0.63, so the results should not be treated as absolute truth.
Paper links
External research summaries. These are not HDATF publications or measured product results.