The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search

Published
Source
arXiv
Paper number
1476
Field
AI / Agents
arXiv ID
2611.19037

Key points

  • The authors built a four-arm audit comparing no critic, random feedback, real critic, and oracle under the same budget and frontier.
  • A naive 8B critic scored below the no-critic floor on 7 of 8 binary tabular datasets.
  • The worst case on ADULT reached a loss of 0.245 AUC.
  • Fault injection showed that random errors are nearly costless while systematically directional errors consume effective search rounds.
  • An idealized model yields the necessary condition q(m−1) > β for a critic to help.
  • A validation gate and a learned detector reduced the average loss, but neither guaranteed the no-critic baseline on every dataset.

Paper links

External research summaries. These are not HDATF publications or measured product results.

Read original (opens in a new tab)