Critique of Agent Model

Key points

• Problem: As LLM systems marketed as coding agents and AI co-scientists multiply, it has become necessary to clarify where automation ends and agency begins, both for building capable systems and for judging what, if anything, to fear. The authors find that existing surveys cover what current systems can do and how to improve them broadly, yet tend to take agency for granted as a label that applies whenever an LLM interacts with an environment.
• Argument: This is a position paper with an analysis, an architecture proposal, and proved theorems. The authors separate agentic systems, which finish tasks through externally orchestrated tools and workflows, from agentive systems, which maintain and revise their own goals, identity, decision-making, self-regulation, and learning inside the model, and they hold that current AI systems are largely agentic. On safety, they argue that GIC is structurally well positioned because harmful behavior decomposes entirely into goal misspecification (the human supplied the wrong objective) and component imperfection, and they hold that the overall goal is supplied from outside and that subgoals, identity, predictions, and plans can be audited by humans at every layer.
• Taxonomy: The authors sort systems called agents by how their behavior is produced into five kinds: program-based bots, LLM wrapper systems (an LLM placed inside fixed orchestration), LLM-centered systems, model-less physical systems (such as industrial robots running programmed routines), and embodied-model systems (robots and driving systems that unify perception, reasoning, and control in one model). The common designs they examine along the five dimensions are short instructions supplied by a human at every step (goal), fixed system prompts with harness engineering (identity), black-box policies trained with end-to-end RL (decision-making), planning expected to emerge from RL or placed in fixed workflow stages (self-regulation), and human-scheduled training pipelines (learning). They argue that these designs share the absence of an internal world model that predicts the consequences of actions.
• Proposal: The GIC agent model has six components: a belief encoder (which turns observations into a belief state, an estimate of the world state), a goal decomposer (which splits a long-term goal into subgoals), an identity evolver (which revises the self-model without retraining), a configurator, a simulative planner, and an actor. The actor handles System I (immediate action), the simulative planner handles System II (predicting the outcomes of candidate actions with the world model and picking a plan), and the configurator handles System III (choosing at each step whether to make a new plan, continue or revise an existing one, or act directly). The world model is trained separately on next-state prediction alone, the two models keep disjoint parameters and connect only through activations and outputs, and training is proposed in three phases: component pretraining that starts from a pretrained LLM, simulative RL inside the world model, and refinement after real-world deployment.
• Evidence: Theorem 2 is a benefit: if the world model's next-state prediction error is bounded by ϵ in total variation (a distance between probability distributions), any policy can be mixed with selectively adopted world-model plans into a policy whose value (expected cumulative reward) is at least as high. Theorem 3 is a limitation of always-on, fixed-depth model-predictive control (MPC): the required horizon grows significantly as the desired planning precision rises. Theorem 1 shows the benefit that, when identity revisions are better than random, learning that revises identity at every step accumulates less expected regret than retraining alone, and Theorem 4 shows the benefit that the value of the best policy trained on a mixture of simulated and real experience is at least the value of the best real-only policy minus a term proportional to the world model's error, and at least as high when the world model is perfect.
• Limits: The authors state that details and preliminary results for scaled-down GIC instantiations appear in companion manuscripts. Of their proposed evaluation dimensions (Performance, Efficiency, Growth), those preliminary results give initial evidence on the first two, and Growth evaluation remains future work. The premise of Theorem 2, an upper bound ϵ on the prediction error, is credible only when the world model is trained for predictive fidelity, and the safety conclusion that harmful behavior converges to zero is stated for a sufficiently well-trained system, unless the goal itself is wrong.

Paper links

External research summaries. These are not HDATF publications or measured product results.