HDATF
English
Start a conversation
Menu

Paper Archive

Every archived paper with its summary, highest number first.

Papers 1,430Archive updated Research MonitoringBenchmarksRSS feed
  1. 081RL / WritingarXiv:2509.04903

    ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning

    • Existing supervised fine-tuning approaches for long-form generation rely heavily on high-quality datasets that are scarce and expensive.
    • Current reinforcement learning, or RL, methods use coarse and subjective evaluation metrics, which fail to capture the subtle and instruction-specific requirements that matter for high-quality long-form writing.
    • LLMs struggle to consistently satisfy fine-grained, multi-faceted, and instruction-specific requirements in complex long-form generation tasks.
    • An automated pipeline using a strong LLM decomposes high-level user instructions into a verifiable checklist of constraints covering content completeness, structural logic, and stylistic formality.
    • The new reward mechanism combines length rewards with fine-grained constraint rewards, and a smaller verifier LLM evaluates whether each constraint is satisfied with a three-level judgment.
    • Training uses the Group Relative Policy Optimization, or GRPO, algorithm, which encourages the LLM to produce long-form responses that satisfy both length and instruction-specific requirements through this comprehensive reward.
    AbstractPDF
  2. 080RAG / EfficiencyarXiv:2509.01092

    REFRAG: Rethinking RAG based Decoding

    • Retrieval-augmented generation (RAG) systems that use large language models (LLMs) suffer from substantial system latency and high memory consumption when processing long retrieved contexts.
    • This creates a tradeoff in which enriching knowledge with longer contexts often reduces system efficiency and throughput.
    • General-purpose optimization for long-context reasoning in LLMs does not properly handle the RAG-specific block-diagonal attention pattern and redundant computation inherent in RAG contexts.
    • A framework using a lightweight encoder compresses RAG contexts into chunk embeddings, greatly reducing the effective input sequence length for a decoder-only LLM.
    • The encoder and decoder are aligned through a continual pretraining (CPT) scheme that includes next-paragraph prediction and reconstruction tasks, with curriculum learning used to manage complexity.
    • A reinforcement-learning policy dynamically expands important context chunks into full token sequences and compresses less important chunks, enabling flexible compression at arbitrary positions.
    AbstractPDF
  3. 079Reasoning / RLarXiv:2509.03646

    Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning

    • Despite empirical success, the mechanistic understanding of how reinforcement learning (RL) improves complex reasoning in LLMs remained unclear.
    • Puzzling phenomena during RL training, such as the aha moment and length-scaling effects, lacked a unified explanation.
    • Existing RL algorithms often apply optimization pressure indiscriminately to every generated token, which leads to inefficient learning.
    • We functionally decomposed LLM-generated tokens into high-level planning tokens, such as logical maneuvers, and low-level execution tokens, such as calculations.
    • We introduced Strategic Grams (SGs), a data-driven method that empirically identifies planning tokens without subjective manual annotation.
    • We developed Hierarchy-Aware Credit Assignment (HICRA), an RL algorithm that concentrates optimization by amplifying advantage signals on identified planning tokens.
    AbstractPDF
  4. 078Research Agents / RLarXiv:2509.06283

    SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents

    • It equips large language models, or LLMs, with advanced reasoning and tool-use capabilities so they can carry out complex, long-horizon tasks while mitigating issues such as hallucination.
    • It develops an effective autonomous single-agent system for deep research that works without rigid multi-agent orchestration and generalizes better.
    • It stabilizes reinforcement learning, or RL, for multi-turn agentic tasks, where existing methods struggle with stability and can degrade the base model's reasoning ability.
    • It develops an agentic reasoning scaffold for a specific LLM family with the minimal set of three tools, Internet search, page viewing, and a code interpreter, and an iterative single-turn workflow.
    • It builds a new synthetic data generation pipeline that produces difficult multi-hop QA and long-form report-writing tasks for agentic training.
    • It implements an end-to-end RL recipe with a length-normalized REINFORCE objective, trajectory filtering, and partial rollouts to ensure stable policy optimization over multi-turn interactions.
    AbstractPDF
  5. 077Medical AI / RAGarXiv:2508.11386

    Retrieval-augmented reasoning with lean language models

    • State-of-the-art retrieval-augmented generation (RAG) and reasoning systems depend on large external language models, which introduce privacy and security risks in applications that handle sensitive data.
    • Deploying and running large frontier language models locally is computationally burdensome for many organizations because of their high resource demands.
    • General-purpose language models often lack deep, verifiable expertise in specialized domains, leading to inaccuracy or hallucination in critical applications.
    • The researchers used Qwen2.5-Instruct models ranging from 1.5B to 32B parameters as an efficient language-model backbone for reasoning and generation.
    • They developed a retrieval system using semantic embeddings, a vector database, and a new strategy that retrieves whole documents and summarizes them in advance to manage context length while preserving relevant information.
    • To supervise supervised fine-tuning of the lightweight language models, they generated synthetic domain-specific data with diverse user queries and detailed reasoning trajectories using high-performance models (GPT-4o and DeepSeek-R1).
    AbstractPDF
  6. 076Computer Use AgentsarXiv:2508.09123

    OpenCUA: Open Foundations for Computer-Use Agents

    • Most state-of-the-art computer-use agent, or CUA, systems are proprietary and closed-source, which makes it harder to study their capabilities, limitations, and risks.
    • There is no scalable open-source infrastructure for collecting diverse, large-scale computer-use data, including real-time user interaction and accessibility trees.
    • Existing open-source CUA datasets have limited scope, scale, and diversity, and they focus on narrow domains rather than general computer-use applications.
    • The authors developed AGENTNET TOOL, a cross-platform application that captures human computer-use demonstrations, including screen video, mouse and keyboard signals, and accessibility trees, across Windows, macOS, and Ubuntu.
    • They built AGENTNET, the first large-scale and diverse computer-use task dataset with 22,625 trajectories spanning multiple operating systems and more than 200 applications and websites.
    • To improve agent planning and error recovery, they propose a scalable training pipeline that integrates a new reflective long-chain-of-thought synthesis and multi-image history context encoding.
    AbstractPDF
  7. 075Computer Use AgentsarXiv:2508.14040

    ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents

    • The human-computer interface is inherently designed for people, which makes robust LLM-based GUI agents difficult to build because machine interaction is more complex.
    • Existing approaches such as behavior cloning suffer from scalability limits, weak generalization, and poor error recovery on diverse and complex desktop tasks.
    • Applying reinforcement learning to desktop automation has been constrained by heavy compute demands, slow convergence, instability, and entropy collapse, where policies lose exploration ability during long training.
    • The API-GUI interaction paradigm integrates direct GUI interaction with programmatic API calls, and it is complemented by an LLM-driven automation workflow for API development to improve efficiency and versatility.
    • To collect data efficiently, the team developed a scalable and reliable Ubuntu environment with distributed multi-node clustering infrastructure that can orchestrate thousands of concurrent virtual desktop environments.
    • They implemented a fully asynchronous RL framework with multi-LLM behavior cloning, step-level Group Relative Policy Optimization (GRPO) with verifiable rewards, and an Entropulse strategy for continued exploration through periodic RL-SFT alternation.
    AbstractPDF
  8. 074Agents / DistillationarXiv:2508.13167

    Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL

    • Existing multi-agent systems, or MAS, suffer from high computational overhead, generalization issues, and a lack of end-to-end data-driven training because LLM backbones are not inherently trained for multi-turn multi-agent workflows.
    • Existing tool-integrated reasoning, or TIR, models are constrained by rigid think-act-observe workflows and do not natively support flexible multi-agent collaboration patterns.
    • Both MAS and TIR paradigms fail to make effective use of end-to-end training to learn complex, dynamic multi-agent, multi-tool orchestration within a single unified LLM.
    • The Chain-of-Agents, or CoA, paradigm integrates multi-agent collaboration inside the decoding process of a single LLM, dynamically activating role agents such as thinking, planning, and reflection as well as tool agents such as search, crawling, and code generation.
    • An agentic supervised fine-tuning, or SFT, framework distills expert-level multi-agent trajectories into CoA-compatible training data and uses progressive quality filtering to ensure high-quality, complex problem-solving patterns.
    • Agentic reinforcement learning, or RL, further refines the LLM's multi-tool orchestration policy through outcome-based rewards, strengthening strategic tool use and adaptive reasoning on demanding task-specific queries.
    AbstractPDF
  9. 073Routing / EfficiencyarXiv:2508.12631

    Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing

    • Balancing high performance, including accuracy and reasoning, with computational efficiency, including cost and latency, is a core challenge in advancing large language models.
    • State-of-the-art proprietary LLMs often come with substantial operating costs and resource demands, which limit broad and cost-effective deployment.
    • Existing test-time routing methods for LLMs often focus mainly on aggregate performance or add overhead by requiring complex training of additional neural networks.
    • The paper develops Avengers-Pro, a test-time routing framework that coordinates diverse LLM ensembles to dynamically select the best model for each query.
    • The framework relies on three lightweight unsupervised operations: query embedding, k-means clustering of queries, and per-cluster model-level performance-efficiency scoring.
    • During online inference, it embeds incoming queries, assigns them to the nearest cluster, and routes them to the model with the highest aggregate performance-efficiency score, adjustable by the tradeoff parameter α.
    AbstractPDF
  10. 072Vision / ReasoningarXiv:2506.22146

    Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs

    • Large vision-language models, or LVLMs, struggle with precise visual understanding such as exact counting, spatial reasoning, and detailed scene description.
    • LVLMs suffer from the binding problem, which prevents them from reliably connecting perceptual features to the correct visual objects and spatial properties in complex scenes.
    • Existing LVLMs often process visual features in parallel, which causes interference and misattribution errors, especially in cluttered environments.
    • The VISER method adds evenly spaced horizontal lines to the input image so that the image is divided into separate visual compartments or rows.
    • It prepends a short text instruction such as 'Scan the image sequentially along the horizontal lines present in the image' to the original prompt so that the LVLM processes the content region by region.
    • This lightweight, model-agnostic approach introduces only negligible compute overhead and requires no architectural changes, fine-tuning, or multi-query inference.
    AbstractPDF
  11. 071Reasoning / DataarXiv:2507.01921

    NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks

    • There was no systematic understanding of which characteristics of reasoning demonstrations are most effective for distilling reasoning ability into smaller language models.
    • The prevailing Less is More hypothesis, which claims that a few handpicked examples are enough for reasoning distillation, mostly came from domain-specific settings and needs reevaluation for general reasoning.
    • Large language models often overthink and produce excessively long reasoning traces, which increases inference cost and reduces deployment efficiency in practice.
    • The authors built NaturalThoughts, a dataset that annotates reasoning traces generated by DeepSeek-R1, the teacher, for diverse NaturalReasoning questions by domain, meta-reasoning strategy, and verbosity.
    • They systematically explored data selection strategies based on diversity, such as question topic, semantic embedding, and reasoning strategy, as well as difficulty, such as answer length, verbosity, and model agreement.
    • They proposed and evaluated a mixed System 1/System 2 distillation approach, including random and difficulty-based mixtures, so student models can adjust reasoning depth at inference time.
    AbstractPDF
  12. 070Research Agents / SurveyarXiv:2506.18096

    Deep Research Agents: A Systematic Examination And Roadmap

    • Beyond existing LLMs and RAG, there is no clear definition or taxonomy for a new class of AI systems that integrate dynamic reasoning, adaptive planning, and iterative tool use.
    • There is a need to unify scattered research efforts into a single framework in order to understand common architectural patterns and foundational technologies.
    • Existing evaluation benchmarks do not adequately assess the full end-to-end capabilities of advanced AI research agents.
    • This paper formally defines deep research, or DR, agents and proposes a unified taxonomy that distinguishes static from dynamic workflows and single-agent from multi-agent systems.
    • It systematically analyzes foundational technologies, including information acquisition strategies such as API-based versus browser-based approaches, diverse tool-use capabilities such as code interpreters and multimodal processing, and memory mechanisms for optimizing long context.
    • It evaluates current tuning paradigms such as SFT, RL, and nonparametric continual learning, reviews key industry implementations, and critically assesses existing benchmarks for DR agents.
    AbstractPDF
  13. 069Security / AgentsarXiv:2506.23260

    From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows

    • The rapid spread of LLM-based AI agents and their complex ecosystems lacks standardized, robust security practices, leading to widespread vulnerabilities.
    • Current integrations often rely on ad hoc authentication and weak verification mechanisms, making them highly vulnerable to many forms of exploitation.
    • Existing security research on LLM agents is fragmented and has not provided a unified end-to-end framework that covers threats across the full communication stack, from host-to-tool interactions to agent-to-agent interactions.
    • A unified end-to-end threat model was developed that systematically analyzes security risks across the entire LLM agent ecosystem, including host-to-tool and agent-to-agent communication.
    • More than 30 different attack techniques were classified into four major areas: input manipulation, model corruption, system and privacy attacks, and protocol vulnerabilities.
    • Rigorous mathematical formulations were provided for each threat category, explicitly defining attacker capabilities, goals, and affected layers to enable systematic analysis.
    AbstractPDF
  14. 068RAG / AgentsarXiv:2506.21931

    ARAG: Agentic Retrieval Augmented Generation for Personalized Recommendation

    • Existing RAG systems for personalized recommendation rely on simple retrieval mechanisms, such as cosine similarity, that fail to capture nuanced user preferences.
    • Existing RAG methods struggle to understand implicit preferences, process long-form user documents, and dynamically rank items based on multifaceted contextual factors.
    • Using LLMs directly for recommendation is computationally expensive, prone to hallucination, and constrained by fixed knowledge cutoffs, so robust grounding is needed.
    • ARAG proposes a multi-agent framework in which specialized LLM agents collaborate to refine context retrieval and item ranking for personalized recommendation.
    • It uses a User Understanding Agent (UUA), a Natural Language Inference (NLI) agent, a Context Summarization Agent (CSA), and an Item Ranking Agent (IRA) to decompose the complex recommendation task.
    • The agents operate in a blackboard-style collaboration, with UUA and NLI running in parallel, followed by CSA and then IRA, to synthesize user preferences and align them with candidate items.
    AbstractPDF
  15. 067Scientific AI / SurveyarXiv:2507.01903

    AI4Research: A Survey of Artificial Intelligence for Scientific Research

    • Despite rapid progress in large language models and their scientific innovation applications, there has been no comprehensive and systematic survey that specifically covers the full cycle of scientific research, AI4Research.
    • Existing surveys usually focus on narrower subareas such as AI for scientific discovery, which has fragmented understanding of AI's broader impact on academic workflows.
    • The lack of an integrated perspective hinders systematic understanding and slows further progress in using AI across all stages of scientific research.
    • The authors developed a taxonomy that categorizes AI applications into five major tasks across the research cycle: scientific understanding, academic surveying, scientific discovery, academic writing, and academic peer review.
    • They formally defined AI4Research and distinguished it from the narrower AI4Science, while providing a rigorous component-level framework for understanding AI's role at each stage.
    • For each component, they conducted an extensive literature review to synthesize current methods, pioneering work, and empirical summaries, and they also organized practical resources such as tools, datasets, and benchmarks.
    AbstractPDF
  16. 066Agents / SLMsarXiv:2506.02153

    Small Language Models are the Future of Agentic AI

    • The current dominant LLM-centric paradigm for AI agents carries high economic cost because reasoning demands enormous compute.
    • General-purpose LLMs are operationally inefficient and add latency when they perform the repetitive, specialized sub-tasks common in agentic workflows.
    • The large energy consumption associated with LLMs raises sustainability concerns, and they often lack precise specialization for narrow agentic tasks.
    • This paper proposes a paradigm shift to specialized small language models as the primary compute unit for most agentic calls.
    • It advocates a heterogeneous agent architecture that uses SLMs for routine and specialized tasks and reserves LLMs for complex general reasoning.
    • It provides a practical six-step algorithm for converting LLM-centric agent applications into SLM specialists, including data collection, task clustering, fine-tuning, and continual improvement.
    AbstractPDF
  17. 065Reasoning / Small ModelsarXiv:2504.15777

    Tina: Tiny Reasoning Models via LoRA

    • Achieving robust multi-step reasoning in large language models remains a major challenge.
    • Current LLM reasoning enhancement methods, such as supervised fine-tuning and reinforcement learning, are resource-intensive and often require substantial compute and large models.
    • The high cost and complexity of RL pipelines limit both accessibility and broad research on RL-based reasoning for language models.
    • It uses the small 1.5-billion-parameter base model DeepSeek-R1-Distill-Qwen-1.5B, which is known for its initial reasoning aptitude.
    • It applies Low-Rank Adaptation (LoRA) for parameter-efficient reinforcement learning with a GRPO-style algorithm, updating only a subset of the model parameters.
    • To ensure cost efficiency and direct comparability with existing models, it uses a minimalist training setup with two NVIDIA L40S GPUs and public reasoning datasets.
    AbstractPDF
  18. 064RL / InferencearXiv:2504.16084

    TTRL: Test-Time Reinforcement Learning

    • Applying reinforcement learning to reasoning tasks for large language models is bottlenecked by the high cost and scarcity of human-annotated ground-truth labels needed for reward estimation.
    • Existing RL methods for LLMs rely heavily on supervised reward signals, which makes it difficult to keep improving on newly encountered or unlabeled test data.
    • The lack of explicit reward signals at inference time limits an LLM's ability to adapt to new distributions and improve generalization without human intervention.
    • It introduces Test-Time Reinforcement Learning, or TTRL, a framework that lets an LLM update its parameters and improve generalization at test time using RL.
    • It generates reward signals without explicit ground-truth labels by repeatedly sampling model outputs and deriving a consensus output through majority vote.
    • It computes a rule-based binary reward from whether each sampled output matches the majority-vote pseudo-label, and uses it to optimize the model policy with an RL algorithm such as GRPO.
    AbstractPDF
  19. 063Vision-LanguagearXiv:2504.16072

    Describe Anything: Detailed Localized Image and Video Captioning

    • Existing vision-language models struggle to generate detailed and accurate captions for specific regions in images and videos.
    • High-quality datasets for training detailed local captioning models are limited.
    • Current evaluation metrics often rely on reference captions that lack comprehensive detail.
    • It developed the DAM architecture for multi-granularity region captioning, with focal prompts and a localized vision backbone.
    • It built the SSL-based data pipeline DLC-SDP using segmentation datasets and unlabeled web images.
    • It introduced DLC-Bench for evaluating detailed local captions without reference captions.
    AbstractPDF
  20. 062GUI AgentsarXiv:2501.12326

    UI-TARS: Pioneering Automated GUI Interaction with Native Agents

    • Existing GUI agents rely heavily on text-based representations and modular frameworks, which limit scalability.
    • There is a lack of large, high-quality data for training end-to-end GUI agents.
    • Complex GUI scenarios require advanced perception, reasoning, and coordination capabilities that current approaches do not have.
    • We developed UI-TARS, an end-to-end model that interacts with interfaces by processing screenshots directly.
    • We built large-scale datasets for perception, grounding, and action traces through automatic collection and curation.
    • We integrated System 2 reasoning through thought augmentation and reflection tuning.
    AbstractPDF
  21. 061LLMs / EfficiencyarXiv:2504.12285

    BitNet b1.58 2B4T Technical Report

    • Large language models are compute-intensive, requiring substantial memory and energy and causing high latency.
    • These resource demands constrain LLM deployment on edge devices and in real-time applications.
    • Previous 1-bit quantization methods either hurt performance, as in post-training quantization, or failed to scale to competitive model sizes.
    • It implements a modified Transformer architecture with custom BitLinear layers for 1.58-bit weights and 8-bit activations.
    • It uses a multi-stage training approach that begins with an unprecedented 4 trillion-token pretraining run and continues with supervised fine-tuning and Direct Preference Optimization.
    • It develops and open-sources a specialized inference library with custom kernels for efficient W1.58A8 operations on both GPUs and CPUs.
    AbstractPDF
  22. 060Reasoning / RLarXiv:2504.13837

    Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

    • It is commonly believed that, like traditional RL discovering new strategies, RLVR gives LLMs new reasoning abilities.
    • No rigorous study has determined whether current RLVR truly expands the range of problems LLMs can solve or merely optimizes sampling of existing solutions.
    • Traditional evaluation metrics often fail to capture an LLM's full reasoning potential or its reasoning capability boundary.
    • Pass@k is introduced as a robust metric that defines the LLM's reasoning capability boundary by measuring the fraction of problems solvable within k attempts.
    • The study systematically evaluates reasoning ability across diverse LLM families, sizes, multiple RL algorithms such as PPO and GRPO, and task domains such as math, code, and visual reasoning.
    • Deep analysis methods, including perplexity analysis and solvable-problem coverage, are used to determine whether RLVR-generated reasoning paths are genuinely novel or already lie within the base model's distribution.
    AbstractPDF
  23. 059Architecture / TrainingarXiv:2503.24322

    NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation

    • Traditional backpropagation is not biologically plausible because of the weight transport problem and the need to synchronize forward and backward passes.
    • Full backpropagation requires storing intermediate activations, which creates substantial memory overhead and limits true parallelization across network layers because of sequential dependencies.
    • The global error signal and sequential credit assignment in backpropagation can cause problems such as catastrophic forgetting and interference in lifelong learning settings.
    • It introduces NoProp, a block-wise neural network learning method inspired by denoising diffusion and flow matching models.
    • Each network block independently learns to denoise a noisy version of the target label using local backpropagation only within that block.
    • It broadcasts the input and target labels to all blocks, allowing each block to optimize its objective independently without global error-signal propagation.
    AbstractPDF
  24. 058Agents / Self-AwarenessarXiv:2504.03553

    Agentic Knowledgeable Self-awareness

    • Current LLM agents lack situational self-awareness, so they use cognitive resources such as reflection and external knowledge in a scattered and inefficient way.
    • Existing agent training methods are vulnerable to unexpected signals, prone to pattern collapse, and often incur high reasoning costs because they rely on a one-size-fits-all approach.
    • Agents also struggle to assess their own abilities dynamically in complex and changing environments, which leads to suboptimal decisions.
    • KnowSelf introduces a three-stage situational classification scheme, Fast, Slow, and Knowledgeable Thinking, to determine an agent's real-time cognitive resource needs.
    • It builds a lightweight offline knowledge system and generates self-awareness data by appending special tokens that indicate the type of thinking required for the trajectories the agent explored.
    • A two-stage fine-tuning process, SFT and RPO, teaches agents to connect situational cues with the right cognitive response, enabling them to activate reflection or external knowledge dynamically during inference.
    AbstractPDF
  25. 057Knowledge Graphs / LLMsarXiv:2504.03137

    LightPROF: A Lightweight Reasoning Framework for Large Language Model on Knowledge Graph

    • Existing KG-enhanced LLM approaches rely on large, computationally expensive models and complex KG traversal strategies.
    • Current methods do not exploit structural information in knowledge graphs effectively, which leads to redundant information and inaccurate reasoning.
    • It is a three-module framework that combines reasoning-graph retrieval, knowledge embedding through a small Transformer adapter, and mixed-prompt reasoning.
    • It is a parameter-efficient training method that freezes the LLM parameters and updates only the Knowledge Adapter.
    • It improves knowledge integration by converting both text content and graph structure into embeddings that are used as soft prompts.
    • It uses constrained breadth-first search with LLM-based scoring to retrieve relevant reasoning paths in the knowledge graph.
    AbstractPDF
  26. 056Pre-training / ReasoningarXiv:2504.04022

    Rethinking Reflection in Pre-Training

    • There is a common belief that complex cognitive abilities such as reflection in large language models emerge mainly during post-training stages like fine-tuning and reinforcement learning.
    • There is no systematic, scalable method for measuring reflection across different stages of pretraining.
    • Benchmarking reflection with standard reasoning datasets is difficult because reflective behavior is rare and error patterns are diverse.
    • To provide a finer-grained measurement framework, it defines multiple dimensions of reflection: situational, self, explicit, and implicit.
    • To systematically elicit and measure reflection by introducing human-like errors into reasoning chains, it develops a programmatic method for generating adversarial chain-of-thought datasets.
    • It uses a prompt-based LLM classifier to detect explicit reflection and measures four key metrics: accuracy, explicit reflection rate, explicit reflection accuracy, and implicit reflection accuracy.
    AbstractPDF
  27. 055Reasoning / EfficiencyarXiv:2504.05185

    Concise Reasoning via Reinforcement Learning

    • Large language models strengthened for reasoning tasks through reinforcement learning, or RL, tend to produce excessively verbose chain-of-thought outputs, which increases compute cost and inference latency.
    • The prior literature contains conflicting observations about whether response length is necessary for reasoning accuracy, with some studies finding that longer responses improve accuracy while others observe diminishing returns.
    • The fundamental mechanism by which RL training affects response length, and whether this verbosity is an essential part of reasoning or merely a byproduct, is not well understood.
    • The paper casts reasoning as a Markov decision process and performs a theoretical analysis of the loss dynamics of Proximal Policy Optimization, or PPO, and Generalized Reinforcement Learning with Policy Optimization, or GRPO, to explain how negative rewards push the policy toward verbosity.
    • It proposes a new two-stage RL training procedure that first strengthens reasoning on hard problems and then explicitly enforces conciseness on a small dataset of problems that are sometimes solvable.
    • It uses a built-in property of PPO, with generalized advantage estimation lambda below 1, where positive rewards encourage shorter responses and negative rewards encourage longer ones, to guide the model toward conciseness without sacrificing accuracy.
    AbstractPDF
  28. 054InterpretabilityarXiv:2504.07096

    OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens

    • LLMs behave as black boxes, making it hard to understand why they produce specific outputs.
    • Existing interpretability methods such as influence functions are computationally intractable at the trillions-of-token scale of modern LLM training datasets.
    • The lack of tools to efficiently analyze memorization in LLMs, trace factual claims back to sources, or identify the origins of generated text raises concerns about transparency, copyright, and PII.
    • OLMoTrace developed a system that uses an expanded infini-gram index to efficiently find maximal exact-match spans within a trillion-token-scale training dataset.
    • The method has five stages: identify maximal matching spans, filter long and unique spans, retrieve containing documents, merge duplicates, and rerank documents by relevance using BM25.
    • The production system uses high-IOPS SSDs on Google Cloud to guarantee the disk I/O throughput needed for real-time performance on large-scale data.
    AbstractPDF
  29. 053Reasoning / SurveyarXiv:2503.24377

    Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models

    • Current LLMs show inefficient reasoning behaviors such as excessive computation and unnecessary deliberation.
    • There is a lack of systematic understanding of how to optimize the tradeoff between reasoning ability and compute cost.
    • The survey develops a structured framework that classifies reasoning inefficiency and potential solutions.
    • It classifies optimization approaches into behavior regulation after training and usage improvement at test time.
    • Reasoning inefficiency arises from both model behavior, such as length bias and deceptive thinking, and model usage, such as poor algorithm choice and compute allocation.
    • The proposed solutions should address both post-training factors, such as data quality and structure, and test-time strategies such as budget allocation and adaptive decoding.
    AbstractPDF
  30. 052Reasoning / EfficiencyarXiv:2504.00810

    Z1: Efficient Test-time Scaling with Code

    • Large language models consume too many tokens during reasoning tasks, even on simple problems that do not need elaborate thinking steps.
    • Prior approaches lack an efficient way to scale test-time reasoning while controlling token usage.
    • The model is fine-tuned on a carefully selected code reasoning dataset of 107K examples with balanced short and long trajectories.
    • The method implements a Shifted Thinking Window that limits reasoning tokens and forces a direct answer once the threshold is exceeded.
    • It removes context separators so that reasoning depth can be adjusted more flexibly.
    • The results suggest that longer reasoning trajectories and larger dataset scale are key to eliciting effective reasoning.
    AbstractPDF
  31. 051Search / AgentsarXiv:2503.20201

    Open Deep Search: Democratizing Search with Open-source Reasoning Agents

    • Existing proprietary search AI solutions limit transparency, innovation, and community-driven development.
    • Current open-source search tools lack sophisticated query processing and contextual retrieval capabilities.
    • It presents Open Search Tool, which includes query reformulation, advanced retrieval, and context augmentation.
    • It presents two reasoning agent architectures, ReAct and CodeAct, that interpret queries and coordinate tool use.
    • It is a plug-and-play framework that can integrate with any underlying language model.
    • Adaptive web search based on result quality improves both efficiency and accuracy.
    AbstractPDF
  32. 050InterpretabilityarXiv:2504.02732

    Why do LLMs attend to the first token?

    • The persistent, unexplained phenomenon in which large language models disproportionately attend to the first token, called an attention sink, lacked a functional explanation.
    • Information degradation in deep Transformers, including rank collapse, representation collapse, and over-squashing, causes loss of individual token information, especially in long sequences.
    • Prior literature did not provide a fundamental understanding of why attention sinks occur or what functional purpose they serve.
    • This paper theoretically connects attention sinks to excessive mixing and representation collapse, and derives a new over-squashing bound that quantifies how sinks regulate information flow.
    • We performed extensive empirical validation on state-of-the-art LLMs Gemma 7B and the LLaMa 3.1 family through perturbation analysis, close inspection of attention patterns, and targeted ablation studies.
    • Controlled pretraining experiments demonstrated a causal relationship between context length, model scale, and the formation and strength of attention sinks.
    AbstractPDF
  33. 049Medical AI / RAGarXiv:2503.23513

    RARE: Retrieval-Augmented Reasoning Modeling

    • LLMs exhibit knowledge hallucination and insufficient reasoning ability in specialized domains.
    • Existing approaches such as RAG mainly supplement knowledge at inference time and do not systematically train domain-specific reasoning during learning.
    • Conventional fine-tuning or pretraining for domain knowledge is costly, causes knowledge entanglement, and remains vulnerable to hallucination because of parameter budget limits.
    • It externalizes domain knowledge into searchable sources and internalizes domain-specific reasoning patterns inside the LLM.
    • It distills high-quality step-by-step reasoning responses that include retrieved knowledge using a teacher model.
    • It trains a lightweight student model with a masked-loss objective that injects retrieved knowledge into the prompt, forcing the model to focus on integrating the provided information and reasoning from it.
    AbstractPDF
  34. 048Research AgentsarXiv:2503.22708

    CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation

    • Existing autonomous scientific discovery, or ASD, systems often focus on only part of the research pipeline or rely on domain-specific languages, which limits the range of possible discoveries.
    • There is a need for an end-to-end semi-automated system that uses large language models for scientific discovery while emphasizing code-based experimentation.
    • Evaluating the reliability and reproducibility of discoveries made by LLM-based ASD systems is difficult because of the inherent variability of language model outputs and the potential unreliability of generated code.
    • CODESCIENTIST implements a semi-automated end-to-end scientific discovery workflow consisting of ideation, planning, experiment design and execution, reporting, and meta-analysis.
    • It uses a mutator-style LLM genetic search approach for ideation, combining fragments of research papers and code blocks to generate diverse research ideas.
    • The system generates and runs code in an instrumented sandbox, uses iterative generate run reflect debug loops, and performs multiple independent executions for each experiment to ensure reproducibility.
    AbstractPDF
  35. 047LLMs / EnterprisearXiv:2504.00698

    Command A: An Enterprise-Ready Large Language Model

    • General-purpose LLMs often lack optimization for enterprise workflows such as retrieval-augmented generation, complex agentic tasks, and robust multilingual support.
    • Deploying state-of-the-art LLMs in enterprise settings is often compute-intensive and expensive, which hinders on-premise or privacy-preserving deployment.
    • Integrating diverse capabilities into a single LLM without catastrophic forgetting or performance tradeoffs remains a major challenge in large-scale model training.
    • Command A balances efficiency and performance with a decoder-only Transformer that uses a new hybrid attention mechanism and Grouped-Query Attention.
    • Its distributed post-training approach trains six specialized expert models, such as Code, Safety, and RAG, on domain-specific data and then merges their parameters into a unified soup model with integrated capabilities.
    • The refinement stage integrates advanced supervised fine-tuning and reinforcement-learning techniques, including Self-improving Robust Preference Optimization and online RLHF for human alignment and specific abilities.
    AbstractPDF
  36. 046Benchmarks / Research AgentsarXiv:2504.01848

    PaperBench: Evaluating AI's Ability to Replicate AI Research

    • Existing AI agent benchmarks are often too narrow and too simple to evaluate performance on long-horizon, open-ended machine learning R&D tasks.
    • In particular, safety and governance frameworks need rigorous, objective, and scalable measures of an AI model's autonomy and ability to conduct scientific research.
    • Human reproduction of research papers is time-consuming and subjective, which makes it hard to scale for consistent evaluation of AI agents.
    • OpenAI researchers introduced PaperBench, a benchmark of 20 recent ICML papers that requires AI agents to reproduce empirical results from scratch without access to the original codebase.
    • For each paper, a detailed hierarchical rubric developed with the original authors provides binary pass or fail criteria across 8,316 specific contribution items.
    • SimpleJudge, an LLM-based automatic judge validated at an F1 score of 0.83 against human grading benchmark JudgeEval, provides scalable and consistent evaluation of agent submissions.
    AbstractPDF
  37. 045Image GenerationarXiv:2502.17437

    Fractal Generative Models

    • Traditional generative models struggle with the computational burden and artificial sequentialization required by high-dimensional, non-sequential data such as raw images.
    • Achieving high-fidelity pixel-level image generation at high resolution is computationally infeasible and difficult because of the enormous dimensionality.
    • Modularity in existing deep learning is often limited to layers or simple blocks, leaving it without the higher-level abstractions needed to compose full generative models.
    • It proposes a new generative modeling paradigm, 'fractal generative models,' that recursively calls existing generative models as atomic generation modules to form self-similar structures.
    • The framework is instantiated as autoregressive (AR) models for pixel-level image generation, FractalAR and FractalMAR, using a divide-and-conquer strategy over image patches.
    • At each hierarchical level, it uses multi-scale patching and local attention mechanisms to greatly reduce the quadratic computational cost of high-resolution image generation.
    AbstractPDF
  38. 044RL / AgentsarXiv:2502.19328

    Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems

    • Existing reward models rely too heavily on human preferences, which can introduce bias and accuracy problems.
    • Current approaches lack systematic verification of factual accuracy and instruction following.
    • Traditional reward models often fail to capture multiple important evaluation criteria at the same time.
    • The authors developed a multi-agent reward system that combines specialized verification agents with human preference signals.
    • They built a router component that intelligently decides which verification agents to call.
    • They also implemented a unified mechanism that combines scores from multiple evaluation sources.
    AbstractPDF
  39. 043Math ReasoningarXiv:2503.00735

    LADDER: Self-Improving LLMs Through Recursive Problem Decomposition

    • Reinforcement learning for large language models (LLMs) requires a progressive curriculum of difficulty in the training tasks, but that is often unavailable and leads to training stagnation or catastrophic performance collapse.
    • Complex reasoning domains such as mathematics pose a substantial challenge because there is a large gap between problems the model can solve and those it cannot solve yet.
    • Existing LLM training and improvement methods often rely heavily on large human-curated datasets or continuous human supervision and feedback.
    • The LADDER framework enables recursive problem decomposition, allowing the LLM to autonomously generate a tree of progressively simpler variants of a complex problem it cannot currently solve, using transformation libraries and induced prompting.
    • A robust numerical verification framework evaluates answer correctness and provides a clear, objective reward signal for reinforcement learning.
    • Test-Time Reinforcement Learning (TTRL) extends LADDER by generating problem-specific focused variants at inference time for previously unsolved test problems, enabling targeted and dynamic learning on individual hard questions.
    AbstractPDF
  40. 042Alignment / SafetyarXiv:2502.16797

    Forecasting Rare Language Model Behaviors

    • Standard LLM evaluations cover thousands of queries, but deployed models process billions, leaving a significant blind spot for rare but potentially harmful behaviors.
    • Behavior that appears safe in limited tests can create substantial risk when exposed to far more queries in deployment.
    • Traditional evaluation methods do not account for the fundamental scale gap or the stochastic nature of language models, which can generate different answers to the same query.
    • Researchers from Anthropic, Mila, and UC Berkeley developed a Gumbel-tail method to predict the risk of undesirable LLM behavior at deployment scale using limited evaluation data.
    • The method estimates each query's trigger probability by repeatedly sampling the model and measuring the fraction of responses that exhibit the target behavior.
    • It uses extreme value theory, specifically the Gumbel distribution, to model and extrapolate the tail behavior of these trigger probabilities from a small set of evaluation queries.
    AbstractPDF
  41. 041Reasoning / RLarXiv:2503.01307

    Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs

    • Some large language models achieve substantial self-improvement through reinforcement learning on verifiable reasoning tasks, while others quickly plateau under the same training.
    • It remained unclear which intrinsic properties or early capabilities enable LLMs to use extra compute effectively for self-improvement.
    • There was limited understanding of the foundational traits that determine success or failure in RL-based self-improvement for LLM reasoning.
    • The team developed a framework that uses a GPT-4o-mini classifier to identify and quantify four cognitive behaviors in LLM reasoning traces: verification, backtracking, subgoal setting, and reverse chaining.
    • They ran comparative RL experiments on models with different self-improvement abilities, Qwen-2.5-3B and Llama-3.2-3B, in the Countdown game.
    • They induced self-improvement with targeted interventions, such as fine-tuning models on synthetic data rich in specific cognitive behaviors and curating pretraining data to amplify those behaviors.
    AbstractPDF
  42. 040Reasoning / SurveyarXiv:2502.21321

    LLM Post-Training: A Deep Dive into Reasoning Large Language Models

    • Pretrained large language models (LLMs) often struggle with factual accuracy and produce hallucinated or logically inconsistent outputs.
    • LLM reasoning is primarily based on statistical patterns rather than explicit logical inference, which limits performance on complex reasoning tasks.
    • Despite massive pretraining, LLMs may still be misaligned with human intent, preferences, and ethical considerations, so additional refinement is needed for trustworthy deployment.
    • This paper systematically surveys and categorizes existing post-training methodologies for LLMs, with a focus on their role in improving reasoning ability.
    • It organizes these techniques into three connected stages: fine-tuning (for example, instruction, CoT, PEFT), reinforcement learning (for example, RLHF, DPO, RLAIF), and test-time scaling (for example, CoT prompting, Tree-of-Thoughts, MCTS).
    • The survey highlights the specific points at which each technique helps refine LLM behavior, improve robustness, and ensure adaptability across applications.
    AbstractPDF
  43. 039Training EfficiencyarXiv:2503.02875

    The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models

    • Existing methods for improving LLM reasoning require either massive labeled data or computationally expensive sampling.
    • Traditional fine-tuning approaches struggle with efficiency and scalability on complex reasoning tasks.
    • The method exploits the Prefix Self-Consistency phenomenon, where different solution paths share a common early reasoning prefix.
    • We develop UPFT, a method that fine-tunes the model on extracted prefix substrings while preserving the reasoning structure.
    • It integrates structure tuning to prevent catastrophic forgetting of reasoning ability.
    • The initial tokens in a reasoning process carry key information and remain highly consistent across different solutions.
    AbstractPDF
  44. 038Architecture / ReasoningarXiv:2502.13842

    Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking

    • Small large language models, or LLMs, face an inherent performance bottleneck when handling complex reasoning tasks because of fixed compute budgets and limited parameter counts.
    • The traditional approach of continually scaling LLM parameters to improve performance runs into diminishing returns and rapidly rising compute and deployment costs, raising sustainability concerns.
    • Empirical analysis shows that "hard" tokens induce persistent gradient oscillations and sharp spikes across Transformer layers, indicating structural burden and insufficient depth for processing complex information.
    • It introduces the concept of an Inner Thinking Step, which breaks the generation of a single token into a series of internal reasoning steps and repeatedly refines its representation.
    • It implements Residual Thinking Connections, or RTC, which accumulate the output of each thinking step to enable gradual representation refinement and break through performance bottlenecks.
    • It applies Adaptive Token Routing, or ATR, to dynamically assign extra thinking steps only to the most important tokens, avoiding unnecessary work on easy tokens and improving compute efficiency, while Thinking Step Encoding, or TSE, provides contextual information at each step.
    AbstractPDF
  45. 037Reasoning / AgentsarXiv:2502.08235

    The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks

    • LRMs tend to prefer extended internal reasoning over interaction with the environment, which leads to inefficient performance.
    • There is no systematic framework for analyzing and quantifying overthinking behavior in AI systems.
    • Understanding the balance between reasoning and action in interactive environments remains limited.
    • This paper develops a comprehensive framework for studying and measuring excessive deliberation in LRMs.
    • Using SWE Bench Verified, the authors analyzed 4,018 trajectories to identify patterns.
    • They introduced a scoring system that was validated against expert human evaluation.
    AbstractPDF
  46. 036Efficiency / Multi-modelarXiv:2502.14815

    Optimizing Model Selection for Compound AI Systems

    • Compound AI systems usually use the same LLM for every module, which limits overall performance.
    • There has been no systematic approach to optimizing model selection across different system components.
    • We developed LLMSelector, a framework that exploits monotonic performance relationships.
    • It efficiently assigns models to modules using LLM-based performance estimation.
    • It implements an iterative optimization approach through sequential module nomination.
    • End-to-end performance is often monotonic with respect to module-level performance.
    AbstractPDF
  47. 035Benchmarks / Code AgentsarXiv:2502.12115

    SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?

    • Existing LLM coding benchmarks rely on unit tests, which makes them vulnerable to grader hacking and insufficient for validating complex full-stack software engineering tasks.
    • Earlier evaluations lack direct economic metrics, so they cannot quantify the real monetary value of LLM outputs in realistic software development scenarios.
    • Current benchmarks are often too narrow, focusing on isolated programming problems or specific repositories rather than the diverse, ambiguous, and end-to-end nature of commercial freelance software engineering work.
    • We developed SWE-Lancer, a benchmark built from more than 1,400 real freelance software engineering tasks at Expensify with up to $1 million in actual monetary rewards.
    • It includes two task types: individual-contributor (IC) SWE tasks, which require code patch generation, and SWE manager tasks, where the model must choose the best solution among multiple proposals.
    • We adopted a rigorous five-stage pipeline that uses end-to-end (E2E) tests written in Playwright and triple-validated by expert engineers to ensure robust, hack-resistant evaluation.
    AbstractPDF
  48. 034LLMs / DiffusionarXiv:2502.09992

    Large Language Diffusion Models

    • The common belief is that the capabilities of large language models, such as scalability and in-context learning, come only from autoregressive architectures.
    • Autoregressive models have limitations, including the curse of reversal, where they struggle with tasks that require bidirectional or reverse reasoning.
    • There has been no large language diffusion model that can match the performance of billion-parameter autoregressive models.
    • We developed LLaDA, a masked diffusion model for discrete token sequences, using a Transformer backbone without causal masking.
    • We optimized a principled cross-entropy loss that upper-bounds the negative log-likelihood and trained an 8B-parameter LLaDA model from scratch on a 2.3-trillion-token dataset.
    • We implemented a flexible iterative sampling strategy at inference time, including low-confidence remasking to balance generation quality and speed.
    AbstractPDF
  49. 033Architecture / EfficiencyarXiv:2502.11089

    Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

    • Traditional attention in LLMs has quadratic complexity in sequence length, making long-context processing extremely expensive during training and inference.
    • Existing sparse attention methods often fail to turn theoretical efficiency gains into real latency reductions because of inefficient memory access patterns and incompatibility with modern hardware structures such as MQA/GQA.
    • Most sparse attention approaches are inference-only and require a pretrained full-attention backbone, which can hurt performance or make end-to-end training impractical or inefficient.
    • Native Sparse Attention (NSA) introduces a hierarchical sparse attention mechanism that combines three parallel branches: compressed attention for global context, selected attention for finely important tokens, and sliding-window attention for local context.
    • A learnable gating mechanism dynamically aggregates the outputs of these branches, allowing coarse and fine information to be flexibly combined while maintaining high sparsity.
    • NSA uses dedicated Triton-based kernels designed for hardware alignment, focusing on group-centered data loading and shared KV fetching to optimize memory access and maximize GPU tensor-core utilization in both training and inference.
    AbstractPDF
  50. 032Reasoning / SurveyarXiv:2502.03671

    Advancing Reasoning in Large Language Models: Promising Methods and Approaches

    • Large language models, or LLMs, often lack robust and systematic reasoning ability, which makes them struggle with complex tasks such as logical deduction, mathematical problem solving, and multi-step reasoning.
    • These shortcomings lead to serious problems such as factual inaccuracy, inconsistency, and hallucination, which limit the reliability of LLMs in critical applications.
    • Integrating the data-driven paradigm of modern LLMs with the structured reasoning of classical AI remains an ongoing and difficult challenge.
    • This paper conducts a comprehensive literature review that systematically collects and analyzes prior work on improving LLM reasoning.
    • It classifies the many reasoning improvement techniques into three major groups: prompting strategies, structural innovations, and learning-based approaches.
    • The survey also reviews the benchmarks and metrics used to evaluate LLM reasoning and identifies open problems and future research directions.
    AbstractPDF