JevGPT: Thinking Fast and Slow in Autonomous Research

Key points

• Problem: A research agent must gather evidence within a limited budget, and candidate sources returned by search have to be inspected before they can inform the next decision. Even with broad retrieval coverage, the evidence that actually informs decisions can be narrower, and the paper aims to close that gap.
• Input and output: It takes a fixed document corpus plus one question or one search topic, and returns an answer for a question or a ranked document list for a topic. This is a system paper with an implementation and experiments.
• Method: System Two turns the parts of the task that still need support into search queries or evidence requests, then merges the returned evidence to choose the next action and the final answer. System One runs three judgment types (probability, choice among options, ordinal score) in parallel over candidate sources, locating evidence passages for question answering (Evidence Localization) or scoring relevance for ranking (Relevance Assessment), and returns the selected original passages with their locations and judgments. Evidence Discovery pre-screens top search candidates against the original task while System Two keeps working and adds new leads to the research history.
• Implementation: The main setup uses GPT-5.5 at medium reasoning effort as System Two and Jev-1.13.0, a fast judgment model from TypeSafe AI, as System One, with a 300-second research budget per task. Retrieval is Hybrid BM25 (BM25 is a term-matching retrieval score), which min-max normalizes the document BM25 score and the best passage BM25 score and adds them with equal weight. Per the official README, the release includes the JevGPT runtime, frozen configurations, evaluation scripts and per-item records for Tables 1 and 2, and new runs need GPT and Jev access plus the benchmark indexes.
• Results: On 830 BrowseComp-Plus questions, JevGPT reached 90.48% accuracy, above the strongest baseline Reason-ColBERT at 90.00%, at a per-question API cost of $0.6081 for System Two and $0.0600 for System One against $1.6090 for Reason-ColBERT. On 105 NeuCLIRBench topics it scored 0.5390 nDCG@20 (ranking quality of the top 20 documents), above 0.5240 for Pi-Serini, the strongest research baseline. In a 415-question ablation with GPT-5.6 Terra, adding Hybrid BM25, Evidence Localization and Evidence Discovery to Pi-Serini in turn raised accuracy from 72.05% to 86.51%.
• Limits: Evaluation covers question answering and document ranking over fixed corpora, and the authors leave open-web research and long-form synthesis for later work. Fast judgment and extended reasoning sit in separate components, and a single model that learns when to use each is presented as the next step. In the ablation, adding Evidence Localization reduced searches but increased model usage and API cost.

Paper links

External research summaries. These are not HDATF publications or measured product results.