IoDResearch: Deep Research on Private Heterogeneous Data via the Internet of Data

Key points

• Problem: Scientific data are often multi-source, heterogeneous and multimodal, and lacking unified representation standards and interoperability mechanisms lowers how efficiently they are used (materials science research, for example, typically handles heterogeneous data from experiments, simulations and literature records). Existing Deep Research agents rely mostly on web search with limited exploration of local heterogeneous and multimodal data, so they retrieve private data inefficiently and often fall short of the FAIR principles (findable, accessible, interoperable, reusable data).
• Input and output: It takes a user's private heterogeneous data (papers, datasets, images, tables and more) and a question or report topic, and returns relevant digital objects, a factual answer or a structured scientific report. This is a system paper that implements and evaluates a data representation and a multi-agent Deep Research system, and it also proposes the authors' own IoD (Internet of Data) Deep Research Benchmark.
• Method: Raw data are wrapped as digital objects (DOs) with a DOI (Digital Object Identifier) and metadata, long documents are split into Level-2 DOs per chunk, an LLM adds attributes such as summaries, hypothetical questions, classification labels and keywords to the metadata, and domain IoDs are joined into one IoD network. A knowledge refinement layer builds vectors, atomic knowledge (minimal factual units such as a typical aspirin dosage of 300 mg) and LLM-extracted knowledge graphs, and MCP (Model Context Protocol) tools offer retrieval at three granularities (DOs, chunks, facts and subgraphs) through strategies including keyword, vector, graph-reasoning and hybrid search. IoDAgents splits into a Planner that drafts a plan for user confirmation, a Worker Team that filters results with an LLM, refines and reissues queries when context is insufficient and uses a code tool for analysis and visualization, and a Reporter Team whose Writer drafts the output and whose Checker corrects inconsistencies with the retrieved context.
• Implementation: Documents are parsed with open-source tools such as MinerU, and every compared method runs on the same raw data with the same base LLM, Qwen-Turbo. The benchmark wraps over 500 documents and papers (about 6 million tokens) from four domains (Chinese law, geophysical exploration, computer science, molecular dynamics) as DOs, and has 200 digital object retrieval questions, 800 question answering (QA) questions for retrieval-augmented generation (RAG) (400 single-domain, 400 cross-domain multi-hop) and 60 report-writing questions (30 single-domain, 30 cross-domain). Reports are scored by Qwen-Turbo at temperature 0, averaged over three scorings on five criteria, and by human experts blind to the method who apply the same criteria plus factual accuracy.
• Results: In Task 1 (digital object retrieval), IoDResearch reached F1 82.64 (precision 76.26, recall 90.18), above LightRAG at 78.93, DO-RAG at 76.21 and Naive RAG at 62.05. In Task 2 (question answering), answer accuracy was 79.98 single-domain and 59.40 cross-domain, above LightRAG at 75.55 and 56.67, the best of the three baselines for this task (Naive RAG, DO-RAG, LightRAG). In Task 3 (report writing), human expert scores were 7.01 single-domain and 6.45 cross-domain, above DeepSearcher at 6.77 and 6.02, the best of the compared methods, and the ablation without the multi-agent component dropped to 6.56 and 5.94.
• Limits: The authors note that the pipeline is mainly text-based, so multimodal data can lose information during preprocessing and context construction, and they leave an end-to-end multimodal pipeline and post-training strategies to future work. Experiments use the authors' own benchmark across four domains with a single base LLM (Qwen-Turbo), and the LLM judge is the same Qwen-Turbo. The comparison with DeepSearcher covers only the report-writing task, and the conclusion limits the claim to the selected baselines and evaluated task settings.

Paper links

External research summaries. These are not HDATF publications or measured product results.