MOSAIC: An Omni-Modal Benchmark for Long-Term Memory and Cross-Modal Reasoning in Conversational Agents
- Published
- Source
- arXiv
- Paper number
- 1362
- Field
- Agents / Memory
- arXiv ID
- 2610.29249
Key points
- MOSAIC is a benchmark of 923 questions (784 answerable and 139 refusal controls) over 11–24-session conversational histories spanning text, image, video, and audio.
- The evaluation splits the pipeline into WRITE, RETRIEVE, DELIVER, and READ stages and changes one stage at a time while freezing the question, reader, prompt, and budget for paired comparison.
- Gold symbolic facts scored 0.846, leaving 15.4 points of residual reading error; gold native evidence dropped to 0.783 and generic captions and ASR text to 0.690.
- Retrieval cost 7.6 points and production packing caps cost another 3.8 points on identical ranked hits, with the full pipeline reaching 0.653.
- By modality, video dropped 17.1 points from gold symbolic facts to native video, making perception the largest candidate bottleneck, and on spatial-configuration items raw images beat OCR-only text by 32.0 points.
- Tighter media budgets collapsed complete-chain delivery from 0.675 to 0.184, and no system answered well and abstained precisely at the same time.
Paper links
External research summaries. These are not HDATF publications or measured product results.