Omni-Workspace: Benchmarking Omni-Modal AI Assistants on Multimodal Workspace Tasks

Published
Source
arXiv
Paper number
1477
Field
AI / Agents
arXiv ID
2611.19202

Key points

  • The authors built a benchmark of 380 tasks in workspaces mixing audio, video, image, and document files.
  • Evaluating 18 models, the best score was Gemini 3.5 Flash at 65.93 on average, and only two models scored above 60.
  • On L2 tasks requiring tool use and code execution, every model scored lower than on L1.
  • Error analysis identified multimodal context inflation and task-state drift as recurring problems.
  • OmniFocus, which adds working-memory management and explicit task-state tracking, improved both performance and efficiency.
  • On Gemini 3.1 Pro Preview, OmniFocus used 271.07k input tokens versus 615.81k for the setting without both mechanisms.

Paper links

External research summaries. These are not HDATF publications or measured product results.

Read original (opens in a new tab)