Haonan Jiang, Guojian Zhan, Jiancong Xie, Shijun Wan, Dongjia Zhao, Cheng Chen, Yahui Liu, Chuan Mu
AllSpark Team
Vision-language models (VLMs) increasingly power consumer-facing AI search, yet evaluating them on the diversity of everyday visual questions remains challenging. Existing benchmarks typically target predefined capabilities—such as multi-hop retrieval or long-form synthesis—whereas real users pose photo-grounded questions spanning a long tail of everyday scenarios. Notably, despite rapid advances in VLMs, users of Xiaohongshu, a mainstream Chinese image-sharing platform, continue to seek assistance from other humans for everyday visual questions. Motivated by this observation, we curate NoteVQA, a benchmark of 252 items spanning 12 topical categories and 7 user intents. Each item is accompanied by a concise reference distilled from expert community responses, together with a human-audited interleaved reference answer that combines textual explanations with supporting visual evidence.
We evaluate both short-answer correctness and interleaved-answer quality. To support the latter, we introduce AgenticInterleave, a single-agent ReAct framework for retrieval-supported answer generation, along with IVR-12, a 12-dimensional rubric for assessing the content, presentation, and image quality of interleaved answers. Across nine frontier VLMs, the highest short-answer accuracy is 52.8%, while augmenting Qwen3.5-397B-A17B with agentic search improves accuracy by only 2.0%. For interleaved answers, the same model running AgenticInterleave scores 3.52 under IVR-12, compared with 4.65 for the human-audited references, with the largest gap observed in content quality. These findings underscore the challenges that everyday visual questions pose for current VLMs, both in answer accuracy and in the quality of visually grounded explanations.
Figure 1: Overview of NoteVQA.
Existing VLM benchmarks emphasize controlled, capability-oriented tasks. In contrast, NoteVQA is derived from naturally occurring information needs: real questions that users posted to a human community after existing AI tools failed to satisfy them. This design yields three distinctive properties:
- Ecological validity. Questions reflect authentic, photo-grounded information needs from everyday life rather than synthetic or templated constructions.
- Long-tail coverage. The benchmark spans 12 topical categories and 7 user intents, capturing the diversity of real user queries.
- Interleaved ground truth. Beyond short answers, each item provides a human-audited reference that interleaves textual explanation with supporting images, enabling evaluation of multimodal answer presentation, not merely correctness.
Figure 2: Data curation pipeline of NoteVQA.
The curation pipeline proceeds in three stages:
- Collection. Photo-grounded questions are sampled from community posts on Xiaohongshu, filtered to retain items that require visual understanding and admit verifiable answers.
- Reference construction. For each question, a concise reference answer is distilled from high-quality expert community responses.
- Interleaved annotation and audit. Annotators compose an interleaved reference answer—textual explanation combined with retrieved supporting images—which subsequently undergoes human audit to ensure factual correctness and visual relevance.
Figure 3: Dataset composition and representative examples from NoteVQA.
The resulting benchmark contains 252 items covering 12 topical categories and 7 user intents.
AgenticInterleave is a single-agent ReAct-style framework for retrieval-supported interleaved answer generation. Given a question and its associated image, the agent iteratively (i) reasons about the information required, (ii) issues web and image search queries, and (iii) composes a final answer that interleaves textual explanation with retrieved visual evidence.
IVR-12 is a 12-dimensional rubric for assessing interleaved answers along three axes:
- Content quality — factual correctness, completeness, and relevance of the textual explanation;
- Presentation quality — organization, readability, and coherence of the interleaved structure;
- Image quality — relevance and informativeness of the supporting visual evidence.
The rubric is applied uniformly to model-generated answers and to the human-audited references, enabling a direct human–model comparison.
We evaluate nine frontier VLMs under both short-answer and interleaved-answer settings. Key findings:
- Short-answer accuracy. The best-performing model attains 52.8% accuracy, indicating that everyday visual questions remain substantially unsolved.
- Limited benefit from agentic search. Augmenting Qwen3.5-397B-A17B with agentic retrieval yields only a 2.0% absolute improvement, suggesting that retrieval alone does not resolve the core difficulty.
- Interleaved-answer quality. Under IVR-12, Qwen3.5-397B-A17B with AgenticInterleave scores 3.52, versus 4.65 for human-audited references. The largest gap lies in content quality, highlighting that generating faithful, visually grounded explanations remains an open challenge.
export SERPER_API_KEY=... # https://serper.dev (web & image search)
export QA_BASE_URL=http://localhost:8000/v1
export QA_MODEL=your-model
export EVAL_DATA_ROOT=/path/to/notevqa_eval_data
cd agentic_interleave
bash run_agentic_interleave.shcd ../ivr12
python ivr12_score.py --source gold # score human-audited references
python ivr12_score.py --help # full argument listIf you find NoteVQA useful in your research, please cite:
@article{jiang2026notevqa,
title = {NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities},
author = {Jiang, Haonan and Zhan, Guojian and Xie, Jiancong and Wan, Shijun and Zhao, Dongjia and Chen, Cheng and Liu, Yahui and Hu, Yao and Mu, Chuan},
journal = {arXiv preprint arXiv:2609.15695},
year = {2026}
}