This is a Plain English Papers summary of a research paper called LiveVQA: AI Challenges Answering Visual Questions with Real-Time Knowledge. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

LiveVQA: Building a Benchmark for Live Visual Knowledge

Introduction

LiveVQA introduces a new benchmark dataset that tests AI systems on their ability to answer questions requiring the latest visual knowledge from the internet. Unlike text-based knowledge problems which have seen significant advances through search-integrated language models, visual knowledge questions remain challenging for current AI systems.

The researchers created LiveVQA with three key design principles: strict temporal filtering to prevent dataset contamination, automated ground truth with human verification, and high-quality authentic image-question pairs. The dataset contains 1,233 news articles with 3,602 visual questions across 14 domains, sourced from six major global news platforms.

Each instance in LiveVQA contains a representative image paired with three types of questions: a basic visual understanding question (Level 1) and two multi-hop questions requiring deeper reasoning and contextual knowledge (Levels 2-3).

LiveVQA comprises 14 different News categories, containing 1233 News and 3602 question-answer pairs.

The LiveVQA Dataset

The dataset provides comprehensive coverage across 14 news categories and 6 major global news sources. It features high-quality image-text coherence and authentic information, with each instance carefully designed to test different levels of visual reasoning ability.

Pipeline of LiveVQA data engine showing the three main modules: news collector, data filter, and Q&A pairs builder.

The distribution of news instances shows meaningful patterns across categories and sources:

Table 1. The distribution of 1,232 news instances across 14 categories and 6 major sources, containing 3,602 VQA.

Dataset Construction

The LiveVQA dataset construction involved three main steps:

1. Data Collection: The researchers selected six global news platforms (CNN, BBC, Yahoo, Forbes, AP News, and Variety) for comprehensive geographic coverage and content diversity. The collection process included URL normalization to filter out advertisements and indexes, structured content extraction using site-specific CSS selectors, and image filtering to prioritize content-relevant images.

2. Raw Data Filtering: To ensure dataset quality, a multi-level filtering mechanism was implemented, covering URL validation, image screening, and duplicate removal. This involved website-specific filters, quality prioritization for images, and hierarchical deduplication techniques.

3. Question-Answer Generation: GPT-4o was employed to generate QA pairs from raw news documents. Each sample comprises an image reflecting the news topic, a basic question about image content, and two multi-hop detailed questions requiring cross-modality reasoning. The basic questions focus on substantive elements like people, objects, or locations, while avoiding queries solely reliant on visual properties like color. The multi-hop questions require deeper contextual reasoning and must be answerable only through the news text.

This careful construction process ensures high-quality visual question answering data that can properly test models' capabilities with latest visual knowledge.

Data Statistics

The final dataset contains 1,232 news items spanning 14 categories and 6 global news platforms, amounting to 3,602 QA pairs. The dataset shows distinct domain specificity - sports news is the most prevalent category, with a significant portion sourced from BBC. News sources also exhibit clear domain preferences: Variety primarily covers film and music, Forbes focuses on business, and AP News emphasizes science and health.

This diverse coverage makes the dataset ideal for comprehensive evaluation of visual question answering capabilities across different domains.

Experiments and Analysis

Experiment Setups

The researchers conducted zero-shot testing on 15 state-of-the-art multimodal large language models (MLLMs), including Gemini 2.0 Flash, Qwen2.5-VL series (3B to 72B), Gemma-3 series (4B to 27B), QVQ-72B-Preview, QVQ-Max, GPT-4o-mini, and GPT-4o.

The evaluation also tested models with search engine capabilities, enabling built-in search functionalities and MM-Search with Gemini-2.0-Flash, GPT-4o-mini, and GPT-4o. For evaluation, GPT-4o-mini was instructed as an impartial judge to provide binary (yes/no) correctness assessments.

Experiment Results

The evaluation revealed several important findings:

Table 2. Overall performance on LiveVQA. See Table 5 for performance on another categorizing taxonomy for live visual knowledge.

Key findings include:

1. Larger models perform better: Within model families like Gemma or Qwen, increasing model size leads to consistently better accuracy across all question difficulty levels. For example, Gemma-3-4b-it achieves only 2.46% on L3-level questions, while Gemma-3-27b-it reaches 4.92%.

2. Strong visual reasoning helps with complex questions: Models with stronger reasoning capabilities, such as QvQ-72B-Preview, outperform their base models on multi-hop questions, highlighting the importance of visual reasoning abilities in live knowledge tasks.

Table 3. Performance comparison between Qwen2.5-VL-72B [35] and QvQ-72B-Preview [24] with different thinking pattern via prompt engineering.

1. Search engines significantly improve performance: Integrating MM-Search with Gemini-2.0-Flash boosts its average accuracy to 29.00%, with substantial gains on harder questions (22.75% on L2 and 13.66% on L3), demonstrating that retrieval-based evidence is particularly helpful for questions beyond models' internal knowledge.

2. Entity-centric vs. abstract knowledge: Models perform better on concrete entity recognition tasks (Person, Organization, Object) but struggle with abstract knowledge like Time and Reason, indicating limitations in causal reasoning and temporal understanding.

3. Domain-specific performance patterns: Models achieve higher performance in domains with rich visual and textual cues but perform poorly in ambiguous or subjective domains like Opinion.

The research also explored whether providing image captions could improve performance on more complex questions:

Table 4. Performance of GPT-4o-mini on LiveVQA using image and image description as input. We find that model perform better in L2 and L3 problems when provided image caption.

This demonstrates that for more complex reasoning tasks (L2 and L3), having the image caption helps models perform better, despite lower performance on basic visual understanding questions (L1).

These findings highlight the importance of multimodal inference capabilities in solving visual question answering tasks that require up-to-date knowledge.

Error Analysis

To understand the limitations of current models, the researchers conducted a comprehensive analysis of failure cases, categorizing errors into several types:

1. Recognition Error: Failure to correctly identify visual elements like people, objects, locations, or text
2. Reasoning Error: Inability to perform deeper reasoning, including inference, causality, or temporal understanding
3. Ambiguous Answer: Responses too generic or indirect to address the question accurately
4. Privacy Restriction: Refusal to answer due to privacy or safety constraints, despite the question being valid
5. Judge Error: Correct answers mistakenly marked as incorrect during evaluation
6. Others: Errors not covered by the above categories, including system failures

These error categories provide valuable insights into the specific challenges that models face when dealing with visual question answering tasks requiring latest knowledge.

Related Works

The LiveVQA research builds on three key areas of previous work:

Live Knowledge Seeking: Recent advances in LLMs as search engines have shown significant progress, especially when integrated with retrieval-augmented generation (RAG) techniques. While tools like Search GPT have effectively combined RAG with online search for textual queries, multimodal search engines are still evolving to provide richer experiences integrating text, images, and other information types.

Live Visual Knowledge: Visual knowledge encompasses both factual information from external sources and commonsense understanding about objects and their interactions. Recent developments are expanding from static data to "live" visual content, addressing challenges in understanding current news, emerging events, and temporally relevant information.

News QA: News Question Answering has evolved from purely text-based approaches to NewsVQA, which integrates textual and visual information. This field faces unique challenges in multimodal fusion, temporal information processing, and scene text understanding.

Conclusion

LiveVQA provides a comprehensive benchmark for evaluating multimodal large language models on live visual knowledge tasks. The extensive experiments across 15 models reveal that while larger models generally perform better, significant challenges remain in addressing complex multi-hop visual questions that require up-to-date knowledge.

The research demonstrates that equipping models with online search tools or GUI-based image search leads to substantial performance improvements, particularly for challenging questions requiring contextual understanding and reasoning.

Despite these advances, even the best models achieve only modest accuracy (29.46% for Gemini-2.0-Flash with search), highlighting that live visual knowledge seeking remains an open challenge requiring further research.

Limitations

The researchers acknowledge several limitations of their study:

1. The dataset primarily structures latest information into visual question answering formats, while additional synthetic data approaches like image captioning or Chain-of-Thought reasoning could further enhance models' capabilities.

2. The research predominantly sources content from mainstream news websites like CNN and BBC, potentially leading to imbalance and incomplete representation of current internet content. Incorporating data from social media platforms could provide a more diverse dataset.

3. More robust visual search tools need to be developed to enhance model performance on latest visual knowledge queries.

These limitations point to promising directions for future research in this rapidly evolving field.

Click here to read the full summary of this paper