InternVL3: Unified Multimodal AI Training Outperforms Open-Source Rivals

This is a Plain English Papers summary of a research paper called InternVL3: Unified Multimodal AI Training Outperforms Open-Source Rivals. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

Breaking New Ground in Multimodal AI Training

InternVL3 marks a significant advancement in the InternVL model series, implementing a native multimodal pre-training approach that fundamentally transforms how vision-language models learn. Unlike most leading multimodal large language models (MLLMs) that adapt text-only models to handle visual inputs through complex post-hoc alignment, InternVL3 jointly acquires multimodal and linguistic capabilities in a single unified pre-training stage.

This innovative approach addresses the alignment challenges typically encountered when retrofitting text-only LLMs with visual processing capabilities. Traditional methods require resource-intensive strategies with specialized domain data and intricate parameter-freezing schedules, whereas InternVL3's unified training paradigm offers a more efficient and integrated approach to multimodal learning.

Native Multimodal Pre-Training: A Unified Approach

The core innovation of InternVL3 lies in its native multimodal pre-training strategy. Rather than first training a text-only LLM and then adapting it for visual inputs, InternVL3 learns from both text corpora and multimodal datasets simultaneously during pre-training. This approach enables the model to develop linguistic and visual capabilities in tandem, creating a more seamlessly integrated multimodal architecture.

The pre-training process uses a multimodal autoregressive formulation where visual tokens serve as conditioning context for text prediction. While gradients naturally propagate through all modalities, the loss computation focuses exclusively on text tokens. This training approach helps the model embed visual information in ways that benefit downstream language tasks.

InternVL3 also incorporates advanced position encoding strategies like Variable Visual Position Encoding (V2PE), which assigns different position increments to textual and visual tokens. This innovation helps accommodate longer multimodal contexts without excessively extending the position window.

Architecture and Model Variants

InternVL3 follows the "ViT-MLP-LLM" architecture paradigm used in previous models but introduces several refinements. The model consists of a vision encoder (InternViT), a two-layer MLP for alignment, and a language model (primarily from the Qwen2.5 series).

Pre-trained models used in the InternVL3 series. The OpenCompass scores for the InternVL3 series were obtained through local testing.

The LLM components are initialized solely from pre-trained base models without employing instruction-tuned variants. For efficiency in processing high-resolution images, InternVL3 incorporates a pixel unshuffle operation, which reduces visual token count to one-fourth of the original value.

Advanced Training and Optimization Strategies

InternVL3 employs a comprehensive three-phase training approach:

1. Native Multimodal Pre-training: The model learns from interleaved multimodal data and text corpora during initial pre-training.

2. Supervised Fine-Tuning (SFT): Building on pre-training, SFT uses higher-quality and more diverse training data across domains like tool usage, 3D scene understanding, GUI operations, long context tasks, video understanding, scientific diagrams, creative writing, and multimodal reasoning.

3. Mixed Preference Optimization (MPO): This phase addresses the gap between training (ground-truth token conditioning) and inference (model-predicted token conditioning) by introducing supervision from both positive and negative samples.

For inference, InternVL3 leverages test-time scaling with Best-of-N evaluation using VisualPRM-8B as a critic model to select the most effective responses for reasoning and mathematical tasks.

To enhance training efficiency, the team extended the InternEVO framework to support flexible sharding strategies for different model components, resulting in a 50-200% training speedup compared to previous models.

Comprehensive Performance Evaluation

InternVL3 demonstrates exceptional performance across diverse multimodal tasks. On the multidisciplinary reasoning benchmark MMMU, InternVL3-78B achieves a score of 72.2%, setting a new state-of-the-art among open-source MLLMs.

For mathematical reasoning, InternVL3 shows strong results across various benchmarks, with InternVL3-78B recording 79.0% on MathVista, 43.1% on MathVision, and 51.0% on MathVerse.

Comparison of multimodal reasoning and mathematical performance. The overall score is the average score of the benchmarks. "w/ VisualPRM-Bo8" denotes that the model is evaluated with Best-of-8 settings, where VisualPRM serves as the critic model.

In OCR, chart, and document understanding tasks, InternVL3 achieves top-tier results, with InternVL3-78B reaching 89.7/96.0% on AI2D, 89.7% on ChartQA, and 95.4% on DocVQA.

Comparison of OCR, chart, and document understanding performance. The table evaluates OCR-related capabilities across 9 benchmarks, including AI2D, ChartQA, TextVQA, DocVQA, InfoVQA, OCRBench, SEED-2-Plus, CharXiv, and VCR.

For multi-image understanding, InternVL3-78B demonstrates strong cross-image reasoning capabilities with scores of 66.3% on BLINK, 79.3% on Mantis-Eval, and 73.2% on MMT-Bench.

The model also excels in video understanding, with InternVL3-78B achieving 72.7/75.7% on Video-MME and 78.7% on MVBench, outperforming many other open-source and closed-source models.

Preserving Strong Language Capabilities

A key achievement of InternVL3 is maintaining robust language capabilities despite its joint multimodal training approach. In fact, InternVL3 models consistently outperform their Qwen2.5 chat counterparts across most language evaluation benchmarks, including MMLU, CMMLU, C-Eval, and GAOKAO.

This enhancement in language capabilities stems from several factors: the integration of approximately 25% pure-language data during multimodal pre-training, joint parameter optimization, and extensive use of high-quality textual corpora during post-training.

Ablation Studies and Analysis

Ablation studies reveal the effectiveness of InternVL3's key components:

1. Native Multimodal Pre-training: Models using native multimodal pre-training showed performance comparable to fully multi-stage trained baselines, even without further optimization.

2. Variable Visual Position Encoding: Different position increment values (δ) were tested, showing that even for tasks with short contexts, relatively small δ values can achieve optimal performance.

3. Mixed Preference Optimization: Models fine-tuned with MPO demonstrated superior reasoning performance compared to counterparts without MPO, with improvements of 4.1 and 4.5 points for the 78B and 38B variants, respectively.

A Step Forward for Open-Source MLLMs

InternVL3 represents a significant advancement in the development of open-source multimodal models. By implementing a native multimodal pre-training approach, incorporating innovations like V2PE, and applying effective post-training techniques, InternVL3 achieves state-of-the-art performance among open-source MLLMs while narrowing the gap with leading proprietary models.

In line with open science principles, the researchers are releasing both the training data and model weights to foster further research and development in next-generation MLLMs. This commitment to transparency and accessibility helps advance the entire field of multimodal AI.

The success of InternVL3 demonstrates the effectiveness of unified training paradigms that jointly optimize for multiple modalities, potentially setting a new direction for the development of future multimodal AI systems.

Click here to read the full summary of this paper