Single Transformer Beats Modular Vision-Language Models in New Study

This is a Plain English Papers summary of a research paper called Single Transformer Beats Modular Vision-Language Models in New Study. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

The Simplicity Revolution in Vision-Language Models

Multimodal Large Language Models (MLLMs) have seen remarkable progress, but most follow a modular design: a pre-trained vision encoder extracts image features, a language model processes text, and a lightweight projector aligns these modalities. While effective, this approach fragments multimodal processing and limits deployment flexibility.

A promising alternative has emerged - eliminating the visual encoder entirely and processing raw image patches and text tokens within a single Transformer. This approach, exemplified by SAIL (Single trAnsformer model for vIsion and Language), removes modality-specific modules, enabling parameter sharing and end-to-end learning of vision-language interactions.

Data scaling curve showing SAIL's superior performance gains with increased training data (left) and comparison with other Single Transformer MLLMs showing SAIL's strong performance on both vision and vision-language tasks (right).

SAIL's analysis uncovers striking advantages of Single Transformer architectures:

1. Superior Data Scaling: SAIL exhibits steeper performance gains as pretraining data scales. While modular MLLMs initially perform better, the gap narrows substantially with more data.

2. Vision-Centric Information Flow: Analysis of attention distributions shows Single Transformers assign significantly higher attention scores to image tokens during prediction, indicating a more direct visual information pathway.

3. Strong Visual Representation: The pretrained Single Transformer inherently serves as a powerful vision encoder, performing impressively on vision-centric tasks like classification and segmentation.

Current Landscape of Vision-Language Models

Paradigms in Vision-Language Model Design

The two primary approaches to MLLMs have distinct advantages and limitations:

Modular MLLMs with Visual Encoders rely on pretrained vision encoders like CLIP-ViT or InternViT to process visual inputs. While this enables effective transfer of visual knowledge, it introduces several drawbacks:

• Separate vision encoders slow down training and inference
• Integration of visual features into LLMs remains challenging
• Balancing interactions between components becomes increasingly complex at scale

Single Transformer-based MLLMs Without Visual Encoders process raw image patches and text tokens through a unified architecture. These models fall into two categories:

• Continuous tokenization (Fuyu-8B, SOLO): Directly maps patches to LLM embeddings via linear projections
• Discrete tokenization (Chameleon, Emu3): Uses VQ-VAE tokenizers to compress images into discrete tokens

Despite innovations from models like EVE and Mono-InternVL, critical gaps remain in understanding the scaling laws and fundamental properties of purely end-to-end trained models.

Vision Representation Learning

Training effective visual representations with language data typically follows three paradigms:

1. Text as Classification Labels: Uses textual descriptions to extract categorical supervision (Tag2Text, RAM, CatLIP)

2. Image-Text Contrastive Learning: Aligns global image-text embeddings within a shared latent space (CLIP, ALIGN)

3. Text as Autoregressive Targets: Treats caption generation as a pretext task for visual representation learning (SimVLM, CapPa)

SAIL aligns with the third category but removes architectural fragmentation by jointly modeling image patches and text tokens in a single Transformer. The approach shows that the resulting model learns transferable vision representations that work well on both multimodal tasks and pure vision tasks.

SAIL: A Single Transformer for Vision and Language

Model Architecture

SAIL uses a unified Transformer architecture that processes both images and text without separate encoders. For text, raw input is tokenized and embedded via the language model's embedding module. For images, input is partitioned into patches and projected into embeddings via linear projection. Special tokens ( and ) mark the boundaries of image patch spans.

SAIL's unified architecture processes both images and text without extra modules (A), uses a mixed attention mechanism (B) allowing bidirectional attention for image patches, and implements multimodal rotary position embeddings (C) for improved spatial modeling.

Two key innovations distinguish SAIL:

1. Bidirectional attention within image patches: Unlike previous Single Transformer approaches that use only causal attention, SAIL implements a mixed attention scheme:

   • Causal attention for text tokens (preserving autoregressive generation)
   • Full bidirectional attention for image tokens (enabling holistic spatial relationships)

2. Multimodal Rotary Position Embeddings (M-RoPE): Decomposes positional encoding into height and width axes, with text tokens sharing uniform position IDs and image tokens adaptively mapping to coordinates.

Pretraining Strategy

SAIL applies a two-stage curriculum:

1. Accelerated Visual Knowledge Acquisition: Pretraining on large-scale image-text pairs with lower resolution (224×224) to maximize data throughput. Pure text corpora are interleaved to prevent losing language capabilities.

2. Enhancing Any-Resolution Image Understanding: Extended pretraining with an any-resolution strategy where images retain their native resolutions and positional embeddings adapt dynamically.

Details of training datasets used across all stages, showing data sources, image resolution settings, and sample counts.

The pretraining objective uses standard language modeling loss only on text tokens, excluding image patches and special visual tokens from loss computation.

Supervised Fine-tuning

During Supervised Fine-tuning (SFT), SAIL trains on publicly available instruction datasets to enhance understanding of complex linguistic instructions and diverse dialogue patterns. This phase fine-tunes the entire network, focusing on aligning the model's responses with human intent through exposure to varied instructional formats and multimodal interactions.

Experimental Results: Simplicity Meets Performance

Experimental Settings

SAIL evaluates across a broad range of benchmarks:

• Vision-language: MMBench-EN, SEEDBench-IMG, MMVet, MME, HallusionBench, MathVista, OCRBench
• Visual question answering: TextVQA, ScienceQA-IMG, AI2D, MMStar, RealWorldQA
• Vision representation: ImageNet-1K, ADE20K, ARO

For pretraining, SAIL initializes from Mistral-7B-v0.1 with patch size 14, using 128 NVIDIA A100 GPUs. The supervised fine-tuning process uses a global batch size of 512 for one epoch.

Vision-Language Task Performance

SAIL consistently outperforms other Single Transformer-based models across diverse vision-language tasks:

Comparison with existing vision-language models on various vision-language benchmarks, including MMStar, MMBench-EN, SEEDBench-Img, MMVet, MME, POPE, HallusionBench, ScienceQA-Img, TextVQA, MathVistaMINI, AI2D, RealWorldQA, and OCRBench. Note that #A-Param denotes the number of activated parameters; #Data represents the pre-training / fine-tuning data volume; #Vtoken indicates the maximum image patch tokens.

Compared to the state-of-the-art modular MLLM LLaVA-OneVision, SAIL achieves comparable performance on several benchmarks including MMStar, SEEDBench-IMG, and RealWorldQA. This demonstrates that scaling up single-transformer pretraining effectively enhances cross-modal alignment between images and text.

Vision Representation Learning Performance

SAIL demonstrates exceptional performance on vision tasks compared to other single transformer models:

Comparison on image classification and semantic segmentation with other encoder-free approaches. Our SAIL outperforms other alternatives by a large margin.

Even when compared with specialized vision backbones, SAIL achieves competitive performance with significantly less training data:

Comparison on image classification and semantic segmentation with other vision backbones. Despite using significantly less training data, SAIL achieves competitive performance.

For additional vision-related tasks, Pixel-SAIL explores SAIL's downstream capabilities in pixel-grounded understanding.

Properties of Single Transformer Models

Model Scaling: As model size increases from 0.5B to 7B parameters, performance improves consistently across vision-language tasks:

As SAIL's model size increases from 0.5B to 7B parameters, training loss decreases (left) and performance on downstream vision-language tasks improves (right).

Data Scaling: Single Transformer models show superior data scaling properties compared to modular MLLMs. While modular MLLMs perform better with limited data, SAIL exhibits a steeper performance curve as data scales, eventually achieving comparable performance with 512M image-text pairs.

Information Flow Pattern: Single Transformer models allocate significantly more attention to image tokens during prediction:

Single transformer models like SAIL allocate significantly higher attention to image tokens during prediction compared to modular MLLMs, indicating more direct visual information flow.

Single Transformer-based MLLMs allocate between 60% and 80% of attention scores to image tokens across all layers when predicting tokens, compared to just 10% to 30% in modular MLLMs. This indicates a more vision-centric approach to multimodal reasoning.

Empirical Observations on Basic Factors

Ablation studies identify key design choices for effective single-transformer MLLMs:

Ablation Study on Basic Factors for Sail: This table presents the impact of different ablation settings on the performance of Sail across various benchmarks. The default setting includes image full attention and the inclusion of pure text data in pretraining.

1. Bidirectional Attention for Image Patches with Multimodal Position Encoding: Enabling full bidirectional attention for image patches paired with multimodal rotary position embeddings significantly improves performance, with a notable gain of 3.1% on TextVQA.

2. Interleaving Pure Text Data During Pretraining: Incorporating text data consistently improves performance across benchmarks, underscoring the importance of preserving language capabilities when training Single Transformer models.

The Future of Single Transformer MLLMs

The extensive analysis of SAIL demonstrates that Single Transformer-based MLLMs offer promising advantages over modular designs. By scaling both training data and model size, this unified architecture achieves performance comparable to modular MLLMs while functioning effectively as a vision backbone.

Key advantages include superior data scalability, vision-centric information flow, and strong visual representation capabilities. Even with less pretraining data than modular approaches, the single-transformer design shows competitive performance on both vision-language and pure vision tasks.

These findings suggest a promising future for simpler, more integrated multimodal architectures. As research continues to refine and enhance Single Transformer designs, we may see these streamlined models become the standard approach for multimodal intelligence, offering better scaling properties and more direct cross-modal interactions than their modular counterparts.

The research on SAIL opens new avenues for exploring how unified architectures can effectively learn from multimodal data without relying on pretrained components, potentially leading to more flexible and powerful vision-language systems.

Click here to read the full summary of this paper