Granite 4.1: Revolutionizing Language Models with 512K Tokens

Granite 4.1: A New Era for Language Models

Granite 4.1 represents a significant advancement in the field of large language models (LLMs). This family of models includes versions with 3 billion, 8 billion, and 30 billion parameters, utilizing a dense and decoder-only architecture. These models have been trained on approximately 15 trillion tokens through a multi-stage pre-training pipeline, incorporating a long context extension of up to 512,000 tokens.

Model Architecture

The Granite 4.1 models are based on a dense transformer architecture with innovations such as Grouped Query Attention (GQA), Rotary Position Embeddings (RoPE), SwiGLU activations, RMSNorm, and Shared input/output embeddings. All models share a common training pipeline, differing only in their architectural dimensions.

Granite 4.1 is trained from scratch on approximately 15 trillion tokens using a five-phase training strategy. Phases 1 and 2 focus on foundational pre-training, phases 3 and 4 conduct intermediate training with a mix of progressively higher-quality data, and phase 5 introduces long context training, extending the context window to 512K tokens. Each phase uses a distinct data mix and a learning rate schedule, gradually transitioning from large-scale web data to more domain-specific content.

Pre-training Phases

Phase 1: General Pre-training (10T tokens)

This phase establishes a broad linguistic understanding using a general mix of training data with a dynamic learning rate schedule.

Data Composition:

• CommonCrawl ~59% — general web data
• Code ~20% — programming languages and repositories
• Math ~7% — mathematical reasoning data
• Technical ~10.5% — scientific articles, technical documentation, and manuals
• Multilingual ~2% — data in non-English languages
• Domain-specific ~1.5% — domain-specific content

Phase 2: Math/Code Pre-training (2T tokens)

Phase 2 significantly increases the proportion of code and mathematical data, moving towards stronger reasoning capabilities while maintaining general linguistic coverage.

Data Composition:

• Math ~35% — a 5x increase from phase 1
• Code ~30% — a 1.5x increase
• CommonCrawl-HQ ~12% — high-quality subset of CommonCrawl
• Synthetic ~9% — high-quality synthetic data
• Multilingual ~3%

Phase 3: Fine-tuning High-Quality Data (2T tokens)

Phase 3 shifts to intermediate training with a more balanced and high-quality mix and an exponentially decaying learning rate schedule.

Data Composition:

• CommonCrawl-HQ ~16.67%
• Technical ~12.5%
• Multilingual ~4.5%
• Long Chain-of-Thought ~12.5% — reasoning trajectories
• Linguistic Instructions ~7.5% — instruction tuning data
• Code Instructions ~4.5% — instruction tuning data

Phase 4: Fine-tuning High-Quality Data (0.5T tokens)

The fourth phase continues intermediate training with a linear decay of the learning rate to zero, focusing the model on the highest quality data available.

Data Composition:

• CommonCrawl-HQ ~40%
• Long Chain-of-Thought ~6%
• Code Instructions ~5%
• Linguistic Instructions ~9%

Phase 5: Long Context Training (LCE)

The fifth and final phase, also part of intermediate training, extends the context window from 4K to 512K through a stepwise long context extension process:

• 32K Extension — using the same data mix as phase 4
• 128K Extension — same data mix as phase 4
• 512K Extension — 80% books + 20% code repository data (8B and 30B only)

The LCE phase uses an exponential learning rate schedule starting at 1e-4 and decaying to 0. To ensure that the model natively handles long sequences without degrading short context performance, we perform model merging after each LCE step.

SFT: Data Preparation and Quality Control

Supervised Fine-Tuning (SFT) transforms the base model into a reliable instruction-following assistant, making data quality crucial. Even a small number of incorrect or hallucinated samples can lead to undesirable behaviors. To address this, we apply a rigorous LLM-as-Judge framework combined with rule-based filtering to select high-quality samples. Together, the pipeline automatically evaluates each sample against structural, semantic, and behavioral criteria, correcting issues where possible and filtering out samples that do not meet our quality standards.

SFT Training Details

After passing through the LLM-as-Judge framework, rule-based filtering, and the overall deduplication pipeline, we fine-tune the base models on these approximately 4.1 million high-quality samples. The following details apply to all model variants:

• Training Configuration:
  • 16 nodes, 4x GB200 per node
  • 5e-6 (linear warm-up 3%, linear decay over ~25K steps)
  • Effective batch size: 256 samples/iteration (~4.2M tokens/iteration)

Reinforcement Learning: Multi-Step RL Pipeline

After SFT, we apply a multi-step reinforcement learning pipeline to further enhance the model's capabilities in specific areas. Rather than a single RL pass, we execute multiple targeted RL steps, each optimizing different capabilities.

Training Methodology

We use GRPO (Group Relative Policy Optimization) with DAPO loss, which provides more stable training signals compared to standard GRPO. However, due to the computationally intensive nature of dynamic sampling, we disable it during our training sessions.

RL Training Configuration

• GRPO in on-policy mode with DAPO loss

The reinforcement learning pipeline for training the Granite 4.1 models consists of four sequential steps: Multi-domain RL, RLHF, Identity and Knowledge Calibration, and Math RL. In this step, the model is trained jointly on a unified mix of data drawn from multiple domains. Each gradient update thus reflects the full diversity of tasks, preventing catastrophic forgetting, improving overall performance on benchmarks, and minimizing regressions on any individual task.

Performance and Licensing

Notably, the 8B instructive model of Granite 4.1 matches or exceeds the performance of the previous Granite 4.0-H-Small model (32B-A9B MoE), despite a simpler dense architecture and a reduced number of parameters. All Granite 4.1 models are released under the Apache 2.0 license, facilitating their use and adaptation by the community.