Timer-XL: A Revolution in Time Series Forecasting

Timer-XL: An Innovative Forecasting Model for Time Series

Introduction to Timer-XL

Timer-XL stands out as a next-generation Transformer model specifically designed for time series forecasting. This model, which relies solely on a decoder, focuses on the ability to generalize and make predictions over extended periods. It offers a unified approach for long-term forecasting, a feature that distinguishes it from existing models.

Key features of Timer-XL include:

• Flexible input and output lengths: Unlike models such as Tiny-Time-Mixers, which require distinct versions for different input or output lengths, Timer-XL operates with a single model for all configurations, without assuming the length of context or prediction.

• Long-term forecasting capability: It effectively manages extended feedback windows, which is crucial for long-term predictions.

• Advanced functionalities: Timer-XL can predict non-stationary univariate series, handle complex multivariate dynamics, and integrate contexts informed by covariates with exogenous variables, all within a unified setup.

• Versatility: The model can be trained from scratch or pretrained on large datasets, with optional fine-tuning to enhance performance.

Timer-XL improves forecasting accuracy through the introduction of TimeAttention, a sophisticated attention mechanism that we will detail later.

The team behind Timer-XL, from the THUML lab at Tsinghua University, possesses deep expertise in time series modeling. They have previously developed notable models such as iTransformer, TimesNet, and Timer, the direct predecessor of Timer-XL.

Model Comparison: Encoder, Decoder, and Encoder-Decoder

Before delving into Timer-XL in detail, it is helpful to understand the different types of foundational models used in time series forecasting. This understanding highlights the advancements that led to the development of Timer-XL.

Applications in Natural Language Processing (NLP)

At the dawn of the Transformer era, a debate emerged regarding the most effective architecture. The initial Transformer was an Encoder-Decoder model. Subsequently, research split into two branches: Encoder-only models, like Google's BERT, and decoder-only models, such as OpenAI's GPT.

• Encoder-Decoder Models: These models use a bidirectional encoder to analyze the input and a causal decoder to generate the output, one token at a time. They excel in sequence-to-sequence tasks, such as translation and summarization.

• Encoder-only Models: These models use bidirectional attention to understand a sentence and predict masked words. They are particularly effective in natural language understanding (NLU) tasks.

• Decoder-only Models: These models use causal attention to learn to predict the next word, thus excelling in natural language generation (NLG) tasks.

In the field of NLP, decoder-only models dominate generation tasks, while encoder-only models are preferred for classification, regression, and named entity recognition (NER).

Applications in Time Series

By the end of 2024 and the beginning of 2025, many foundational models were published, providing abundant evidence of what works best in time series forecasting.

These models come in several forms:

• Decoder Models: Such as TimesFM (Google) and Time-MOE.

• Encoder Models: Such as MOIRAI (Salesforce) and MOMENT.

• Encoder-Decoder Models: Such as Chronos (Amazon).

So far, decoder and encoder-decoder models outperform encoders in forecasting. The authors of Timer-XL have confirmed this trend through extensive experiments.

There is also a category of versatile models used for forecasting, classification, imputation, etc. MOMENT and UNITS belong to this category and are encoder-only models.

Timer, a versatile model, is a decoder-only model. Its successor, Timer-XL, surpasses Timer in forecasting but specializes solely in this task.

For tasks requiring a general understanding of time series, such as imputation or anomaly detection, encoder models may be more suitable. However, for time series forecasting, decoders currently have the advantage.

This is why the authors evolved from Timer's generalist design to Timer-XL's specialization in forecasting. Both models are decoders, but the decoder architecture is particularly beneficial for the forecasting task.

Long-Term Forecasting

One of the main advantages of Transformer models lies in their ability to handle long sequences of context. Modern large language models (LLMs), like Gemini, can process up to 1 million tokens. Although they are not perfect at this scale, they generally remain reliable up to 100,000 tokens.

In contrast, time series models are still lagging behind. Transformer-based and deep learning forecasting models often struggle beyond 1,000 tokens. Recent foundational models, such as MOIRAI, can handle up to 4,000 tokens.

Two key questions arise here:

• What is the maximum supported context length?

• How does the model handle the increase in context length in terms of performance?

Timer-XL stands out for its ability to better manage the increase in context compared to other models.

For daily datasets, such as traffic data, it is possible to use up to one year of data (approximately 8,760 data points). This makes Timer-XL particularly suitable for high-frequency forecasting, a setup where foundational models often perform poorly.

TimeAttention: The Innovative Mechanism of Timer-XL

The attention mechanism is at the heart of Transformers, a major advancement in NLP. However, in the context of time series, it can prove to be a double-edged sword.

Transformer models for time series are prone to overfitting. It is not possible to use raw attention as in NLP, as self-attention is permutation invariant (the order of tokens does not matter, which should not be the case when temporal information is involved).

Timer-XL introduces a causal variant of attention called TimeAttention.

TimeAttention incorporates:

• Rotary positional embeddings (ROPE) to capture temporal dependencies.

• Binary biases (ALIBI) to capture dependencies between variates.

• Causal self-attention.

The goal of TimeAttention is to ensure:

• No permutation invariance for temporal information — the order of data points or temporal tokens must be significant.

• Permutation invariance among variates or features — the order of variates should not be significant (for example, if we have two covariates X1 and X2, their order does not matter, only the relationship between them counts). This ensures equivalence under permutation.

The attention score between the query (m,i) and the key (n,j), where i,j represent temporal indices and (m,n) represent variate indices, is calculated in a specific manner.

TimeAttention enables Timer-XL to effectively manage temporal dependencies while preserving the integrity of relationships between variates.