GPT-2 on GTX1050: The Ambitious Training of an AI Model

Training GPT-2 from Scratch on a GTX1050

The initiative to train a GPT-2 model on a GTX1050 began out of pure curiosity, without a specific plan. The initial goal was to understand the process of training language models and to explore the possibility of doing so on local hardware. This project evolved over time, shifting from the idea of creating a coding assistant to a model focused on Turkish taxation.

The Beginning of the Experiment

At the start of this adventure, the author only had a GTX1050, a GPU with limited VRAM, and a local Linux setup. Although the hardware was not ideal, the aim was to explore the possibilities of experimentation under constrained hardware conditions. Extended training sessions raised practical concerns, such as heat management, system stability, and the risks of power interruptions.

A Different Initial Goal

The initial idea was not to work on Turkish language models, but to develop a coding model locally. The author began by downloading large open-source code datasets, some reaching several tens of gigabytes, and attempted to launch training experiments on their local machine. However, these attempts failed almost immediately due to the excessive size of the datasets and memory instability, highlighting the need to build a workflow suited to hardware constraints.

Change of Direction

Faced with the failure of the coding experiments, the author decided to change direction and focus on an assistant centered around Turkish taxation. Working in the field of accounting and taxes, they already had structured knowledge and sources of legal texts. An initial dataset was prepared from Turkish tax law documents, and a small fine-tuning experiment was attempted on GPT-2-like models, but the results were disappointing.

Model Issues

The base model did not understand Turkish well enough, which led the author to search for pre-trained models in Turkish on Hugging Face. Several alternatives were downloaded and tested, but the results remained unsatisfactory. Some models produced incorrect Turkish, others generated unstable outputs, and some failed to generalize usefully. This experience underscored that not all pre-trained models are necessarily well-trained or practically usable.

Discovering Training on Wikipedia

Once the main issue was identified as the lack of a good base model in Turkish, attention shifted to training a small GPT-like model capable of properly learning Turkish. With GPT-2 size architectures being the only realistic options for the available hardware, the author explored small-scale pre-training in Turkish, which led them to Wikipedia datasets.

Tokenizer Issues

While Wikipedia was more manageable than the enormous code datasets, each step introduced new challenges. Training the tokenizers became a major bottleneck, with issues related to vocabulary size, preprocessing stability, and memory constraints. Many failures occurred even before the model training began.

Evolving the Workflow

A major turning point was reached when the author began splitting the datasets into smaller parts. Initially, they experimented with about six data chunks, hoping that sequential training would reduce the pressure on runtime and memory instability. This approach worked better than expected, allowing for progressive training across multiple parts of the dataset and evolving into a chain training workflow.

The Birth of the Training Loop

As the number of dataset parts increased, manually managing training sessions became impractical. At first, sessions were launched individually and monitored manually, but this method ceased to work as the workflow expanded to dozens, then over a hundred dataset parts. A system capable of continuing training automatically while allowing for inspection of intermediate results became necessary, leading to the first version of the loop-based training system.

Experimentations on Dataset Order

Once the chain workflow was stabilized, the author focused on the structure of the datasets. An interesting question was whether the order of the datasets affected the quality of learning. Three main categories of datasets were used: dialogue-style text, structured dictionary-style language, and Wikipedia articles. The initial intuition, influenced by human learning patterns, was that the model might learn more naturally if the datasets followed a progression similar to language acquisition. This phase of experimentation was crucial in the evolution of the project.