FineBooks and EleutherAI: Revolutionizing OCR for AI with dots.mocr

FineBooks and EleutherAI: Revolutionizing OCR for AI with dots.mocr

The FineBooks project, a collaboration between Hugging Face and EleutherAI, evaluated 14 open-source OCR models on over 2,000 pages of historical books to determine their ability to convert scanned texts into clean training data for AI language models.

Smaller models often outperformed larger ones, with the best-performing model achieving over 97% character accuracy at a cost of less than two dollars per thousand pages.

While researchers believe the output quality is sufficient for AI training needs, they note that the models remain too error-prone for use in academic or scientific applications.

OCR Challenges

Training open-source AI language models on public domain books involves dealing with poor-quality texts. Libraries extracted these texts from scans years ago using optical character recognition, and the results are often riddled with errors. The Talkie project quantified the potential damage: a language model trained on OCR text learns with only 30% of the efficiency of a model trained on human transcriptions of the same books.

Evaluation of OCR Models

The FineBooks project tested whether current open-source OCR models can solve this problem. The team ran 14 open-weight models on 2,165 pages of historical books and published the results in a ranking. The best models achieved character accuracy above 97% for less than two dollars per thousand pages.

Reference Pages

Three of the 2,165 reference pages include:

• A single-column English natural history text,
• A multilingual table of contents,
• An illustration plate where the entire transcription is a single artist credit line.

Improving Text Recognition

Millions of public domain pages require better text recognition. When EleutherAI and its partners released Common Pile last year, the largest openly licensed training corpus to date, it contained around 300,000 public domain books with texts derived from previous OCR runs. The authors of FineBooks assert that reprocessing these books with better models is one of the most effective ways to enhance training datasets for AI.

The project chose the Biodiversity Heritage Library (BHL) as its first target, which holds over 300,000 digitized documents in natural history totaling more than 64 million pages. The BHL offers its collection for bulk download via AWS.

Measuring OCR Quality

Measuring OCR quality requires pages with known correct transcriptions. The team utilized work from the IMPACT project and BHL-Europe: between 2011 and 2012, experts transcribed six volumes of the BHL in English, French, German, and Latin with an error rate of about one character for every 2,000. This data is available under a CC-BY license in a GitHub repository and forms the basis of the new reference dataset.

Model Performance

All 14 models are available for free and run on local hardware without an API key. The metric used is the character error rate (CER), which represents the proportion of misrecognized characters. The ranking divides the results into a "diplomatic" variant that counts the modernization of archaic characters as an error and a "reading" variant that tolerates such changes.

The dots.mocr model uses only 3 billion parameters, while Qwen3.5-9B scores lower despite being nearly three times larger. OvisOCR2 ranks second with only 0.9 billion parameters at 46 cents per thousand pages. The model size and OCR quality are not correlated for historical documents.

Cost per 1,000 Pages

• PaddleOCR-VL-1.6

The evaluation only covers Antiqua fonts in four languages. It does not account for Fraktur, non-Latin scripts, or handwriting, and FineBooks is limited to single-column book pages. The team plans to reprocess about 200,000 public domain BHL documents with one of the best models and publish the text as open datasets. New models are continuously added to the ranking, and the evaluation framework is openly available.

Use for AI and Academic Limitations

The authors of FineBooks assess the results based on the intended use. For training language models, the best performers work well enough, they write. The models produce far fewer errors than older processing chains, and reprocessing a collection the size of the BHL is realistic at the measured costs.

Libraries face a different problem, the team notes. Their systems rely on ALTO XML, a format with word-level coordinates. The new models produce Markdown or plain text without word positions, preventing them from integrating into existing library infrastructure.

For academic transcriptions, accuracy remains insufficient, but not because the models misread characters. They silently modernize them, replacing the long "s" or ligatures with modern equivalents. A targeted adjustment could resolve this issue, according to the team.