Forms: A Detector and a LLM Achieve 97% Detection Rate

A developer has created PaperKit, an application that separates detection and understanding to fill out forms. On layouts never seen before, the addition of a detector upstream of a vision model raises the detection of empty boxes to 97%, compared to 62% for the model alone. The author details a real-world execution while pointing out marginal failures and evaluation uncertainties.

Recognized Limitations and Concrete Cases of Failure

The author reports several failures observed in practice: misaligned boxes, entire sections skipped, drift on unseen layouts, and ambiguities on two-parent forms. He specifies that the tuning set used is small, that only one reference model has been properly tested, and that a more thorough evaluation remains necessary. A real-world execution on a child custody form in Florida is described, with indications of the associated times and costs.

Chosen Approach: Separating Localization and Labeling

Filling out a form requires identifying the nature of each field and precisely locating its position. For this, PaperKit employs an affordable detector that generates a large number of potential boxes, even proposing more to avoid missing any. A vision model paired with an LLM then reviews each proposal and assigns a field type and a property to each, while the phone handles the filling. The application was developed using Claude Code, and an AI assistant managed the tests on the author's mobile device while also contributing to the draft writing.

Comparisons on Set-Aside Pages and Numerical Results

An evaluation was conducted on modified pages set aside between a method relying solely on the model and the pipeline combining the detector and the model. The model-only method shows disappointing results for identifying checkboxes and the exact positioning of boxes, while the pipeline displays much better accuracy, achieving 97% detection of empty boxes when the detector intervenes before the model. On unknown forms, the vision model alone identifies 62% of empty boxes, compared to 97% with the upstream detector. The author created the application and notes that the figures and screenshots come from real executions. He reports that a school form took 30 minutes to fill out manually and that a generic AI approach missed 4 out of 10 boxes during a trial.