Count Anything: the AI that challenges the limits of visual counting

Count Anything: A Revolutionary AI for Object Counting

A new artificial intelligence model, "Count Anything," has been developed by researchers from Tsinghua University and other institutions. This model stands out for its ability to count and label objects in a wide variety of images, ranging from satellite photos to medical scans, using just a text prompt.

The system is based on Meta's SAM3 model and combines two distinct approaches: it draws boxes around large objects and places points on small, dense targets. This method allows for the merging of results without counting the same object twice. "Count Anything" adds small adapter components for the counting task instead of retraining the entire model from scratch.

A Breakthrough in the Multimodal Field

Current large language models can describe images, interpret graphs, and extract text from photos. However, accurately counting objects remains a challenge. "Count Anything" aims to bridge this gap by providing a universal solution for various types of images, whether it involves counting heads in crowds, cars in satellite photos, cells in medical scans, or bacterial colonies in the lab.

Obtaining accurate counts has real-world implications, whether for a doctor reading a scan, a farmer estimating crop yields, or an urban planner analyzing traffic. Until now, each of these tasks required its own specialized system.

Methodology and CLOC Dataset

The key innovation lies in the combination of two complementary approaches: one for visible objects and another for small, dense ones. The results are then merged, retaining only the most reliable prediction to avoid double counting.

To train this model, the researchers created the CLOC dataset. Existing public datasets were generally designed for a single objective, such as tumor cells or satellite images. The researchers merged them, cleaned up conflicting labels, and published the result under the name CLOC. This dataset includes approximately 220,000 images, 619 categories, and 15 million labeled objects, covering six varied image domains, from everyday photos to satellite and drone images, as well as medical tissue samples and microscopic cell images.

Error metrics decrease significantly as the CLOC training data increases, demonstrating the value of large inter-domain counting datasets.

Performance and Challenges

In comparative testing, "Count Anything" ranked well above competing systems like CountGD, CLIP-Count, and Grounding DINO. On average, the model miscounts about nine objects per queried category in an image, while the best competing model is over twice that margin. For pure crowd counting, "Count Anything" remains competitive but does not quite match the best specialized systems.

In direct comparisons, "Count Anything" consistently approaches the actual number of objects compared to CLIP-Count, CountGD++, and SAM3 across all tested queries.

However, the researchers acknowledge certain limitations. When terms are ambiguous or highly specialized, the model may miss objects or classify them incorrectly. In extremely dense scenes with significant occlusion, it also becomes challenging to determine whether two predictions refer to the same object or two different objects.

The code for "Count Anything" is available on GitHub.

A Reflection on the Limits of AI Systems

The extent to which current AI systems still struggle with basic visual tasks was recently demonstrated by the BabyVision benchmark. In tests with 80 children, most state-of-the-art models scored lower than an average three-year-old. Even the best models, like Gemini 3 Pro, only reached 50%, while adults scored over 94%. The gap was particularly pronounced when counting occluded 3D blocks, where the best model succeeded only 20.5%. Humans completed this task without any errors.