General Compute and SambaNova: Revolutionizing AI Inference

A Growing Demand for AI Infrastructure

The rise of artificial intelligence technologies has led to a significant increase in demand for computers capable of handling complex AI models. However, two major challenges arise: acquiring the right chips and integrating them into data centers that can generate revenue. General Compute, an emerging company specializing in inference, seems to have found promising solutions to these issues. Through its innovations, General Compute successfully raised $15 million in a seed funding round, achieving a post-money valuation of $60 million. This funding was led by FUSE VC, with participation from Carya Venture Partners and Village Global Ventures.

The Quest for the Ideal Chips

The popularity of GPUs has exploded, but it is becoming increasingly clear that they are not optimal for executing AI models once they are trained. The computational needs for the inference phase, where a model generates responses, differ from those of training. New chips are therefore being developed to meet these specific requirements. The $20 billion transaction between Nvidia and Groq in December, along with the IPO of Cerebras valued at $57 billion last week, illustrate this trend.

Faced with the limited capacity of these companies, General Compute's co-founders, Finn Puklowski and Jason Goodison, explored other options. They turned to the specialized chips from SambaNova, a company backed by Intel, which focuses on inference and has been somewhat overshadowed in Silicon Valley discussions. This could change with the imminent launch of SambaNova's new chips this year. These chips, featuring a flexible architecture and increased memory for contextual storage, claim to outperform not only GPUs but also other specialized chips like those from Groq or Cerebras. Puklowski asserts that these new chips can generate 600 to 700 tokens per second, compared to about 250 tokens per second for GPUs.

General Compute has already ordered $300 million worth of SN50 chips from SambaNova and plans to be the first cloud provider to deploy them.

Optimizing Data Center Infrastructure

In addition to addressing the chip issue, General Compute is also tackling the question of their installation. SambaNova's air-cooled, energy-efficient chips allow for integration into existing data center infrastructures without requiring significant investments in new facilities. Puklowski is exploring colocation agreements, which involve installing General Compute's hardware in other companies' facilities. These agreements include not only data center providers but also cryptocurrency miners looking to repurpose their infrastructure, as the cost of producing a bitcoin often exceeds its market price.

General Compute launched its cloud offering last week, claiming it is already the fastest for running MiniMax 2.7, a powerful open-source language model.

Investments and Future Prospects

Joe Hasselmann, a venture capitalist who invested in Groq in 2021, recently launched a new fund, Evercrest Capital Partners, focused on AI. General Compute is among his first investments. Hasselmann sees similarities between SambaNova's partnership with General Compute and the relationship between CoreWeave and Nvidia, as well as the association of Groq's chips with its previous cloud offering.

According to Hasselmann, it is crucial for SambaNova to have a diverse set of clients capable of placing their chips in high-growth environments. "As much as General Compute is betting on SambaNova, SambaNova is betting on General Compute," he stated.

Towards a New Architecture for AI

The central question remains which computing architecture will dominate the future of AI. Inference clouds represent a bet on a world where many models and agents coexist, without a single provider dominating the market. In this context, the speed and cost of inference become key competitive factors. The recent $113 million raise by OpenRouter in its Series B illustrates the company's ability to provide its clients access to multiple models to optimize their token spending.

Speed is essential in this field, both to reduce costs and to enhance capabilities. Puklowski aims to transform workloads from an hour into tasks of five to ten minutes, and to make audio agents for customer service, which require rapid inference to be effective, more economical. "If you're using ChatGPT and it gives you 50 tokens per second, that's already much faster than our reading capability," he explained to TechCrunch. "Now that we are moving to agent-to-agent interactions, where agents read for us or query databases, speed becomes crucial."