Qwen 2.5 Outperforms ChatGPT: The Winning Bet of a Local AI Server

Qwen 2.5: A Serious Competitor for ChatGPT

Six months ago, a figure changed our perception of AI capabilities: the Qwen 2.5 Coder 32B model achieved an impressive score of 92.9 on the HumanEval benchmark, surpassing GPT-4o, which scored 90.2. HumanEval is an industry standard for evaluating code generation, covering 164 programming problems across various languages. This result highlighted that a free open-source model, running on hardware accessible to the general public, could compete with a model for which our team was paying $30 per user per month. This realization triggered a rigorous audit of our expenses.

A Revealing Cost Audit

Before embarking on setting up a new infrastructure, we analyzed the AI tasks performed by our team of ten over a typical week. The distribution of tasks turned out to be more imbalanced than expected:

• ~45% were writing tasks, including emails, documentation, and summaries.
• ~30% involved coding, such as debugging and function generation.
• ~15% were analysis tasks, such as data interpretation.
• ~10% required cutting-edge capabilities or real-time information.

The audit revealed that the 10% of tasks requiring advanced intelligence were subsidizing the remaining 90%. We were paying a monthly fee per user for tasks where a local 14B model produced results comparable to those of GPT-4o. The question was not whether the local AI was better, but whether the quality difference justified the additional cost. For our team, the answer was no, especially at $300/month.

Hardware Choice: A Strategic Decision

To host our AI locally, we opted for a RTX 3090–24GB VRAM graphics card, purchased second-hand for $600. The 24 GB VRAM threshold is crucial for running 32B models with Q4 quantization. 14B models are possible below this threshold, but they perform less effectively on complex tasks.

Here’s the hierarchy of hardware capabilities:

• CPU only: 16–64 GB RAM for 7B models (3–8 tok/s), suitable for simple tasks.
• RTX 3070 / 4060 Ti: 8 GB for 7B–8B models, sufficient for everyday tasks.
• RTX 3080 / 4080: 16 GB for 13B–14B models, close to the limit on most tasks.
• RTX 3090 / 4090: 24 GB for 32B–34B models, competitive with GPT-4o.
• Dual 3090 / A6000: 48 GB+ for 70B models, offering cutting-edge capabilities.

The total cost of the infrastructure amounted to ~$1,200, including the GPU, a second-hand workstation, and 2 TB of NVMe storage. The break-even point compared to our ChatGPT Team subscription was reached in four months.

Model Selection

We tested each major open-source model against our task distribution before making our final choice.

General Tasks — Qwen 2.5 14B

• Pull Command: ollama pull qwen2.5:14b

This model efficiently handles writing, email drafting, summarization, and Q&A. It fits within 9 GB of VRAM with Q4 quantization, leaving 15 GB for other processes. In writing tasks, the results of Qwen 2.5 14B were indistinguishable from those of GPT-4o in blind tests.

Coding Tasks — Qwen 2.5 Coder 32B

• Pull Command: ollama pull qwen2.5-coder:32b

This model excels in Python, TypeScript, Go, Rust, SQL, and shell scripting, producing idiomatic outputs and precise debugging explanations. It uses ~20 GB of VRAM in Q4, leaving little margin on a 24 GB card.

Reasoning Tasks — DeepSeek R1 14B

• Pull Command: ollama pull [deepseek](/dossier/deepseek)-r1:14b

DeepSeek R1 employs a chain-of-thought architecture, externalizing its reasoning process before providing an answer. This approach yields more accurate results on complex analytical tasks.

Voice Pipeline

• Speech-to-Text: pip install faster-whisper# or via Ollama: ollama pull whisper

Whisper Large v3 Turbo achieves a word error rate of less than 3% on clean audio, equivalent to OpenAI's paid Whisper API.

• Text-to-Speech: pip install kokoro

Kokoro (82M parameters) runs on CPU, producing natural speech rated above models ten times larger, with a response time of less than 200 ms.

Document Q&A — RAG with nomic-embed-text

• Pull Command: ollama pull nomic-embed-text

Nomic-embed-text is the embedding model that enables Retrieval Augmented Generation (RAG). It converts documents into vector representations, stored in Qdrant, allowing the AI to retrieve relevant information.