Eight LLM Inference Engines to Consider

Serving an LLM in production is not just about loading weights into a library. Eight specialized frameworks cover cache management, request batching, and APIs, from data centers to local machines. Their common goal: to increase concurrency and reduce latency without altering the models.

Running Locally or at the Edge: Three Concrete Approaches

On constrained devices or workstations, llama.cpp offers a C/C++ runtime with no external dependencies, capable of running on consumer GPUs as well as on CPU-only machines. Its small footprint facilitates deployments, and the llama-server component provides an OpenAI-compatible HTTP server with parallel decoding and continuous batching, useful for low-latency responses to one request at a time. Local setup takes just a few minutes via its repository.

Ollama aims for a ready-to-use local experience by bundling model downloads, versioning, and service behind a CLI and an API. It accepts parallel requests per model via OLLAMA_NUM_PARALLEL, queues subsequent requests up to OLLAMA_MAX_QUEUE, and allocates memory in fixed parallel slots, with usage increasing with parallelism and context length. It targets single-user or small team use cases, while high levels of concurrency benefit from the mechanisms of vLLM or SGLang. Its documentation covers installation, model catalog, and concurrency parameters.

For heterogeneous targets, MLC-LLM relies on Apache TVM to compile model libraries on-the-fly or in advance for web, iOS, and Android. The same tool supports Linux, Windows, macOS, iOS, Android, and browsers, and is compatible with AMD, NVIDIA, Apple, and Intel GPUs. An OpenAI-compatible REST server offers a server mode that increases request concurrency on dedicated GPUs. The guides detail weight conversion and compilation necessary for mobile and web deployments.

Maximizing Speed on NVIDIA GPUs and Quantization

In exclusively NVIDIA configurations, TensorRT-LLM provides a specific inference library based on the TensorRT ecosystem, designed to reduce latency. This solution manages dynamic batching, a paged KV cache leveraging block reuse, as well as various low-precision quantizations, including FP8 for H100 and later generations, FP4 for Blackwell GPUs, and AWQ in INT4. Integration with the Triton Inference Server is done using a specific backend, and the repository provides documentation on building and the model compatibility matrix. This specialization tends to translate into speed gains on an entirely NVIDIA fleet.

To combine decoding throughput and memory compression, LMDeploy brings together two engines, TurboMind in C++/CUDA and a PyTorch engine. It implements a persistent batch and a blocked KV cache for concurrency, supports weight-only quantizations like AWQ 4 bits, and KV cache quantization in INT8/INT4. Automatic prefix caching can be activated in parallel with quantization. Two APIs, offline and server, allow comparisons on the same model and hardware, and the guides list supported models and options.

Large-Scale Stream Orchestration and API Compatibility

For high-traffic services, vLLM is a common choice. Its PagedAttention mechanism segments the KV cache into small blocks, increasing the number of requests batched per GPU. Continuous batching allows new requests to enter an ongoing batch, and an OpenAI-compatible server limits client-side modifications. vLLM operates both as a library for offline processing and as a standalone server, on NVIDIA, AMD, and Intel GPUs, as well as on CPUs and TPUs. The documentation also covers model slicing via tensor parallelism.

SGLang combines a fast backend with a language for describing interactions with the model, useful for complex prompt chains with branches. The co-design of backend/frontend provides fine control over multi-step generations, while RadixAttention reuses the KV cache for requests sharing a prefix. The system supports structured outputs and constrained decoding for, for example, JSON returns or tool calls. In addition to serving LLMs, SGLang supports diffusion models for images and videos and offers both server mode and an embedded engine.

What These Frameworks Bring Compared to Generic Libraries

A production application receives streams of requests, each requiring memory for the KV cache, a place in a batch, and an output stream. While PyTorch or Transformers may suffice to serve a model, the queue, cache, and API still need to be built. Inference frameworks encapsulate these layers: token generation, cache and batch management, and API interface. The result is more simultaneous requests and lower latency, without modifying the weights. The offering covers both high-throughput deployments in data centers and local, edge, or resource-constrained contexts.

Low-Level Customization and Selection Criteria

Some teams seek deeper control. MAX, the framework from Modular, operates on CPU and GPU, allows customizations at the model and kernel level, and also serves diffusion models. Service options—batch size, memory usage, sequence length—are exposed via the max serve command, with explicit matching to hardware capabilities in the documentation. To distribute a model across multiple GPUs, the tensor parallelism documented by vLLM is another useful building block.

The choice depends on traffic profile, available hardware, and deployment targets. Single-user or small team workloads can accommodate Ollama, while environments with many simultaneous users benefit from the batching and paged cache mechanisms of engines like vLLM or SGLang. Multi-platform needs may steer users toward MLC-LLM, designed to cover different classes of devices with the same runtime.