llama.cpp

llama.cpp exists to make model inference work on whatever hardware you already own. It is a plain C and C++ implementation with no dependencies, aimed at running language and vision models locally or in the cloud with minimal setup and strong performance.

The project is built on the ggml tensor library and ships two main tools: a command line interface for talking to a model and a server that exposes an OpenAI-compatible API with a built-in web UI. Either can pull a model straight from Hugging Face by name.

Hardware coverage is the reason most people end up here.

• Apple silicon: a first-class target, optimized through ARM NEON, Accelerate and Metal.
• x86 and RISC-V: AVX, AVX2, AVX512 and AMX on x86, with RVV and related extensions on RISC-V.
• GPU backends: custom CUDA kernels for NVIDIA, HIP for AMD, MUSA for Moore Threads, plus Vulkan, SYCL, OpenCL and WebGPU.
• Quantization: integer formats from 1.5-bit to 8-bit cut memory use and speed inference up.
• Hybrid inference: CPU and GPU share the work, so a model larger than the available VRAM still runs.

Pre-built binaries accompany each release, a Docker image is documented, and building from source covers anything unusual. Guides exist for Android builds, multi-GPU setups and performance troubleshooting, and the tools include a server, a completion utility and grammar-constrained output. It suits anyone who wants inference under their own control on a hardware list that reaches far past the usual two vendors.