Magnitude self-optimizes an LLM inference engine to your exact hardware, compiling kernels on-device so open models run up to 2x faster than llama.cpp.
Yes, Magnitude is open source under the Apache-2.0 license.
Magnitude is free to use.
Yes, Magnitude can be self-hosted (the source is available under the Apache-2.0 license).
Magnitude is an open-source inference engine for agents that optimizes itself for your exact hardware: it compiles and tunes its own kernels on your device so open models run faster on the machine you already have. Open source: yes (Apache-2.0); self-hostable; free OSS. It has ~6k stars and is very active (~1,000+ commits on main), and the maker reports open models running up to 2x faster than llama.cpp across Apple Silicon, NVIDIA, AMD, or CPU.
Most local-inference stacks ship generic kernels and hope they suit your chip. Magnitude instead treats the hardware as part of the problem: it compiles and tunes its kernels on the device where it runs, so the same open model is served with code shaped to that specific CPU or GPU. The maker's benchmark puts the result at up to 2x the throughput of llama.cpp, with coverage spanning Apple Silicon, NVIDIA, AMD, and plain CPU. For an agent that makes many model calls, more tokens-per-second on owned hardware is the difference between a snappy loop and a slow one — and because it is self-hosted and Apache-2.0, the engine, the models, and the tuning all stay on infrastructure you control. Open source: yes (Apache-2.0); self-hostable; free OSS.
Agents are only as cheap and fast as the inference underneath them, and the moment you run open models yourself the per-call economics are set by how well the engine uses your hardware. Magnitude's bet is that device-specific kernel compilation beats generic binaries — and if the 2x-over-llama.cpp claim holds on your box, that is real money and latency back on every agent step, with nothing leaving your infrastructure. For a GTM team self-hosting models behind prospecting, enrichment, or support agents, it is worth a head-to-head. The honest read: the speedup is a vendor benchmark and will vary with your model, quantization, and chip, and on-device tuning adds a first-run cost — so measure it against your current stack before you commit, rather than taking the 2x at face value.