Get jitllm
Serves OpenAI and Anthropic APIs

jitllm: An operating system for LLM inference.

Your models. Your hardware. Full speed.JIT kernels that match your hardware · paging so big models still run · auto-tuning & layer orchestration

Get jitllmRead the source

Why jitllm

One binary you drop onto a machine and start running models.

Inference engines are bulky and rigid: a lot to set up, and getting the most out of your hardware is another job.

jitllm borrows from operating systems and JIT compilers. It detects your CPUs and GPUs, generates kernels for them, and places the model at runtime. Models larger than memory page from disk, and layers move between devices without losing the conversation.

Watch it schedule

A small simulation of the runtime: models load from .jlm, placement picks a device for every block, blocks page from disk straight onto that device, and a token runs through them in order, growing each device's KV cache. Blocks and their KV move between devices only when placement changes. Click a model to send it a request, add CPUs and GPUs, or change where the blocks go.

Loaded models

    Devices
    Placement

    One engine, every way in

    Each program is one self-contained binary, for Linux, macOS and Windows on x86-64 and arm64. GPUs need only their driver.

    A desktop app on the same engine

    Model downloads, conversion, text and image chat, and a live view of where each block of the model sits, in one window. The engine runs inside the app's own process.

    The desktop app
    The jitllm desktop app, Chat screenThe jitllm desktop app, Chat screen
    Qwen3-30B-A3B, a 30B mixture of experts, answering on a laptop with a 4 GB GPU: one block on the card, the rest on the CPU, its experts paged from disk.

    And in a terminal

    The same screens in a terminal, over SSH too, sharing the desktop app's settings and chats. The arrow keys move the model's blocks between the CPU and the GPU, and the conversation keeps its history across the move.

    The terminal app
    The jitllm terminal app, Chat screen
    Qwen3-30B-A3B answering on a laptop with a 4 GB GPU, the engine beside the conversation.

    Bring the models you want

    Convert from GGUF or Hugging Face once, then run the .jlm anywhere jitllm runs.

    All models

    What an OS does for programs, jitllm does for models.

    It is one process. Inside it, models are the programs, their blocks are the threads, and every CPU core and accelerator on the machine is somewhere they can run.

    Placement, paging and tuning are automatic, and every one of those decisions can be forced by the application.

    Operating systemjitllm
    processa loaded model
    threada layer, a block
    CPU coresCPU cores and every accelerator
    schedulerwhich block runs where
    physical pagea packed block in a slot
    page faulta page-in
    executablethe .jlm container
    system callsOpenAI, Anthropic, Connect APIs

    The executable format

    Convert once. Page cheaply forever.

    An OS loads programs from a format built for loading. .jlm is that for models: GGUF or Hugging Face weights, with their quantization preserved, laid out once so evicting and reloading them during inference never repeats the preparation.

    One file carries the weights, configuration, tokenizer, chat templates and optional vision tower, for the CLI, the server, the desktop app and the Go library alike.

    The format
    • No repacking on page-in

      Quantized weights are stored in the layout the GPU backends share, so a reload is a read.

    • Straight to the needed bytes

      Predictable block offsets and addressable expert ranges let the pager fetch only what a token uses.

    • One layout, every device

      CUDA, Vulkan and Metal read the same container, so execution can move between backends and reuse it.

    Measured against the engines you know

    Same machine, same pass, same GGUF file. Ratios are jitllm ÷ the other engine; above 1.00× means jitllm is faster.

    4.62×

    llama.cpp's prefill on Qwen3-Next-80B-A3B

    4× V100, 512-token prompt

    5.5×

    vLLM's throughput, 128 sequences batched

    Llama-3.1-8B Q4_K_M, one V100

    34/35

    models prefill faster than on llama.cpp

    V100, 360M to 120B, 1 to 5 GPUs; the 35th level at 1.00×

    35/35

    models decode faster than on llama.cpp

    V100, 1.01× to 1.35×

    jitllm speed -p 512 -n 128 -r 2 against llama-bench -p 512 -n 128 -ngl 99 -r 3 on the file jitllm converted from; vLLM 0.18.1, generated tokens over wall clock. Q4_K_M unless named. One pass per host. Hosts, method and the CPU board.

    How it compares

    Pick an engine. Every feature is checked against that project's own documentation and source, and links to it.

    The full comparison

    Checked against each project's own documentation and source; every mark links to it. Out of date? Tell us.

    Install jitllm

    One self-contained binary per program, for Linux, macOS and Windows on x86-64 and arm64. GPU backends use the drivers already on the machine; jitllm finds them at run time.

    Full guide

    macOS

    jitllm for Apple silicon (.dmg)

    Windows

    jitllm setup for x64 (.exe)jitllm setup for Arm (.exe)

    • The setup for the other architecture: x64, Arm
    • The desktop app alone, portable, installs nothing: x64, Arm
    • PowerShell: irm https://jitllm.org/install.ps1 | iex

    Linux

    curl -fsSL https://jitllm.org/install.sh | sh

    Every archive, with checksums.txt: the latest release.

    Install and fetch a model

    Install the CLI and the server from the latest release, then convert Qwen3-30B-A3B, a 30B mixture of experts that uses 3B per token (18.6 GB to download). On Windows: irm https://jitllm.org/install.ps1 | iex.

    shell
    $ curl -fsSL https://jitllm.org/install.sh | sh
$ mkdir -p models
$ jitllm convert -o models qwen3-30b-a3b models/qwen3.jlm

    Serve and stream

    Start the API server with automatic device selection, then stream a chat from another terminal.

    shell
    $ jitllmd serve -addr localhost:4242 -models ./models \
    -load qwen3.jlm -id qwen3 -devices auto
$ curl http://localhost:4242/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"qwen3","stream":true,
         "messages":[{"role":"user","content":"Hi"}]}'

    The engine

    Four subsystems between your application and the hardware.

    How it works
    01 · discover

    Enumerates the hardware

    Reads the CPU's capabilities and finds installed CUDA and Vulkan libraries or the Metal framework. Shared CPU/GPU memory is counted once, as one pool.

    SSE · AVX2 · VNNI · NEON · CUDA · Vulkan · Metal

    02 · compile

    Generates the code

    Native x86-64 and ARM64 code and GPU kernels, specialized for the model's shapes and weight formats, then tuned by measurement: kernel layouts, worker counts, tiles, splits.

    x86-64 · ARM64 · PTX · SPIR-V · MSL

    03 · schedule

    Places the work

    Blocks run on the CPU or on any GPU, spread across several at once. Placement can change between tokens: grow onto a GPU, give blocks back as context grows, tune the split.

    CPU ⇄ GPU · multi-GPU · live migration

    04 · page

    Manages memory

    Weights, experts, KV pages and recurrent state move between VRAM, RAM and storage. Run models larger than RAM, and make room for another model without discarding a conversation.

    weights · experts · KV · state

    Systems
    Linux · macOS · Windows
    Architectures
    x86-64 · ARM64
    Backends
    CPU · CUDA · Vulkan · Metal · ROCm (untested)
    Setup
    one executable, no Python, no SDK
    Models
    57 graphs · 17 vision projectors · each gated against its reference
    Hot path
    a warm decode token allocates nothing on the Go heap
    Faster tokens
    speculative decoding · int8 KV cache · prefix caching

    The server

    One binary, jitllmd, many models and many sessions at once. Applications reach it through APIs they already speak: point a client at http://localhost:4242/v1 and use the model's id as its name.

    API guide
    OpenAI-compatiblePOST /v1/chat/completions

    Chat, completions, embeddings, Batches and Files, streamed, with tool calls in each model's own format (24 families) and tool_choice enforced by constrained decoding.

    Anthropic-compatiblePOST /v1/messages

    The messages endpoint, streamed, with tool use and pictures through image blocks on vision models, for clients written against Anthropic's API.

    Connect APImodels · sessions · devices

    A control plane: load and unload models, create, park and resume sessions, move blocks between devices, and /metrics for Prometheus.

    • Continuous batchingOn the CPU and the GPU, prompts fed in chunks beside running decodes, with KV-aware admission.
    • Fairness-fairness 0-100 time-slices sessions through the card by parking them to the host and back, with per-request priority.
    • Structured outputFrom a JSON schema, plus log-probabilities and n choices from one prefill.
    • Decision answersOne question against fixed labels through /v1/systemone.

    Built for agents

    For the agent using jitllm, and for the agent changing it.

    llms.txt
    Capabilitiesjitllm capabilities

    What this build runs, generated from the code: architectures, quant types, backends, API fields and their refusals. Also at GET /v1/capabilities.

    Doctorjitllm doctor -json

    The host, its devices and a model's compatibility, each problem with a next action.

    Recipes · llms.txtdocs/recipes

    Tested recipes from install to a running server, a connected client, tools, conversion, placement and diagnosis; llms.txt indexes the documentation.

    The OpenAI- and Anthropic-compatible APIs work with the SDKs and agent frameworks as they are, and a forced tool call is a call. For an agent changing jitllm, AGENTS.md holds the project's binding rules, the model list is generated from the converter, and the model-free suite reports a missing model or device as a skip, never as a pass.