# jitllm

> jitllm is an operating system for LLM inference: one self-contained executable that compiles its kernels for the machine at run time, places a model's blocks across every CPU core and GPU, and pages weights, experts and KV state through memory. It serves OpenAI- and Anthropic-compatible APIs and embeds as a Go library.

Things to know:

- The engine reads only `.jlm` containers: `jitllm convert <gguf|hf-dir|hf://repo|catalog-name> out.jlm` makes one, and a GGUF given to the engine is refused with that command.
- Put flags before the model path; a flag after it is refused.
- `jitllmd serve` serves `POST /v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `GET /v1/models` (OpenAI), `POST /v1/messages` (Anthropic) `GET /v1/capabilities` and `GET /healthz` on one address.
- `jitllm capabilities -json` (or `GET /v1/capabilities`) says what this build converts, runs and serves; `jitllm doctor -json [model.jlm]` says whether this machine can run, with a next step for each thing missing (exit 0 ok, 2 missing, 1 error).
- Every recipe below is run as a test against a real server; start with one when integrating.
- Every page is also served as Markdown: replace a page's trailing slash with `.md`.

## Use jitllm

Install it, get a model, and connect an application.

- [jitllm documentation](http://jitllm.org/docs.md): What jitllm is, what it runs on, and where to start.
- [Install](http://jitllm.org/docs/install.md): Install jitllm, jitllmd and the two apps from a release on Linux, macOS or Windows, or build them from source.
- [Quickstart](http://jitllm.org/docs/get-started.md): Fetch a 30B mixture of experts, serve it over the OpenAI API and send it a request.
- [Start a server and make the first request](http://jitllm.org/docs/recipes/start-a-server.md): Serve a model with `jitllmd` and get a completion, then a chat reply, from its OpenAI-compatible API.
- [Connect an OpenAI-compatible client](http://jitllm.org/docs/recipes/openai-clients.md): Point the official `openai` SDKs, or plain curl, at `jitllmd`.
- [Tools and structured output](http://jitllm.org/docs/recipes/tools-and-structured-output.md): Let a model call your functions, force it to call one, and hold a reply to a JSON schema.
- [Choose a supported checkpoint and convert it](http://jitllm.org/docs/recipes/convert-a-checkpoint.md): Find out whether jitllm runs a model before you download it, then convert it to the `.jlm` container the engine reads.
- [Serve an API](http://jitllm.org/docs/guides/serve.md): Run jitllmd for OpenAI and Anthropic clients, with streaming, tool calls, sessions and the Connect control plane.
- [Convert models](http://jitllm.org/docs/guides/convert.md): Turn a GGUF, a Hugging Face model or a catalog name into a .jlm container.
- [Run from the command line](http://jitllm.org/docs/guides/run.md): Prompts, chat templates, sampling, images, embeddings and benchmarks with the jitllm binary.
- [Embed it in Go](http://jitllm.org/docs/guides/go.md): Use the engine as a Go library, with the same model format and placement control as the CLI and server.
- [The desktop app](http://jitllm.org/docs/guides/desktop.md): Chat, choose, convert and watch a model from one window, on the same engine the CLI and the server use.
- [The terminal app](http://jitllm.org/docs/guides/terminal.md): Chat with a model, choose one, download one and move its blocks between the CPU and the GPU, all from a terminal.
- [Supported models](http://jitllm.org/docs/reference/models.md): The architectures jitllm runs, the weight formats it reads, and the catalog convert fetches by name.
- [Compared with other engines](http://jitllm.org/docs/compare.md): jitllm's features checked against llama.cpp, Ollama, vLLM, SGLang, TensorRT-LLM, MLX, mistral.rs, ZML, Magnitude and colibri and the measured results.

## Operate jitllm

Memory, devices, diagnosis and the reference for running it.

- [Memory budgets and device placement](http://jitllm.org/docs/recipes/memory-and-placement.md): Hold a model to a memory budget, see what the budget costs, and put blocks on the devices you choose.
- [Diagnose a slow or failed request](http://jitllm.org/docs/recipes/diagnose.md): Read what the server says about a request before guessing:
- [Recipes](http://jitllm.org/docs/recipes.md): Short, tested paths from an intent to a working result.
- [Devices and placement](http://jitllm.org/docs/guides/devices.md): Choose GPUs, budget their memory, and let jitllm move blocks between the CPU and devices while it serves.
- [Memory and paging](http://jitllm.org/docs/guides/memory.md): How jitllm runs models larger than RAM, what a page budget is, and how to set one.
- [Troubleshooting](http://jitllm.org/docs/guides/troubleshooting.md): Common errors and what to do about them.
- [Hardware support](http://jitllm.org/docs/reference/hardware.md): The operating systems, CPU instruction sets and GPU backends jitllm generates code for.
- [CLI reference](http://jitllm.org/docs/reference/cli.md): Every jitllm and jitllmd subcommand and flag.
- [Benchmarks](http://jitllm.org/docs/reference/benchmarks.md): jitllm against llama.cpp, vLLM, mistral.rs and ZML on V100s, a Xeon and an M4 (with MLX on the M4), with the method behind every number.

## Contribute to jitllm

How the engine works inside, and the rules a change is held to.

- [How it works](http://jitllm.org/docs/concepts/how-it-works.md): jitllm as an operating system for inference: a compiler per machine, a scheduler for blocks, and a pager for weights and state.
- [Kernel fusion and block bridging](http://jitllm.org/docs/concepts/fusion.md): How jitllm removes per-op dispatch cost without giving up the block as the unit of placement, relocation and paging.
- [The .jlm format](http://jitllm.org/docs/reference/jlm.md): jitllm's model container, and why the engine runs no other weight format.
- [Project rules (AGENTS.md)](https://github.com/jitllm/jitllm/blob/main/AGENTS.md): the binding rules: measurement, correctness gates, architecture scope
- [Contributing](https://github.com/jitllm/jitllm/blob/main/CONTRIBUTING.md): how to send a change
- [Testing](https://github.com/jitllm/jitllm/blob/main/docs/testing.md): how tests find models, and every model-selecting variable
- [Agent evaluations](https://github.com/jitllm/jitllm/blob/main/dev/agenteval/README.md): tasks that measure whether an agent can use and change jitllm

## Optional

- [Everything above in one file](http://jitllm.org/llms-full.txt)