Serve an API
jitllmd serve runs the engine as a server. One address carries three surfaces, all projections of the same engine with no second inference path behind any of them:
| Path | Surface |
|---|---|
POST /v1/chat/completions |
OpenAI chat completions |
POST /v1/completions |
OpenAI text completions |
GET /v1/models |
OpenAI model list |
POST /v1/embeddings |
OpenAI embeddings |
/v1/files, /v1/batches |
OpenAI Files and Batch APIs |
POST /v1/messages |
Anthropic messages |
POST /v1/systemone |
a decision model’s answers (jitllm’s own) |
/jitllm.v1.* |
the Connect control plane (Connect, gRPC and gRPC-Web) |
GET /v1/capabilities |
what this build serves: every route, each request field and its limits, and the combinations refused |
GET /metrics |
engine counters in Prometheus text format |
GET /healthz, GET /health |
returns ok |
As a service
Section titled “As a service”jitllmd service install # on localhost:4242, now and at every loginjitllmd service statusIt runs jitllmd serve over the models folder as a systemd user unit on Linux, a launchd agent on macOS (the app’s “Start the server at login” is the same agent) and a login entry on Windows (the installer’s option), restarted after a crash. stop, start and uninstall manage it. Logs: journalctl --user -u jitllm, ~/Library/Logs/jitllm/jitllmd.log, %LOCALAPPDATA%\jitllm\jitllmd.log.
Models by name
Section titled “Models by name”Every .jlm in the models folder is available by its file name, with or without .jlm, or by an alias, and a request naming one that is not loaded loads it (with -devices, -gpu-layers and -sessions; -on-demand=false turns this off). GET /v1/models lists the loaded models, then the available ones with "loaded": false. A model that does not fit memory pages from disk; a load that fails says which model, its size and the budget.
jitllmd models -alias Qwen3-8B-Q4_K_M=qwen3 # several names per model, unique across the folderjitllmd models -disable qwen3 # listed, never loaded on demandjitllmd models -enable qwen3Aliases and enable/disable are kept in the folder’s jitllm-models.json and survive a restart. jitllm convert NAME gives the container its catalog name as an alias.
Start the server
Section titled “Start the server”jitllmd serve -addr localhost:4242 -models ./models \ -load qwen3-8b.jlm -id qwen3 -devices auto| Flag | Default | Meaning |
|---|---|---|
-addr |
localhost:4242 |
listen address. Loopback by default: nothing off this machine reaches it. An address with no host (:4242) or a non-loopback one opens it to the network, and serve warns unless -api-key is set |
-api-key KEY |
JITLLM_API_KEY, else none |
require Authorization: Bearer KEY (and x-api-key: KEY on /v1/messages) on every route but /healthz |
-allow-origin LIST |
none | comma-separated browser origins that may call the server cross-origin. Loopback pages and clients that send no Origin are unaffected; any other origin is refused |
-models DIR |
the models folder | directory scanned by the model list; its containers are available by name. JITLLM_MODELS sets the default. |
-on-demand |
true | load a model a request names that is not loaded, unless it is disabled |
-load FILE |
a .jlm to load at startup, absolute or relative to -models |
|
-id NAME |
generated (m-…) |
the model name clients send. The file name without .jlm works too. |
-devices SPEC |
CPU only | where to run it, in the device grammar. Empty opens no device. |
-maxmem SIZE |
engine’s own budget | host page budget, e.g. 8G |
-gpu-layers N |
-1 | at most N blocks on a device; -1 is as many as fit |
-sessions N |
1 | concurrent sessions each linear device block reserves a recurrent state for; attention history is paged and reserves nothing |
-max-seq N |
model’s context | default KV capacity per session, in positions |
-max-batch N |
0 | requests of one model that decode as rows of one step; 0 is the engine’s bound (the widest step across sessions), 1 turns batching off |
-prompt-chunk N |
0 | prompt tokens a joining request feeds into one shared step; 0 is one device prefill chunk |
-step-prompt-tokens N |
0 | prompt tokens one step carries beside decoding rows; 0 measures it: the tokens that cost one decode step’s time |
-step-cost X |
0 | with -step-prompt-tokens 0, how many decode steps’ time a step carrying prompt tokens may take; 0 is the engine’s default |
-fairness N |
100 | 0 to 100: 0 serves requests first come first served, each to its end (the most tokens a second); higher time-slices running rows for waiting ones and feeds the next prompts beside the oldest (shorter, evener waits for a first token) |
-joint-steps MODE |
auto |
auto times a joint step against the rows one after another, per row count, and runs the faster; always or never force it |
-warm |
true | after every load, run a short prefill and a decode step so the first request does not pay for them |
-no-mem-cache |
false | turn off each model’s memory cache of prompt prefixes: every request prefills its whole prompt |
-mem-cache-max SIZE |
an eighth of the model’s host share | bound each model’s memory cache |
-host-mem SIZE |
eight tenths of what the host has | host memory the loaded models divide |
-session-pool N |
0 | reset sessions each model keeps warm for requests; 0 is -sessions, -1 none |
-max-queue N |
0 | requests one model holds, running or waiting, before it answers 429; 0 is 64, -1 unbounded |
-retry-after D |
1s |
the Retry-After a 429 for a full queue names |
-kv-f16 |
engine’s own | the KV cache width of every load that names none: -kv-f16 binary16, -kv-f16=false f32 |
-kv-cache DIR |
where sessions created with prompt_cache keep their prompt prefixes; without it such a session is refused |
|
-kv-cache-max SIZE |
8G |
what -kv-cache may occupy before its least recently used pages go; 0 is unbounded |
-image-max SIZE |
20M |
the most encoded bytes one picture in a request may be |
-image-max-pixels N |
16777216 | the most pixels (width x height) one picture may be, read from its header before decoding |
-image-max-count N |
8 | the most pictures one request may carry |
-image-max-request SIZE |
64M |
the most encoded bytes all of a request’s pictures may be |
-image-decode-timeout D |
5s |
how long decoding one picture may take |
-image-fetch-allow LIST |
comma-separated hosts a picture named by an http(s) URL may be fetched from: exact names, or *.example.com for one label under it. Empty fetches nothing; inline base64 pictures are always taken |
|
-image-fetch-http |
false | let a fetch use plain http (https only otherwise) |
-image-fetch-ports LIST |
443,80 |
ports a fetch may connect to |
-image-fetch-concurrency N |
4 | remote fetches running at once across the server |
-image-fetch-timeout D |
10s |
how long one remote fetch may take |
More models can be loaded and unloaded while the server runs; see Manage a running server. Requests beyond what a model runs at once wait in its queue; past -max-queue the server answers 429 with a Retry-After.
Batching
Section titled “Batching”Requests to a model whose every block and the output head are on one device, or whose every block and head are on the CPU, are batched continuously. Each request becomes a row of the model’s step loop, and every step runs one token of every row in one pass over the weights (one device step, or one host pass); each row reads its own session’s history and samples with its own settings. A joining request’s prompt is fed a chunk at a time inside the same steps, beside the rows already generating, so admitting it never stalls them.
-max-batch caps how many requests share a step, -prompt-chunk sets how many prompt tokens a step carries for a joining request, and with -joint-steps auto the server times a joint step against running the rows one after another, for each row count, and keeps the faster. jitllmd stats shows the batch.
A model split between the CPU and a device runs one request at a time.
OpenAI chat completions
Section titled “OpenAI chat completions”Any OpenAI SDK works with the base URL http://HOST:PORT/v1. No API key is checked, but most SDKs insist on one, so pass any string.
curl http://localhost:4242/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "qwen3", "messages": [ {"role": "system", "content": "You are terse."}, {"role": "user", "content": "Name three prime numbers."} ], "max_tokens": 64, "temperature": 0.7, "stream": true }'Supported fields: model, messages, max_tokens or max_completion_tokens, temperature, top_p, top_k, min_p, repetition_penalty, seed, stop, stream, n, logprobs and top_logprobs, response_format, ignore_eos, tools and tool_choice, plus jitllm’s jitllm_session, jitllm_priority and jitllm_speculate. The prompt goes through the model’s own chat template. A streamed response is server-sent events ending in data: [DONE]; finish_reason is stop, length, tool_calls, or cancelled when the request was cancelled. Without max_tokens the reply runs until the model ends it or the session’s context is full, which finishes as length; a max_tokens larger than the context has room for is cut to that room.
/v1/completions takes a raw prompt instead of messages, with echo to return the prompt too.
logprobsare the model’s raw distribution (the log-softmax of the logits, before temperature and the sampling cuts), as vLLM returns by default.nasks for several continuations; the prompt is prefilled once and every other choice restores it.response_format(json_object, orjson_schemawith itsschema) constrains every sampled token to the schema; a schema keyword that is not supported is a 400 naming it.jitllm_speculate: trueturns on speculative decoding for a greedy request: it drafts with the model’s own prediction block where it has one and by prompt lookup otherwise, and the output is plain greedy’s token for token.
The server reference has how these combine.
OpenAI embeddings
Section titled “OpenAI embeddings”An embedding model loaded on the server answers POST /v1/embeddings:
jitllmd serve -models ./models -load minilm.jlm -id minilm
curl http://localhost:4242/v1/embeddings \ -H 'Content-Type: application/json' \ -d '{"model": "minilm", "input": ["a sentence to embed", "and another"]}'input is a string, a list of strings, a list of token ids or a list of lists of them. Each vector is pooled and L2-normalised the way the model was trained. encoding_format is float (the default) or base64, the vector’s little-endian float32 bytes; dimensions, when given, must be the model’s own width, since vectors are not truncated.
Anthropic messages
Section titled “Anthropic messages”curl http://localhost:4242/v1/messages \ -H 'Content-Type: application/json' \ -d '{ "model": "qwen3", "system": "You are terse.", "messages": [{"role": "user", "content": "Name three prime numbers."}], "max_tokens": 64, "stream": true }'import anthropic
client = anthropic.Anthropic(base_url="http://localhost:4242", api_key="unused")msg = client.messages.create( model="qwen3", max_tokens=256, messages=[{"role": "user", "content": "Why is the sky blue?"}],)print(msg.content[0].text)system may be a string or a list of text blocks, and so may content. Supported sampling fields are temperature, top_p, top_k and stop_sequences. A stream is the named event sequence (message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop), and stop_reason is end_turn, max_tokens, stop_sequence or tool_use.
Pictures
Section titled “Pictures”On a model converted with its vision projector (jitllm convert model.gguf mmproj.gguf out.jlm), a chat request may carry pictures: image_url content parts on /v1/chat/completions, image blocks on /v1/messages, and images on a Connect ChatMessage. PNG, JPEG, WebP, GIF (its first frame), BMP and TIFF are taken, inline as base64:
{"model": "smolvlm", "messages": [{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}, {"type": "text", "text": "What is in this picture?"}]}]}The pictures keep the order you sent them and go where the model’s own chat template puts its image markers, and the model’s vision tower encodes each one as jitllm run -image does. They combine with tools and a forced tool_choice, with n (the prompt, picture included, is prefilled once), and with a continued session (the history keeps its earlier pictures).
One picture may be 20 MiB and 4096x4096 pixels, and a request may carry 8 (-image-max, -image-max-pixels, -image-max-count). Inline pictures are always taken. A picture named by an http(s) URL is fetched only from the hosts listed in -image-fetch-allow, matched exactly (or one label under a *.example.com entry) on the first URL and every redirect, over https on ports 443 or 80 unless -image-fetch-http and -image-fetch-ports say otherwise, with no userinfo, no downgrade to http, at most three redirects, and never to a private, loopback, link-local, NAT64, IPv4-mapped or other special-use address, checked as each connection is made. A picture sent to a model without a vision tower, an unsupported, mismatched, oversized or undecodable picture, or a remote URL with fetching off is a 400 that names the part, never a reply from a model that did not see it. A request with pictures prefills alone rather than joining the step loop.
Tool calls
Section titled “Tool calls”Both APIs accept tool definitions in their own shapes: OpenAI’s tools with function.parameters, Anthropic’s tools with input_schema. They reach the model’s own chat template, so a model trained for tool use sees the prompt it was trained on: OpenAI definitions exactly as sent, Anthropic ones converted to the OpenAI shape with input_schema kept byte for byte.
curl http://localhost:4242/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "qwen3", "messages": [{"role": "user", "content": "What is the weather in Paris?"}], "tools": [{ "type": "function", "function": { "name": "get_weather", "description": "Current weather for a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]} } }] }'The model’s output is parsed back into tool_calls (OpenAI) or tool_use blocks (Anthropic). While streaming, ordinary text streams as it arrives; text that is, or may be becoming, a call is held back and emitted as a call once it parses. Send the tool’s result back as a tool message (OpenAI) or a tool_result block (Anthropic) and the conversation continues.
tool_choice: "none" (Anthropic: {"type": "none"}) withholds the tools, and "auto" (the default) lets the model call one or answer. "required" and a named function (Anthropic: {"type": "any"} and {"type": "tool", "name": ...}) force a call: the reply is sampled under a grammar that admits only a call to a declared tool, or to the named one. parallel_tool_calls: false (Anthropic: disable_parallel_tool_use) limits the reply to one call.
Two call formats are parsed, which cover the models trained for tools: Hermes-style <tool_call>{...}</tool_call> blocks (Qwen2.5, Qwen3 and most fine-tunes), and a bare JSON call opening the reply (Llama 3.1 and 3.2, Mistral v0.3). Only a call to a tool you declared counts, so a model asked to answer in JSON does not have its answer taken for a call.
What jitllm adds to a response
Section titled “What jitllm adds to a response”Every response carries one extra field, jitllm, which clients that do not know it ignore. It is in the body when not streaming, and on the final chunk (OpenAI) or the message_delta event (Anthropic) when streaming:
"jitllm": { "session_id": "s-1727614512345-1", "queued_ms": 0, "queue_depth_on_entry": 0, "device_blocks": 36, "cpu_blocks": 0, "device_ids": ["cuda:0"], "prefill_ms": 41, "decode_ms": 1204, "decode_tokens_per_second": 104.6, "bytes_per_token": 816010912}It says where the request ran and how long it waited, which an OpenAI response has nowhere else to put.
A request may also name a session instead of a model with "jitllm_session" and the id jitllmd sessions -new printed, to run on a session you created with its own devices and KV capacity. A session named that way takes precedence over model.
Manage a running server
Section titled “Manage a running server”jitllmd is also a client of a running server, through the same Connect API. Add -addr host:port to reach one that is not local.
jitllmd models # the model directory, and what is loadedjitllmd models -load gemma-3-4b.jlm -id gemma -devices cuda:0jitllmd models -unload gemma # -force closes sessions holding it instead of refusing
jitllmd devices # hardware and spendable memoryjitllmd stats -watch # engine counters, streamed
jitllmd sessions -new -model qwen3 -devices cuda:0 -max-seq 8192jitllmd sessions # list themjitllmd sessions -reset SESSION # back to position 0
jitllmd place -session SESSION # where each block isjitllmd place -session SESSION -devices cuda:0 -gpu-layers 22jitllmd place -session SESSION -move 20:35 -to hostjitllmd place -model qwen3 -maxmem 12G # re-budget the pager
jitllmd run -model qwen3 -chat -n 64 "Hello" # an ephemeral sessionjitllmd run -session SESSION -chat "Hello" # keep the session's historyjitllmd run -session SESSION -continue "and then?" # continue from its positionSESSION is the id sessions -new printed. place -devices must name devices the model was loaded with, and place with no -devices brings every block home to the CPU. A move waits for the session’s current generation to finish, and the conversation keeps its history: the KV cache and any recurrent state move with the blocks. jitllmd <verb> -h lists each verb’s flags; the CLI reference has them all.
Ask the server what it serves
Section titled “Ask the server what it serves”GET /v1/capabilities returns the same manifest as jitllm capabilities -json, for the running build: the architectures and quantizations it converts and runs, each backend and the hardware it is verified on, every route with its request fields and their limits, and the combinations the server refuses rather than ignores, each with the message it returns. An agent or a client can read it before sending anything. On the machine itself, jitllm doctor says whether this build can run here, and what to do about anything missing.
The Connect API
Section titled “The Connect API”Everything the client does is a Connect RPC, generated from the protobuf definitions in proto/jitllm/v1. Any Connect, gRPC or gRPC-Web client can call it, over HTTP/1.1 or cleartext HTTP/2.
| Service | Methods |
|---|---|
ModelService |
ListModels, GetModel, LoadModel, UnloadModel, Convert, Tokenize, Detokenize, ApplyChatTemplate |
DeviceService |
ListDevices, GetDevice, GetMemoryTopology |
SessionService |
CreateSession, GetSession, ListSessions, CloseSession, ResetSession, GetDeviceQueue |
PlacementService |
GetPlacement, SetPlacement, RelocateBlocks, SetRelocation, TuneSeam, GetResidency, SetPageBudget |
InferenceService |
Generate (streaming), Complete, Cancel, Embed |
TelemetryService |
GetStats, WatchStats (streaming), GetServerInfo |
curl http://localhost:4242/jitllm.v1.DeviceService/ListDevices \ -H 'Content-Type: application/json' -d '{}'