Skip to content

Serve an API

jitllmd serve runs the engine as a server. One address carries three surfaces, all projections of the same engine with no second inference path behind any of them:

Path Surface
POST /v1/chat/completions OpenAI chat completions
POST /v1/completions OpenAI text completions
GET /v1/models OpenAI model list
POST /v1/embeddings OpenAI embeddings
/v1/files, /v1/batches OpenAI Files and Batch APIs
POST /v1/messages Anthropic messages
POST /v1/systemone a decision model’s answers (jitllm’s own)
/jitllm.v1.* the Connect control plane (Connect, gRPC and gRPC-Web)
GET /v1/capabilities what this build serves: every route, each request field and its limits, and the combinations refused
GET /metrics engine counters in Prometheus text format
GET /healthz, GET /health returns ok
One address, one engine. No surface has an inference path of its own; a request on any of them runs in a session of the same loaded model.
Terminal window
jitllmd service install # on localhost:4242, now and at every login
jitllmd service status

It runs jitllmd serve over the models folder as a systemd user unit on Linux, a launchd agent on macOS (the app’s “Start the server at login” is the same agent) and a login entry on Windows (the installer’s option), restarted after a crash. stop, start and uninstall manage it. Logs: journalctl --user -u jitllm, ~/Library/Logs/jitllm/jitllmd.log, %LOCALAPPDATA%\jitllm\jitllmd.log.

Every .jlm in the models folder is available by its file name, with or without .jlm, or by an alias, and a request naming one that is not loaded loads it (with -devices, -gpu-layers and -sessions; -on-demand=false turns this off). GET /v1/models lists the loaded models, then the available ones with "loaded": false. A model that does not fit memory pages from disk; a load that fails says which model, its size and the budget.

Terminal window
jitllmd models -alias Qwen3-8B-Q4_K_M=qwen3 # several names per model, unique across the folder
jitllmd models -disable qwen3 # listed, never loaded on demand
jitllmd models -enable qwen3

Aliases and enable/disable are kept in the folder’s jitllm-models.json and survive a restart. jitllm convert NAME gives the container its catalog name as an alias.

Terminal window
jitllmd serve -addr localhost:4242 -models ./models \
-load qwen3-8b.jlm -id qwen3 -devices auto
Flag Default Meaning
-addr localhost:4242 listen address. Loopback by default: nothing off this machine reaches it. An address with no host (:4242) or a non-loopback one opens it to the network, and serve warns unless -api-key is set
-api-key KEY JITLLM_API_KEY, else none require Authorization: Bearer KEY (and x-api-key: KEY on /v1/messages) on every route but /healthz
-allow-origin LIST none comma-separated browser origins that may call the server cross-origin. Loopback pages and clients that send no Origin are unaffected; any other origin is refused
-models DIR the models folder directory scanned by the model list; its containers are available by name. JITLLM_MODELS sets the default.
-on-demand true load a model a request names that is not loaded, unless it is disabled
-load FILE a .jlm to load at startup, absolute or relative to -models
-id NAME generated (m-…) the model name clients send. The file name without .jlm works too.
-devices SPEC CPU only where to run it, in the device grammar. Empty opens no device.
-maxmem SIZE engine’s own budget host page budget, e.g. 8G
-gpu-layers N -1 at most N blocks on a device; -1 is as many as fit
-sessions N 1 concurrent sessions each linear device block reserves a recurrent state for; attention history is paged and reserves nothing
-max-seq N model’s context default KV capacity per session, in positions
-max-batch N 0 requests of one model that decode as rows of one step; 0 is the engine’s bound (the widest step across sessions), 1 turns batching off
-prompt-chunk N 0 prompt tokens a joining request feeds into one shared step; 0 is one device prefill chunk
-step-prompt-tokens N 0 prompt tokens one step carries beside decoding rows; 0 measures it: the tokens that cost one decode step’s time
-step-cost X 0 with -step-prompt-tokens 0, how many decode steps’ time a step carrying prompt tokens may take; 0 is the engine’s default
-fairness N 100 0 to 100: 0 serves requests first come first served, each to its end (the most tokens a second); higher time-slices running rows for waiting ones and feeds the next prompts beside the oldest (shorter, evener waits for a first token)
-joint-steps MODE auto auto times a joint step against the rows one after another, per row count, and runs the faster; always or never force it
-warm true after every load, run a short prefill and a decode step so the first request does not pay for them
-no-mem-cache false turn off each model’s memory cache of prompt prefixes: every request prefills its whole prompt
-mem-cache-max SIZE an eighth of the model’s host share bound each model’s memory cache
-host-mem SIZE eight tenths of what the host has host memory the loaded models divide
-session-pool N 0 reset sessions each model keeps warm for requests; 0 is -sessions, -1 none
-max-queue N 0 requests one model holds, running or waiting, before it answers 429; 0 is 64, -1 unbounded
-retry-after D 1s the Retry-After a 429 for a full queue names
-kv-f16 engine’s own the KV cache width of every load that names none: -kv-f16 binary16, -kv-f16=false f32
-kv-cache DIR where sessions created with prompt_cache keep their prompt prefixes; without it such a session is refused
-kv-cache-max SIZE 8G what -kv-cache may occupy before its least recently used pages go; 0 is unbounded
-image-max SIZE 20M the most encoded bytes one picture in a request may be
-image-max-pixels N 16777216 the most pixels (width x height) one picture may be, read from its header before decoding
-image-max-count N 8 the most pictures one request may carry
-image-max-request SIZE 64M the most encoded bytes all of a request’s pictures may be
-image-decode-timeout D 5s how long decoding one picture may take
-image-fetch-allow LIST comma-separated hosts a picture named by an http(s) URL may be fetched from: exact names, or *.example.com for one label under it. Empty fetches nothing; inline base64 pictures are always taken
-image-fetch-http false let a fetch use plain http (https only otherwise)
-image-fetch-ports LIST 443,80 ports a fetch may connect to
-image-fetch-concurrency N 4 remote fetches running at once across the server
-image-fetch-timeout D 10s how long one remote fetch may take

More models can be loaded and unloaded while the server runs; see Manage a running server. Requests beyond what a model runs at once wait in its queue; past -max-queue the server answers 429 with a Retry-After.

Requests to a model whose every block and the output head are on one device, or whose every block and head are on the CPU, are batched continuously. Each request becomes a row of the model’s step loop, and every step runs one token of every row in one pass over the weights (one device step, or one host pass); each row reads its own session’s history and samples with its own settings. A joining request’s prompt is fed a chunk at a time inside the same steps, beside the rows already generating, so admitting it never stalls them.

-max-batch caps how many requests share a step, -prompt-chunk sets how many prompt tokens a step carries for a joining request, and with -joint-steps auto the server times a joint step against running the rows one after another, for each row count, and keeps the faster. jitllmd stats shows the batch.

A model split between the CPU and a device runs one request at a time.

Any OpenAI SDK works with the base URL http://HOST:PORT/v1. No API key is checked, but most SDKs insist on one, so pass any string.

Terminal window
curl http://localhost:4242/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen3",
"messages": [
{"role": "system", "content": "You are terse."},
{"role": "user", "content": "Name three prime numbers."}
],
"max_tokens": 64,
"temperature": 0.7,
"stream": true
}'

Supported fields: model, messages, max_tokens or max_completion_tokens, temperature, top_p, top_k, min_p, repetition_penalty, seed, stop, stream, n, logprobs and top_logprobs, response_format, ignore_eos, tools and tool_choice, plus jitllm’s jitllm_session, jitllm_priority and jitllm_speculate. The prompt goes through the model’s own chat template. A streamed response is server-sent events ending in data: [DONE]; finish_reason is stop, length, tool_calls, or cancelled when the request was cancelled. Without max_tokens the reply runs until the model ends it or the session’s context is full, which finishes as length; a max_tokens larger than the context has room for is cut to that room.

/v1/completions takes a raw prompt instead of messages, with echo to return the prompt too.

  • logprobs are the model’s raw distribution (the log-softmax of the logits, before temperature and the sampling cuts), as vLLM returns by default.
  • n asks for several continuations; the prompt is prefilled once and every other choice restores it.
  • response_format (json_object, or json_schema with its schema) constrains every sampled token to the schema; a schema keyword that is not supported is a 400 naming it.
  • jitllm_speculate: true turns on speculative decoding for a greedy request: it drafts with the model’s own prediction block where it has one and by prompt lookup otherwise, and the output is plain greedy’s token for token.

The server reference has how these combine.

An embedding model loaded on the server answers POST /v1/embeddings:

Terminal window
jitllmd serve -models ./models -load minilm.jlm -id minilm
curl http://localhost:4242/v1/embeddings \
-H 'Content-Type: application/json' \
-d '{"model": "minilm", "input": ["a sentence to embed", "and another"]}'

input is a string, a list of strings, a list of token ids or a list of lists of them. Each vector is pooled and L2-normalised the way the model was trained. encoding_format is float (the default) or base64, the vector’s little-endian float32 bytes; dimensions, when given, must be the model’s own width, since vectors are not truncated.

Terminal window
curl http://localhost:4242/v1/messages \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen3",
"system": "You are terse.",
"messages": [{"role": "user", "content": "Name three prime numbers."}],
"max_tokens": 64,
"stream": true
}'
import anthropic
client = anthropic.Anthropic(base_url="http://localhost:4242", api_key="unused")
msg = client.messages.create(
model="qwen3",
max_tokens=256,
messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
print(msg.content[0].text)

system may be a string or a list of text blocks, and so may content. Supported sampling fields are temperature, top_p, top_k and stop_sequences. A stream is the named event sequence (message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop), and stop_reason is end_turn, max_tokens, stop_sequence or tool_use.

On a model converted with its vision projector (jitllm convert model.gguf mmproj.gguf out.jlm), a chat request may carry pictures: image_url content parts on /v1/chat/completions, image blocks on /v1/messages, and images on a Connect ChatMessage. PNG, JPEG, WebP, GIF (its first frame), BMP and TIFF are taken, inline as base64:

{"model": "smolvlm", "messages": [{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}},
{"type": "text", "text": "What is in this picture?"}]}]}

The pictures keep the order you sent them and go where the model’s own chat template puts its image markers, and the model’s vision tower encodes each one as jitllm run -image does. They combine with tools and a forced tool_choice, with n (the prompt, picture included, is prefilled once), and with a continued session (the history keeps its earlier pictures).

One picture may be 20 MiB and 4096x4096 pixels, and a request may carry 8 (-image-max, -image-max-pixels, -image-max-count). Inline pictures are always taken. A picture named by an http(s) URL is fetched only from the hosts listed in -image-fetch-allow, matched exactly (or one label under a *.example.com entry) on the first URL and every redirect, over https on ports 443 or 80 unless -image-fetch-http and -image-fetch-ports say otherwise, with no userinfo, no downgrade to http, at most three redirects, and never to a private, loopback, link-local, NAT64, IPv4-mapped or other special-use address, checked as each connection is made. A picture sent to a model without a vision tower, an unsupported, mismatched, oversized or undecodable picture, or a remote URL with fetching off is a 400 that names the part, never a reply from a model that did not see it. A request with pictures prefills alone rather than joining the step loop.

Both APIs accept tool definitions in their own shapes: OpenAI’s tools with function.parameters, Anthropic’s tools with input_schema. They reach the model’s own chat template, so a model trained for tool use sees the prompt it was trained on: OpenAI definitions exactly as sent, Anthropic ones converted to the OpenAI shape with input_schema kept byte for byte.

Terminal window
curl http://localhost:4242/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen3",
"messages": [{"role": "user", "content": "What is the weather in Paris?"}],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Current weather for a city",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
}
}]
}'

The model’s output is parsed back into tool_calls (OpenAI) or tool_use blocks (Anthropic). While streaming, ordinary text streams as it arrives; text that is, or may be becoming, a call is held back and emitted as a call once it parses. Send the tool’s result back as a tool message (OpenAI) or a tool_result block (Anthropic) and the conversation continues.

tool_choice: "none" (Anthropic: {"type": "none"}) withholds the tools, and "auto" (the default) lets the model call one or answer. "required" and a named function (Anthropic: {"type": "any"} and {"type": "tool", "name": ...}) force a call: the reply is sampled under a grammar that admits only a call to a declared tool, or to the named one. parallel_tool_calls: false (Anthropic: disable_parallel_tool_use) limits the reply to one call.

Two call formats are parsed, which cover the models trained for tools: Hermes-style <tool_call>{...}</tool_call> blocks (Qwen2.5, Qwen3 and most fine-tunes), and a bare JSON call opening the reply (Llama 3.1 and 3.2, Mistral v0.3). Only a call to a tool you declared counts, so a model asked to answer in JSON does not have its answer taken for a call.

Every response carries one extra field, jitllm, which clients that do not know it ignore. It is in the body when not streaming, and on the final chunk (OpenAI) or the message_delta event (Anthropic) when streaming:

"jitllm": {
"session_id": "s-1727614512345-1",
"queued_ms": 0,
"queue_depth_on_entry": 0,
"device_blocks": 36,
"cpu_blocks": 0,
"device_ids": ["cuda:0"],
"prefill_ms": 41,
"decode_ms": 1204,
"decode_tokens_per_second": 104.6,
"bytes_per_token": 816010912
}

It says where the request ran and how long it waited, which an OpenAI response has nowhere else to put.

A request may also name a session instead of a model with "jitllm_session" and the id jitllmd sessions -new printed, to run on a session you created with its own devices and KV capacity. A session named that way takes precedence over model.

jitllmd is also a client of a running server, through the same Connect API. Add -addr host:port to reach one that is not local.

Terminal window
jitllmd models # the model directory, and what is loaded
jitllmd models -load gemma-3-4b.jlm -id gemma -devices cuda:0
jitllmd models -unload gemma # -force closes sessions holding it instead of refusing
jitllmd devices # hardware and spendable memory
jitllmd stats -watch # engine counters, streamed
jitllmd sessions -new -model qwen3 -devices cuda:0 -max-seq 8192
jitllmd sessions # list them
jitllmd sessions -reset SESSION # back to position 0
jitllmd place -session SESSION # where each block is
jitllmd place -session SESSION -devices cuda:0 -gpu-layers 22
jitllmd place -session SESSION -move 20:35 -to host
jitllmd place -model qwen3 -maxmem 12G # re-budget the pager
jitllmd run -model qwen3 -chat -n 64 "Hello" # an ephemeral session
jitllmd run -session SESSION -chat "Hello" # keep the session's history
jitllmd run -session SESSION -continue "and then?" # continue from its position

SESSION is the id sessions -new printed. place -devices must name devices the model was loaded with, and place with no -devices brings every block home to the CPU. A move waits for the session’s current generation to finish, and the conversation keeps its history: the KV cache and any recurrent state move with the blocks. jitllmd <verb> -h lists each verb’s flags; the CLI reference has them all.

GET /v1/capabilities returns the same manifest as jitllm capabilities -json, for the running build: the architectures and quantizations it converts and runs, each backend and the hardware it is verified on, every route with its request fields and their limits, and the combinations the server refuses rather than ignores, each with the message it returns. An agent or a client can read it before sending anything. On the machine itself, jitllm doctor says whether this build can run here, and what to do about anything missing.

Everything the client does is a Connect RPC, generated from the protobuf definitions in proto/jitllm/v1. Any Connect, gRPC or gRPC-Web client can call it, over HTTP/1.1 or cleartext HTTP/2.

Service Methods
ModelService ListModels, GetModel, LoadModel, UnloadModel, Convert, Tokenize, Detokenize, ApplyChatTemplate
DeviceService ListDevices, GetDevice, GetMemoryTopology
SessionService CreateSession, GetSession, ListSessions, CloseSession, ResetSession, GetDeviceQueue
PlacementService GetPlacement, SetPlacement, RelocateBlocks, SetRelocation, TuneSeam, GetResidency, SetPageBudget
InferenceService Generate (streaming), Complete, Cancel, Embed
TelemetryService GetStats, WatchStats (streaming), GetServerInfo
Terminal window
curl http://localhost:4242/jitllm.v1.DeviceService/ListDevices \
-H 'Content-Type: application/json' -d '{}'