CLI reference
Flags always go before the model path; anything after it is the prompt or text. Every subcommand prints its own flags with -h.
Every size, on jitllm and jitllmd alike, is a byte count or a number with a unit: 4294967296, 4G, 512M.
jitllm
Section titled “jitllm”| Command | What it does |
|---|---|
jitllm hardware |
the CPU tier, every GPU each backend can open, and the spendable memory |
jitllm doctor [-json] [MODEL.jlm] |
whether this machine and build can run, and a container if given; exit 0 ok, 2 something missing (each with a next step), 1 the inspection failed. It downloads nothing and runs no kernel |
jitllm capabilities [-json] |
what this build converts, runs and serves: architectures, projectors, quantization, backends and where each is verified, every API route and field, and the combinations refused |
jitllm library [-refs] |
the catalog convert fetches by name; -refs prints sizes and hf:// references for scripts |
jitllm convert [flags] SOURCE [MMPROJ] [OUT.jlm] |
write a container from a GGUF, a Hugging Face directory or URL, or a catalog name. See Convert models. |
jitllm run [flags] MODEL.jlm [PROMPT...] |
generate; with no prompt, an interactive chat, or the prompt read from piped stdin |
jitllm embed [-lines] [-ids] MODEL.jlm TEXT... |
print an embedding model’s normalised vector as JSON |
jitllm decide [flags] MODEL.jlm REQUEST.json|- |
a decision model’s answers to a /v1/systemone request body, printed as the server would return them |
jitllm speed [flags] MODEL.jlm |
prompt and generation rates, warmed, as llama-bench measures them; or time to first token, or sessions decoded together |
jitllm batch [flags] MODEL.jlm PROMPT... |
decode many sequences together on one device and report the aggregate rate |
jitllm verify [flags] MODEL.jlm |
run on the CPU and a device and report where their logits differ |
jitllm bench [-tokenizer FILE] MODEL.jlm |
one-core decode on the generated kernels, as a fraction of a memory read rate measured in the same process |
jitllm tokenize [-tokenizer FILE] MODEL TEXT... |
print token ids |
jitllm info FILE.gguf |
a GGUF’s keys and tensors, before converting it |
jitllm asm Q4_0|Q8_0|Q4_K[x4] |
disassemble the x86-64 kernel generated for a format on this machine |
jitllm version |
the build’s version |
| Flag | Default | Meaning |
|---|---|---|
-n N |
until the end | tokens to generate; by default the model runs until it ends its reply or fills its context |
-chat |
off | apply the model’s chat template |
-system TEXT |
a system message, with -chat |
|
-image FILE |
an image for a vision model | |
-devices SPEC |
auto |
where to run; see the device grammar |
-placement MAP |
where named blocks and the head run, with ! to pin and ~ to stream; see Placement maps |
|
-placement-strict |
off | fail when a -placement entry cannot be met, instead of placing that block elsewhere |
-gpu-layers N |
-1 | cap the blocks offered to devices; -1 fits as many as the card holds |
-vram SIZE |
0 | device weight budget; 0 asks each device what is free and keeps an eighth of it, at least 256 MiB |
-maxmem SIZE |
automatic | host weight budget; see Memory and paging |
-gpu-grow |
off | start on the CPU and move blocks to the device while generating |
-relocate |
on | hand the device’s last block to the CPU when the context cannot grow on the device. On wherever -devices admits the host; -relocate=false turns it off |
-tune-seam |
off | measure whether fewer device blocks are faster, and move the seam |
-no-gpu-fallback |
off | fail instead of finishing on the CPU when the device breaks |
-kv-cache DIR |
reuse KV pages of earlier prompts with a shared prefix, and spill history there under -kv-budget |
|
-kv-cache-max SIZE |
8 GiB | cache size before least-recently-used eviction; 0 is unbounded |
-kv-budget SIZE |
0 | KV history held in memory; older pages spill to -kv-cache and come back when read. 0 is unbounded |
-depth N |
0 | pad the prompt to N tokens, to measure decode at that context length |
-tokenizer FILE |
a tokenizer.json whose pre-tokenizer replaces the model’s |
|
-spec MODE |
speculative decoding, verified in one pass: mtp drafts with the model’s own multi-token-prediction block, lookup with the tokens that followed the sequence’s last n-gram earlier in it (any model, greedy only), auto the first of the two the model can use; see Speculative decoding |
|
-spec-k N |
0 | tokens drafted a round; 0 chooses each round from the measured acceptance and timing |
-spec-min-p F |
0 | stop a round’s drafting at the first draft below this probability |
-spec-rollback MODE |
auto |
how a hybrid’s recurrent state is taken back after a rejected draft: auto, rows or replay |
-temp, -top-k, -top-p, -min-p, -repeat-penalty, -repeat-last-n, -seed |
greedy | sampling, named as in llama.cpp; see Sampling |
-cpuprofile FILE |
write a Go CPU profile |
| Flag | Default | Meaning |
|---|---|---|
-devices SPEC |
auto |
as for run |
-p N |
512 | prompt tokens |
-n N |
128 | generated tokens |
-r N |
3 | timed repetitions, after one untimed warm-up |
-vram SIZE |
0 | device weight budget |
-ttft |
off | time to first token instead: cold (opening the model to the first token) and warm (a fresh session tokenizing, prefilling and sampling a -p-token prompt, -r times) |
-sessions N |
decode N separate sessions -n tokens each, one after another and then together, interleaved over -r rounds |
|
-gcstats |
off | what the Go collector did during each timed phase: allocations a token, cycles, pauses, GC CPU |
-cpuprofile FILE, -tgprofile FILE |
a CPU profile of the timed prompts, or of the timed decodes | |
-memprofile PREFIX |
allocation profiles around the first timed prompt and decode | |
-spec, -spec-k, -spec-min-p, -spec-rollback |
speculative decoding, as for run |
| Flag | Default | Meaning |
|---|---|---|
-devices SPEC |
gpu:0 |
the one device every block and the head go on |
-batch N |
16 | sequences decoded together |
-n N |
128 | tokens per sequence |
-logits |
off | read every row’s logits back and pick on the host, the path a sampled request takes, instead of the device’s argmax |
-cpuprofile FILE |
write a Go CPU profile |
verify
Section titled “verify”| Flag | Default | Meaning |
|---|---|---|
-devices SPEC |
auto |
the device to compare with the CPU |
-n N |
32 | tokens |
-dlogit F |
0 | the largest per-logit difference to accept; an argmax flip inside it counts as a tie. 0 does not check |
-migrate |
off | move the CPU/GPU seam during the comparison |
-prefill N |
0 | also compare the batched prefill of N tokens |
-verify |
off | recompute every served matrix product on the host and report disagreements |
-vram SIZE, -maxmem SIZE |
automatic | device and host budgets |
-ab ARM |
run a paired A/B of one engine choice instead (split, submit, head, graph, softmax, fuse, ropetab, attnchunk, attnpair), with -rounds N (default 21) and -depth N |
convert, embed, decide, library, tokenize, capabilities, doctor
Section titled “convert, embed, decide, library, tokenize, capabilities, doctor”| Command | Flag | Meaning |
|---|---|---|
convert |
-o DIR |
where the container, and a downloaded source, land. Without it and without OUT.jlm, the container goes into JITLLM_MODELS (or ./models), or beside its source when that directory does not exist |
-q8 |
safetensors only: store every weight matrix as Q8_0 | |
-chat-template FILE |
store this file’s chat template(s) in place of the source’s: a tokenizer_config.json, a chat_template.json, a .jinja file or a model directory |
|
-plan |
safetensors only: read the shards’ headers (by range, for a repository) and print the container the conversion would write, its size, tensor count and anything it would refuse, without converting | |
embed |
-lines |
embed each line of stdin, loading the model once |
-ids |
print the token ids to stderr | |
embed, decide |
-devices SPEC, -gpu-layers N |
as for run |
decide |
-lines |
answer each line of stdin as one request, loading the model once |
-max-seq N |
the longest prompt one question may render to, in tokens (default 8192) | |
-repeat N |
answer each request N times, to tell a cold call from a warm one | |
-cpuprofile FILE |
a CPU profile of the answering, not the load | |
library |
-refs |
name, size in bytes and hf:// references, one per line |
tokenize, bench |
-tokenizer FILE |
as for run |
capabilities |
-json |
the whole manifest as JSON (the same document as the repository’s docs/capabilities.json, with this build’s version) |
doctor |
-json |
the report as JSON on stdout; logs go to stderr |
jitllmd
Section titled “jitllmd”| Command | What it does |
|---|---|
jitllmd serve [flags] |
run the server |
jitllmd run [flags] PROMPT... |
stream a generation from a running server |
jitllmd models |
list, describe, load and unload models |
jitllmd devices |
the server’s hardware and spendable memory |
jitllmd sessions |
create, list, describe, reset and close sessions |
jitllmd place |
read and move a session’s blocks, tune its seam, and re-budget a model’s pager |
jitllmd stats |
engine counters, once or streamed |
Every client command takes -addr (host:port, :port or a full URL; default localhost:4242). -api-key KEY (or JITLLM_API_KEY) sends the key a server started with -api-key wants. With the default address and nothing answering on it, a server on the old default port 127.0.0.1:8080 is used and the command says so once.
| Flag | Default | Meaning |
|---|---|---|
-addr |
localhost:4242 |
listen address. Loopback by default: nothing off this machine reaches it. An address with no host (:4242) or a non-loopback one opens it to the network, and serve warns unless -api-key is set |
-api-key KEY |
JITLLM_API_KEY, else none |
require Authorization: Bearer KEY (and x-api-key: KEY on /v1/messages) on every route but /healthz |
-allow-origin LIST |
none | comma-separated browser origins that may call the server cross-origin. Loopback pages and clients that send no Origin are unaffected; any other origin is refused |
-models DIR |
JITLLM_MODELS, else models |
directory scanned by the model list and used to resolve a bare model name |
-load FILE |
a .jlm to load at startup, absolute or relative to -models |
|
-id NAME |
derived | the model name clients send |
-devices SPEC |
CPU only | devices for -load, in the device grammar |
-maxmem SIZE |
engine’s own | host page budget for -load, e.g. 8G |
-gpu-layers N |
-1 | at most N blocks on a device; -1 is as many as fit |
-sessions N |
1 | concurrent sessions each linear device block reserves a recurrent state for; attention history is paged and reserves nothing |
-max-seq N |
model’s context | default KV capacity per session, in positions |
-max-batch N |
0 | requests of one model (on one device, or wholly on the CPU) that decode as rows of one step; 0 is the engine’s bound, 1 turns batching off |
-prompt-chunk N |
0 | prompt tokens a joining request feeds into one shared step; 0 is one device prefill chunk |
-step-prompt-tokens N |
0 | prompt tokens one step carries beside decoding rows; 0 measures the tokens that cost one decode step’s time |
-step-cost X |
0 | with -step-prompt-tokens 0, how many decode steps’ time a step carrying prompt tokens may take |
-fairness N |
100 | 0 to 100: 0 serves requests first come first served, each to its end (the most tokens a second); higher time-slices running rows for waiting ones and feeds the next prompts beside the oldest (shorter, evener waits for a first token) |
-joint-steps MODE |
auto |
auto times a joint step against the rows one after another and runs the faster; always or never |
-warm |
true | after every load, run a short prefill and a decode step so the first request does not pay for them |
-no-mem-cache |
false | turn off each model’s memory cache: every request prefills its whole prompt |
-mem-cache-max SIZE |
an eighth of the model’s host share | bound each model’s memory cache |
-host-mem SIZE |
eight tenths of what the host has | host memory the loaded models divide |
-session-pool N |
0 | reset sessions each model keeps for requests; 0 is -sessions, -1 none |
-max-queue N |
0 | requests one model holds, running or waiting, before it answers 429; 0 is 64, -1 unbounded |
-retry-after D |
1s |
the Retry-After a 429 for a full queue names |
-kv-f16 |
engine’s own | the KV cache width of every load that names none: -kv-f16 binary16, -kv-f16=false f32 |
-kv-cache DIR |
where sessions created with prompt_cache keep their prompt prefixes; without it such a session is refused |
|
-kv-cache-max SIZE |
8G |
what -kv-cache may occupy before its least recently used pages go; 0 is unbounded |
-image-max SIZE |
20M |
the most encoded bytes one picture in a request may be |
-image-max-pixels N |
16777216 | the most pixels (width x height) one picture may be, read from its header before decoding |
-image-max-count N |
8 | the most pictures one request may carry |
-image-max-request SIZE |
64M |
the most encoded bytes all of a request’s pictures may be |
-image-decode-timeout D |
5s |
how long decoding one picture may take |
-image-fetch-allow LIST |
comma-separated hosts a picture named by an http(s) URL may be fetched from: exact names, or *.example.com for one label under it. Empty fetches nothing; inline base64 pictures are always taken |
|
-image-fetch-http |
false | let a fetch use plain http (https only otherwise) |
-image-fetch-ports LIST |
443,80 |
ports a fetch may connect to |
-image-fetch-concurrency N |
4 | remote fetches running at once across the server |
-image-fetch-timeout D |
10s |
how long one remote fetch may take |
-version S |
dev |
the version GetServerInfo reports |
| Flag | Meaning |
|---|---|
-model ID |
generate in an ephemeral session on this model |
-session ID |
generate in an existing session, keeping its history |
-continue |
continue from the session’s position instead of prefilling from scratch |
-chat, -system TEXT |
apply the model’s chat template, with a system message |
-n N |
tokens to generate (default: until the model ends its reply or fills the session’s context) |
-ignore-eos |
keep generating past an end-of-generation token, up to -n or the end of the context; stop strings still stop |
-stop A,B |
stop strings |
-temp, -top-k, -top-p, -min-p, -repeat-penalty, -repeat-last-n, -seed |
sampling, as for jitllm run |
-queue-timeout MS |
how long to wait for a device another session holds; 0 waits indefinitely |
-echo |
have the server send the prompt’s tokens back first |
-ids |
print the generated token ids to stderr |
-quiet |
no started/finished lines on stderr |
models
Section titled “models”| Flag | Meaning |
|---|---|
| (none) | list the model directory and what is loaded; -loaded lists only loaded models, -dir scans another directory |
-load FILE |
load a container, with -id, -devices, -gpu-layers, -maxmem as for serve, and -kv-f16 to force an f16 KV cache |
-unload ID |
unload a model; -force closes sessions holding it instead of refusing |
sessions
Section titled “sessions”| Flag | Meaning |
|---|---|
| (none) | list sessions; -model filters |
-new |
create a session on -model, with -id, -devices, -gpu-layers, -max-seq, -gpu-grow, -keep-off-host, -prompt-cache and -cache-key |
-reset ID |
back to position 0 |
-close ID |
close it |
-queue DEVICE |
the queue waiting on a device |
| Flag | Meaning |
|---|---|
-session ID |
report where each of the session’s blocks runs |
-devices SPEC |
re-place the session’s blocks on these devices (they must be ones the model was loaded with); empty brings every block home |
-gpu-layers N |
at most N blocks on a device |
-head-on-device |
place the output projection on a device too |
-move FIRST:LAST -to WHERE |
relocate a block range to host or a device |
-relocate |
adopt one block per token as device memory frees |
-tune-seam |
measure whether fewer device blocks are faster and move there, with -tokens, -rounds and -warmup |
-model ID |
report the model’s pager residency; with -maxmem SIZE, re-budget it |
| Flag | Meaning |
|---|---|
-watch |
stream a snapshot every -interval milliseconds (default 1000) |
-model ID, -session ID |
only this model or session |
Environment
Section titled “Environment”Read by jitllm. jitllmd reads only JITLLM_MODELS (the default of -models) and, to find ROCm, JITLLM_ROCM and on Windows HIP_PATH; the Go packages read none.
| Variable | Effect |
|---|---|
JITLLM_MODELS |
default directory for downloaded models and converted containers |
JITLLM_ROCM |
the ROCm library directory for the native AMD backend (on Windows, HIP_PATH’s bin is the default) |
HF_TOKEN |
token for private or gated Hugging Face repositories (HUGGING_FACE_HUB_TOKEN and ~/.cache/huggingface/token also work) |
JITLLM_GPU_STREAM=1 |
place mixture blocks without their expert banks and upload only the chosen experts each token |
JITLLM_ARENA=BYTES |
bound the host copy that streamed blocks are uploaded from; unset, a streaming placement sizes it |
JITLLM_KV_F16=0 or 1 |
force the KV cache’s width instead of the per-CPU-tier default (f16 on AVX2, f32 elsewhere) |
JITLLM_CORESET, JITLLM_CORES=N |
which cores decode uses (p by default) and how many of them |
JITLLM_NUMA=off |
leave weight memory to the kernel’s first-touch policy instead of interleaving across NUMA nodes |
JITLLM_HUGEPAGES=0 |
keep weight memory on the kernel’s default page size |
JITLLM_OFFHEAP=0 |
put weight memory back on the Go heap (a control for measurements) |
GOMAXPROCS |
when set, overrides the thread count jitllm aligns to its decode cores |
The rest of the JITLLM_* names cmd/jitllm reads are measurement switches for engine development. The Go packages take options instead; see Embed it in Go.