Skip to content

CLI reference

Flags always go before the model path; anything after it is the prompt or text. Every subcommand prints its own flags with -h.

Every size, on jitllm and jitllmd alike, is a byte count or a number with a unit: 4294967296, 4G, 512M.

Command What it does
jitllm hardware the CPU tier, every GPU each backend can open, and the spendable memory
jitllm doctor [-json] [MODEL.jlm] whether this machine and build can run, and a container if given; exit 0 ok, 2 something missing (each with a next step), 1 the inspection failed. It downloads nothing and runs no kernel
jitllm capabilities [-json] what this build converts, runs and serves: architectures, projectors, quantization, backends and where each is verified, every API route and field, and the combinations refused
jitllm library [-refs] the catalog convert fetches by name; -refs prints sizes and hf:// references for scripts
jitllm convert [flags] SOURCE [MMPROJ] [OUT.jlm] write a container from a GGUF, a Hugging Face directory or URL, or a catalog name. See Convert models.
jitllm run [flags] MODEL.jlm [PROMPT...] generate; with no prompt, an interactive chat, or the prompt read from piped stdin
jitllm embed [-lines] [-ids] MODEL.jlm TEXT... print an embedding model’s normalised vector as JSON
jitllm decide [flags] MODEL.jlm REQUEST.json|- a decision model’s answers to a /v1/systemone request body, printed as the server would return them
jitllm speed [flags] MODEL.jlm prompt and generation rates, warmed, as llama-bench measures them; or time to first token, or sessions decoded together
jitllm batch [flags] MODEL.jlm PROMPT... decode many sequences together on one device and report the aggregate rate
jitllm verify [flags] MODEL.jlm run on the CPU and a device and report where their logits differ
jitllm bench [-tokenizer FILE] MODEL.jlm one-core decode on the generated kernels, as a fraction of a memory read rate measured in the same process
jitllm tokenize [-tokenizer FILE] MODEL TEXT... print token ids
jitllm info FILE.gguf a GGUF’s keys and tensors, before converting it
jitllm asm Q4_0|Q8_0|Q4_K[x4] disassemble the x86-64 kernel generated for a format on this machine
jitllm version the build’s version
Flag Default Meaning
-n N until the end tokens to generate; by default the model runs until it ends its reply or fills its context
-chat off apply the model’s chat template
-system TEXT a system message, with -chat
-image FILE an image for a vision model
-devices SPEC auto where to run; see the device grammar
-placement MAP where named blocks and the head run, with ! to pin and ~ to stream; see Placement maps
-placement-strict off fail when a -placement entry cannot be met, instead of placing that block elsewhere
-gpu-layers N -1 cap the blocks offered to devices; -1 fits as many as the card holds
-vram SIZE 0 device weight budget; 0 asks each device what is free and keeps an eighth of it, at least 256 MiB
-maxmem SIZE automatic host weight budget; see Memory and paging
-gpu-grow off start on the CPU and move blocks to the device while generating
-relocate on hand the device’s last block to the CPU when the context cannot grow on the device. On wherever -devices admits the host; -relocate=false turns it off
-tune-seam off measure whether fewer device blocks are faster, and move the seam
-no-gpu-fallback off fail instead of finishing on the CPU when the device breaks
-kv-cache DIR reuse KV pages of earlier prompts with a shared prefix, and spill history there under -kv-budget
-kv-cache-max SIZE 8 GiB cache size before least-recently-used eviction; 0 is unbounded
-kv-budget SIZE 0 KV history held in memory; older pages spill to -kv-cache and come back when read. 0 is unbounded
-depth N 0 pad the prompt to N tokens, to measure decode at that context length
-tokenizer FILE a tokenizer.json whose pre-tokenizer replaces the model’s
-spec MODE speculative decoding, verified in one pass: mtp drafts with the model’s own multi-token-prediction block, lookup with the tokens that followed the sequence’s last n-gram earlier in it (any model, greedy only), auto the first of the two the model can use; see Speculative decoding
-spec-k N 0 tokens drafted a round; 0 chooses each round from the measured acceptance and timing
-spec-min-p F 0 stop a round’s drafting at the first draft below this probability
-spec-rollback MODE auto how a hybrid’s recurrent state is taken back after a rejected draft: auto, rows or replay
-temp, -top-k, -top-p, -min-p, -repeat-penalty, -repeat-last-n, -seed greedy sampling, named as in llama.cpp; see Sampling
-cpuprofile FILE write a Go CPU profile
Flag Default Meaning
-devices SPEC auto as for run
-p N 512 prompt tokens
-n N 128 generated tokens
-r N 3 timed repetitions, after one untimed warm-up
-vram SIZE 0 device weight budget
-ttft off time to first token instead: cold (opening the model to the first token) and warm (a fresh session tokenizing, prefilling and sampling a -p-token prompt, -r times)
-sessions N decode N separate sessions -n tokens each, one after another and then together, interleaved over -r rounds
-gcstats off what the Go collector did during each timed phase: allocations a token, cycles, pauses, GC CPU
-cpuprofile FILE, -tgprofile FILE a CPU profile of the timed prompts, or of the timed decodes
-memprofile PREFIX allocation profiles around the first timed prompt and decode
-spec, -spec-k, -spec-min-p, -spec-rollback speculative decoding, as for run
Flag Default Meaning
-devices SPEC gpu:0 the one device every block and the head go on
-batch N 16 sequences decoded together
-n N 128 tokens per sequence
-logits off read every row’s logits back and pick on the host, the path a sampled request takes, instead of the device’s argmax
-cpuprofile FILE write a Go CPU profile
Flag Default Meaning
-devices SPEC auto the device to compare with the CPU
-n N 32 tokens
-dlogit F 0 the largest per-logit difference to accept; an argmax flip inside it counts as a tie. 0 does not check
-migrate off move the CPU/GPU seam during the comparison
-prefill N 0 also compare the batched prefill of N tokens
-verify off recompute every served matrix product on the host and report disagreements
-vram SIZE, -maxmem SIZE automatic device and host budgets
-ab ARM run a paired A/B of one engine choice instead (split, submit, head, graph, softmax, fuse, ropetab, attnchunk, attnpair), with -rounds N (default 21) and -depth N

convert, embed, decide, library, tokenize, capabilities, doctor

Section titled “convert, embed, decide, library, tokenize, capabilities, doctor”
Command Flag Meaning
convert -o DIR where the container, and a downloaded source, land. Without it and without OUT.jlm, the container goes into JITLLM_MODELS (or ./models), or beside its source when that directory does not exist
-q8 safetensors only: store every weight matrix as Q8_0
-chat-template FILE store this file’s chat template(s) in place of the source’s: a tokenizer_config.json, a chat_template.json, a .jinja file or a model directory
-plan safetensors only: read the shards’ headers (by range, for a repository) and print the container the conversion would write, its size, tensor count and anything it would refuse, without converting
embed -lines embed each line of stdin, loading the model once
-ids print the token ids to stderr
embed, decide -devices SPEC, -gpu-layers N as for run
decide -lines answer each line of stdin as one request, loading the model once
-max-seq N the longest prompt one question may render to, in tokens (default 8192)
-repeat N answer each request N times, to tell a cold call from a warm one
-cpuprofile FILE a CPU profile of the answering, not the load
library -refs name, size in bytes and hf:// references, one per line
tokenize, bench -tokenizer FILE as for run
capabilities -json the whole manifest as JSON (the same document as the repository’s docs/capabilities.json, with this build’s version)
doctor -json the report as JSON on stdout; logs go to stderr
Command What it does
jitllmd serve [flags] run the server
jitllmd run [flags] PROMPT... stream a generation from a running server
jitllmd models list, describe, load and unload models
jitllmd devices the server’s hardware and spendable memory
jitllmd sessions create, list, describe, reset and close sessions
jitllmd place read and move a session’s blocks, tune its seam, and re-budget a model’s pager
jitllmd stats engine counters, once or streamed

Every client command takes -addr (host:port, :port or a full URL; default localhost:4242). -api-key KEY (or JITLLM_API_KEY) sends the key a server started with -api-key wants. With the default address and nothing answering on it, a server on the old default port 127.0.0.1:8080 is used and the command says so once.

Flag Default Meaning
-addr localhost:4242 listen address. Loopback by default: nothing off this machine reaches it. An address with no host (:4242) or a non-loopback one opens it to the network, and serve warns unless -api-key is set
-api-key KEY JITLLM_API_KEY, else none require Authorization: Bearer KEY (and x-api-key: KEY on /v1/messages) on every route but /healthz
-allow-origin LIST none comma-separated browser origins that may call the server cross-origin. Loopback pages and clients that send no Origin are unaffected; any other origin is refused
-models DIR JITLLM_MODELS, else models directory scanned by the model list and used to resolve a bare model name
-load FILE a .jlm to load at startup, absolute or relative to -models
-id NAME derived the model name clients send
-devices SPEC CPU only devices for -load, in the device grammar
-maxmem SIZE engine’s own host page budget for -load, e.g. 8G
-gpu-layers N -1 at most N blocks on a device; -1 is as many as fit
-sessions N 1 concurrent sessions each linear device block reserves a recurrent state for; attention history is paged and reserves nothing
-max-seq N model’s context default KV capacity per session, in positions
-max-batch N 0 requests of one model (on one device, or wholly on the CPU) that decode as rows of one step; 0 is the engine’s bound, 1 turns batching off
-prompt-chunk N 0 prompt tokens a joining request feeds into one shared step; 0 is one device prefill chunk
-step-prompt-tokens N 0 prompt tokens one step carries beside decoding rows; 0 measures the tokens that cost one decode step’s time
-step-cost X 0 with -step-prompt-tokens 0, how many decode steps’ time a step carrying prompt tokens may take
-fairness N 100 0 to 100: 0 serves requests first come first served, each to its end (the most tokens a second); higher time-slices running rows for waiting ones and feeds the next prompts beside the oldest (shorter, evener waits for a first token)
-joint-steps MODE auto auto times a joint step against the rows one after another and runs the faster; always or never
-warm true after every load, run a short prefill and a decode step so the first request does not pay for them
-no-mem-cache false turn off each model’s memory cache: every request prefills its whole prompt
-mem-cache-max SIZE an eighth of the model’s host share bound each model’s memory cache
-host-mem SIZE eight tenths of what the host has host memory the loaded models divide
-session-pool N 0 reset sessions each model keeps for requests; 0 is -sessions, -1 none
-max-queue N 0 requests one model holds, running or waiting, before it answers 429; 0 is 64, -1 unbounded
-retry-after D 1s the Retry-After a 429 for a full queue names
-kv-f16 engine’s own the KV cache width of every load that names none: -kv-f16 binary16, -kv-f16=false f32
-kv-cache DIR where sessions created with prompt_cache keep their prompt prefixes; without it such a session is refused
-kv-cache-max SIZE 8G what -kv-cache may occupy before its least recently used pages go; 0 is unbounded
-image-max SIZE 20M the most encoded bytes one picture in a request may be
-image-max-pixels N 16777216 the most pixels (width x height) one picture may be, read from its header before decoding
-image-max-count N 8 the most pictures one request may carry
-image-max-request SIZE 64M the most encoded bytes all of a request’s pictures may be
-image-decode-timeout D 5s how long decoding one picture may take
-image-fetch-allow LIST comma-separated hosts a picture named by an http(s) URL may be fetched from: exact names, or *.example.com for one label under it. Empty fetches nothing; inline base64 pictures are always taken
-image-fetch-http false let a fetch use plain http (https only otherwise)
-image-fetch-ports LIST 443,80 ports a fetch may connect to
-image-fetch-concurrency N 4 remote fetches running at once across the server
-image-fetch-timeout D 10s how long one remote fetch may take
-version S dev the version GetServerInfo reports
Flag Meaning
-model ID generate in an ephemeral session on this model
-session ID generate in an existing session, keeping its history
-continue continue from the session’s position instead of prefilling from scratch
-chat, -system TEXT apply the model’s chat template, with a system message
-n N tokens to generate (default: until the model ends its reply or fills the session’s context)
-ignore-eos keep generating past an end-of-generation token, up to -n or the end of the context; stop strings still stop
-stop A,B stop strings
-temp, -top-k, -top-p, -min-p, -repeat-penalty, -repeat-last-n, -seed sampling, as for jitllm run
-queue-timeout MS how long to wait for a device another session holds; 0 waits indefinitely
-echo have the server send the prompt’s tokens back first
-ids print the generated token ids to stderr
-quiet no started/finished lines on stderr
Flag Meaning
(none) list the model directory and what is loaded; -loaded lists only loaded models, -dir scans another directory
-load FILE load a container, with -id, -devices, -gpu-layers, -maxmem as for serve, and -kv-f16 to force an f16 KV cache
-unload ID unload a model; -force closes sessions holding it instead of refusing
Flag Meaning
(none) list sessions; -model filters
-new create a session on -model, with -id, -devices, -gpu-layers, -max-seq, -gpu-grow, -keep-off-host, -prompt-cache and -cache-key
-reset ID back to position 0
-close ID close it
-queue DEVICE the queue waiting on a device
Flag Meaning
-session ID report where each of the session’s blocks runs
-devices SPEC re-place the session’s blocks on these devices (they must be ones the model was loaded with); empty brings every block home
-gpu-layers N at most N blocks on a device
-head-on-device place the output projection on a device too
-move FIRST:LAST -to WHERE relocate a block range to host or a device
-relocate adopt one block per token as device memory frees
-tune-seam measure whether fewer device blocks are faster and move there, with -tokens, -rounds and -warmup
-model ID report the model’s pager residency; with -maxmem SIZE, re-budget it
Flag Meaning
-watch stream a snapshot every -interval milliseconds (default 1000)
-model ID, -session ID only this model or session

Read by jitllm. jitllmd reads only JITLLM_MODELS (the default of -models) and, to find ROCm, JITLLM_ROCM and on Windows HIP_PATH; the Go packages read none.

Variable Effect
JITLLM_MODELS default directory for downloaded models and converted containers
JITLLM_ROCM the ROCm library directory for the native AMD backend (on Windows, HIP_PATH’s bin is the default)
HF_TOKEN token for private or gated Hugging Face repositories (HUGGING_FACE_HUB_TOKEN and ~/.cache/huggingface/token also work)
JITLLM_GPU_STREAM=1 place mixture blocks without their expert banks and upload only the chosen experts each token
JITLLM_ARENA=BYTES bound the host copy that streamed blocks are uploaded from; unset, a streaming placement sizes it
JITLLM_KV_F16=0 or 1 force the KV cache’s width instead of the per-CPU-tier default (f16 on AVX2, f32 elsewhere)
JITLLM_CORESET, JITLLM_CORES=N which cores decode uses (p by default) and how many of them
JITLLM_NUMA=off leave weight memory to the kernel’s first-touch policy instead of interleaving across NUMA nodes
JITLLM_HUGEPAGES=0 keep weight memory on the kernel’s default page size
JITLLM_OFFHEAP=0 put weight memory back on the Go heap (a control for measurements)
GOMAXPROCS when set, overrides the thread count jitllm aligns to its decode cores

The rest of the JITLLM_* names cmd/jitllm reads are measurement switches for engine development. The Go packages take options instead; see Embed it in Go.