LLM

Provider config

The provider is selected and configured by the runtime, not the program — the .syn never names a host or key, so it can't redirect the call or leak the key. Each knob resolves process environ > .env > default.

.env (gitignored)§

SYNSEMA_LLM_PROVIDER=deepseek
DEEPSEEK_API_KEY=sk-...

The key reaches the runtime without entering the process environment — no child process or program can read it (the program would need require secret, and even then sees it redacted). No export/source needed.

Knobs§

You can also force the provider per-run: synsema run app.syn --provider anthropic (flag > env > .env).

The provider is what a --labels ceiling names (engine v0.6.46+). With labels on, private data reaches a model only through a ceiling item llm=<principal>@<host>, and the host is the configured provider's: api.anthropic.com, api.openai.com, api.deepseek.com, api.minimax.io, local for the embedded provider (and for an unconfigured one), or the host of SYNSEMA_LLM_BASE_URL without its port (llm=customers@localhost for the Ollama setup below). Changing these knobs changes which item the host has to write — see LLM primitives § Private data to a model.

Timeouts no longer cap long generations. The network providers request the API response as a stream and reassemble it internally, so the read-timeout detects dead connections (60s of real silence) instead of limiting total generation time. Verified live: a 177s / 68 KB generation (MiniMax-M3, a reasoning model) completes where the old fixed 60s window killed it. With SYNSEMA_LLM_HTTP_STREAM=0 the timeout caps the whole call, as before.

Local / on-prem (100% private)§

Any OpenAI-compatible server (Ollama, LM Studio, vLLM):

SYNSEMA_LLM_PROVIDER=openai
SYNSEMA_LLM_BASE_URL=http://localhost:11434/v1   # Ollama
SYNSEMA_LLM_MODEL=llama3.1
OPENAI_API_KEY=ollama                            # any non-empty value

Embedded local provider (local) — a model in-process, zero network§

The official binaries ship it: the runtime runs a quantized GGUF inside the Synsema process, with no server, no API key and no socket at all — the only provider that works under a total deny net. Always explicit, never auto-selected.

Since v0.6.27, SYNSEMA_LLM_MODEL takes three forms, and none of them downloads anything:

SYNSEMA_LLM_PROVIDER=local

SYNSEMA_LLM_MODEL=/models/qwen2.5-3b-instruct-q4_k_m.gguf   # 1. a path to a .gguf
SYNSEMA_LLM_MODEL=qwen3:0.6b                                # 2. a model:tag already in the Ollama cache
SYNSEMA_LLM_MODEL=org/repo                                  # 3. a repo already in the Hugging Face cache

If you already use Ollama, you are done — nothing to download, nothing to convert. Ollama stores plain GGUF blobs in a content-addressed store, so Synsema reads the file directly and gets the sha256 of the weights for free. ollama does not need to be running. A name that is not there lists the three places it looked, with real paths.

SYNSEMA_LLM_MAX_TOKENS applies as usual.

Which architectures run§

The architecture is the general.architecture string in the GGUF header, not the brand name. Which ones run depends on the engine — see Local inference and architectures for the difference and for how to add one yourself:

Anything else fails with a clear [local error: …] that names the engine which can run it. Popular families convert to GGUF declaring one of those names, so real coverage is wider than the list suggests. Families verified live (probe = load + question + coherent answer + clean stop):

A supported arch loads and runs; chat usability also needs a recognized chat template (chatml / llama3 / [INST] / gemma; otherwise plain fallback). phi and glm GGUFs are still rejected.

If you ran a llama, Mistral or Gemma GGUF before v0.6.27, its answers were wrong. Those files store token ranks, not log-probabilities, and the tokenizer was segmenting them as if they were log-probabilities: The capital of France is entered the model as eleven fragments instead of five words. Nothing failed — the model just answered badly. Fixed in v0.6.27. The BPE family (qwen, llama 3) was never affected.

What is actually configured — synsema llm status§

It prints the models already on disk and the architectures of the engine that will actually run, each with its origin and its sha. --json carries the same under an inference key, with the full sha, so a script can compare two runs instead of reading prose:

Modelos locales ya descargados (2):
  gemma3:270m       ollama:gemma3:270m  sha256:735af2139dc6
  qwen3:0.6b        ollama:qwen3:0.6b   sha256:7f4030143c1c
  Usá SYNSEMA_LLM_MODEL=<nombre> — no se descarga nada.

Arquitecturas que corre el backend `rust` (4):
  gemma3 (20 pasos por capa, en el binario, sha b935fcfab2d6)
  llama (16 pasos por capa, en el binario, sha a580fa45fdb6)
  qwen2 (19 pasos por capa, en el binario, sha fbba11641175)
  qwen3 (18 pasos por capa, ./archdefs/qwen3.archdef, sha 5b8d9db76a68)
  Definiciones del operador: ./archdefs

On a binary without the feature, SYNSEMA_LLM_PROVIDER=local prints a clear stderr notice and stays offline — it never silently falls back to another provider.

Honest limits, and building for speed§

Measured (Qwen2.5 Q4_K_M, no rebuild): generation ~13 tok/s (0.5B) / ~5 tok/s (3B); prefill ~12 tok/s — built for short prompts (hundreds of tokens, not thousands). Model load is paid once per process (~6s / ~19s): under serve, the first request loads and the rest reuse. RAM with candle: ~1GB (0.5B) / ~2.4GB (3B); with the rust engine, roughly the size of the .gguf (the file is memory-mapped and the big weights stay quantized).

Rebuilding for speed is only worth it on candle. candle picks its AVX2 quantized kernels at compile time (#[cfg(target_feature = "avx2")]), and Rust's default x86-64 target does not enable AVX2 — so a plain build runs the scalar path. Measured gain: prefill ~3.1× on a 0.5B (~11 → ~35 tok/s). The rust engine does not need this: it dispatches on the CPU it finds (AVX / AVX2+FMA / AVX-512 / NEON), so the binary you downloaded already uses your machine's instructions.

# only if you stay on candle:
RUSTFLAGS="-C target-cpu=native" cargo install --path crates/synsema-cli --features llm-local --force

native ties the binary to that machine's CPU — right for your own VPS; for a binary you distribute, use -C target-cpu=x86-64-v3 (AVX2+FMA, x86 CPUs from ~2015).

---

The egress to the configured host is part of require llm — not a separate net grant (the embedded provider needs no egress at all). Offline (no key), the ops return placeholders; branch on llm_available(). To bypass the built-in ops entirely and hit the API yourself, see Provider API directly.