---
slug: 52-llm-provider-native
title: Provider config
description: Configure the LLM provider for Synsema — Anthropic, OpenAI, MiniMax, DeepSeek, any OpenAI-compatible local server, or an embedded GGUF running inside the process — via .env, with the key never exposed to the program.
example_ids: []
---

# Provider config

The provider is selected and configured by the **runtime**, not the program — the `.syn` never names a host or key, so it can't redirect the call or leak the key. Each knob resolves **process environ > `.env` > default**.

## `.env` (gitignored)

```bash
SYNSEMA_LLM_PROVIDER=deepseek
DEEPSEEK_API_KEY=sk-...
```

The key reaches the runtime **without** entering the process environment — no child process or program can read it (the program would need `require secret`, and even then sees it redacted). No `export`/`source` needed.

## Knobs

| Env var / `.env` entry | Purpose | Default |
|---|---|---|
| `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` / `MINIMAX_API_KEY` / `DEEPSEEK_API_KEY` | API key; presence auto-selects the provider | — (offline if absent) |
| `SYNSEMA_LLM_PROVIDER` | Force `anthropic` / `openai` / `minimax` / `deepseek` | auto |
| `SYNSEMA_LLM_MODEL` | Model id | `claude-sonnet-4-6` / `gpt-4o` / `MiniMax-M3` / `deepseek-chat` |
| `SYNSEMA_LLM_MAX_TOKENS` | Output cap | `4096` |
| `SYNSEMA_LLM_BASE_URL` | Endpoint base (point at a local/compatible server) | official |
| `SYNSEMA_LLM_TIMEOUT` | HTTP timeout in seconds for the network providers. With the default streaming transport it measures **silence between bytes** — each chunk renews it — so a minutes-long generation flows and a dead host still fails fast. Invalid/≤0 → default | `60` |
| `SYNSEMA_LLM_HTTP_STREAM` | Internal SSE transport for the network providers (the language ops still return complete text; `llm_stream` emits real chunks). `0`/`false` → classic non-stream path, escape hatch for odd proxies | `1` (on) |
| `SYNSEMA_LLM_BUDGET` | Per-**process** LLM token budget (input + output, all ops). At the ceiling, every op degrades to the marker `[llm budget exceeded: used N of M tokens]` — no error, no network call, one stderr notice. Consumption is readable with `llm_usage()` (see [LLM primitives](/en/0.6.x/50-llm-primitives)). Invalid/`0` → no budget, with a warning | — (unlimited) |

You can also force the provider per-run: `synsema run app.syn --provider anthropic` (flag > env > `.env`).

> **Timeouts no longer cap long generations.** The network providers request the API response as
> a stream and reassemble it internally, so the read-timeout detects *dead connections* (60s of
> real silence) instead of limiting total generation time. Verified live: a 177s / 68 KB
> generation (MiniMax-M3, a reasoning model) completes where the old fixed 60s window killed it.
> With `SYNSEMA_LLM_HTTP_STREAM=0` the timeout caps the whole call, as before.

## Local / on-prem (100% private)

Any OpenAI-compatible server (Ollama, LM Studio, vLLM):

```bash
SYNSEMA_LLM_PROVIDER=openai
SYNSEMA_LLM_BASE_URL=http://localhost:11434/v1   # Ollama
SYNSEMA_LLM_MODEL=llama3.1
OPENAI_API_KEY=ollama                            # any non-empty value
```

## Embedded local provider (`local`) — GGUF in-process, zero network

With a binary compiled with the `llm-local` feature, the runtime runs a quantized GGUF **inside the Synsema process** (candle, CPU): no server, no API key, no socket at all — the only provider that works under a total `deny net`. Always explicit, never auto-selected:

```bash
SYNSEMA_LLM_PROVIDER=local
SYNSEMA_LLM_MODEL=/models/qwen2.5-0.5b-instruct-q4_k_m.gguf   # path to the .gguf (required)
```

| Knob | Purpose | Default |
|---|---|---|
| `SYNSEMA_LLM_CTX` | Context window (capped to the GGUF's own limit) | `4096` |
| `SYNSEMA_LLM_THREADS` | CPU threads for inference | all cores |
| `SYNSEMA_LLM_TEMPERATURE` | `0` = greedy/deterministic; `>0` = sampling (fixed seed) | `0` |
| `SYNSEMA_LLM_MAX_CONCURRENT` | Max model instances; `1` serializes concurrent calls under `serve` | `1` |
| `SYNSEMA_LLM_STREAM_BUFFER` | Chunks in flight between generation and `llm_stream` emission | `32` |

`SYNSEMA_LLM_MAX_TOKENS` applies as usual. Supported architectures: **llama, qwen2, qwen3** — this is the `general.architecture` string in the GGUF header, not the brand name; anything else fails with a clear `[local error: …]`. Popular families convert to GGUF declaring one of those three, so real coverage is wider than the three names suggest. Families verified live (probe = load + question + coherent answer + clean stop):

| Model (GGUF probed) | declares | Verified |
|---|---|---|
| Qwen2.5 Instruct 0.5B/3B | `qwen2` | ✅ |
| Qwen3 0.6B | `qwen3` | ✅ thinking model — emits raw `<think>…</think>`; budget `SYNSEMA_LLM_MAX_TOKENS` for it |
| Qwen3 4B **Instruct**-2507 | `qwen3` | ✅ (2.5 GB GGUF, ran in 8 GB RAM). The 4B **Thinking**-2507 also runs but thinks for thousands of tokens per answer — impractical on CPU; prefer the Instruct variant |
| Mistral 7B Instruct v0.3 | `llama` | ✅ its `[INST]` template is auto-detected |
| Llama 3.2 1B Instruct | `llama` | ✅ llama3 template auto-detected |
| SmolLM2 135M Instruct | `llama` | ✅ chatml |
| DeepSeek-R1-Distill-Qwen 1.5B | `qwen2` | ✅ runs in plain mode; strip the `<think>` tags in your code |
| TinyLlama 1.1B Chat | `llama` | ⚠️ loads and runs, but its zephyr chat template isn't recognized → plain fallback, behaves like a base model |

A supported arch loads and runs; **chat usability also needs a recognized chat template** (chatml / llama3 / `[INST]`; otherwise plain fallback). gemma/phi/glm GGUFs are rejected on purpose: candle exposes no public KV-cache reset for them yet, and request isolation comes first.

Tip — models pulled with **ollama** are plain GGUF blobs (and its CDN is far faster than single-stream HF downloads): `ollama pull llama3.2:1b`, then point `SYNSEMA_LLM_MODEL` at the blob under `~/.ollama/models/blobs/sha256-…` (the digest is in the manifest under `~/.ollama/models/manifests/…`; ollama does not need to be running).

On a binary **without** the feature, `SYNSEMA_LLM_PROVIDER=local` prints a stderr notice and stays offline — it never silently falls back to another provider.

### Build it — and build it fast

```bash
# plain build (works everywhere, but candle's AVX2 kernels stay OFF — slow prefill):
cargo install --path crates/synsema-cli --features llm-local --force

# ~3× faster prefill: candle picks its AVX2 quantized kernels at COMPILE time,
# and Rust's default x86-64 target does not enable them:
RUSTFLAGS="-C target-cpu=native" cargo install --path crates/synsema-cli --features llm-local --force
```

Measured (Qwen2.5 Q4_K_M): generation ~13 tok/s (0.5B) / ~5 tok/s (3B); prefill with the flag ~35 tok/s (0.5B) — built for **short prompts** (hundreds of tokens, not thousands). Model load is paid **once per process** (~6s / ~19s): under `serve`, the first request loads, the rest reuse. RAM: ~1GB (0.5B) / ~2.4GB (3B). `native` ties the binary to that machine's CPU — right for your own VPS; for a binary you distribute, use `-C target-cpu=x86-64-v3` (AVX2+FMA, x86 CPUs from ~2015).

---

The egress to the configured host is part of `require llm` — **not** a separate `net` grant (the embedded provider needs no egress at all). Offline (no key), the ops return placeholders; branch on `llm_available()`. To bypass the built-in ops entirely and hit the API yourself, see **[Provider API directly](/en/0.6.x/53-llm-api-direct)**.