LLM

Local inference and architectures

The local LLM provider (Provider config) and the laya judge backend (Judge) run on Synsema's own inference layer. This page is about the part that is configuration, not code: which engine runs, which architectures it knows, and how to add one without recompiling anything.

Why an architecture is a file here§

Every runtime that runs GGUF models — llama.cpp, candle, Ollama — writes each architecture by hand, in its own language. Adding one means writing code, opening a pull request, waiting for review and waiting for a release. That is a human bottleneck, and it does not scale down: the smaller the team, the slower it gets.

Synsema hit it from the outside. A three-line function we contributed to candle was approved and then sat open for over two months, and with it two quantized architectures our users needed. Building the same shape here would make us the bottleneck, and we are fewer people than they are.

So in Synsema an architecture is a text file that the already-compiled binary reads at startup. The operations (matmul, RMSNorm, RoPE, attention, SwiGLU, …) are compiled in; what is missing is the order and the parameters, and those are data. .archdef is to the inference engine what a .syn is to the interpreter: the engine is not rebuilt to run a new one.

Nobody has to wait for us.

Choosing the engine§

SYNSEMA_INFER_BACKEND=rust         # the engine written by us — the one that runs .archdef files
                                   # anything else, or unset: candle (the default)
SYNSEMA_INFER_ARCHDEF=./archdefs   # a directory of <arch>.archdef files

Both resolve process environ > .env > default, like every other knob, and synsema init writes both into .env.example.

Switching engines changes the generated text. Both are correct; they approximate the same numbers differently (candle quantizes the activation before the matmul, we multiply in f32). The engine is therefore part of what you declare to reproduce an output, next to the binary and the weights. candle stays the default while both exist.

Definitions only run on the rust engine. With candle the list is compiled and a .archdef is ignored — synsema llm status says exactly that instead of pretending otherwise.

What is in effect right now§

synsema llm status          # engine, architectures, origin and sha of each
synsema llm status --json   # the same under "inference", with the FULL sha

Arquitecturas que corre el backend `rust` (4):
  gemma3 (20 pasos por capa, en el binario, sha b935fcfab2d6)
  llama (16 pasos por capa, en el binario, sha a580fa45fdb6)
  qwen2 (19 pasos por capa, en el binario, sha fbba11641175)
  qwen3 (18 pasos por capa, ./archdefs/qwen3.archdef, sha 5b8d9db76a68)
  Definiciones del operador: ./archdefs

The sha is the sha256 of the text of the definition. Two runs with the same sha ran the same architecture — that is why it is printed. A file of yours named after one of ours wins, and the line shows it, so a substitution is never silent.

The format§

Three sections, in this order. block runs once per layer, with {i} replaced by the layer index. Tensors are named exactly as the GGUF names them, so writing a definition is mostly copying the tensor list out of the file.

This is qwen3.archdef, shipped in the binary, in full:

# qwen3 — llama with a per-head RMSNorm on Q and K, before RoPE. That is the whole difference.
#
# Order matters: normalising after RoPE gives a different model, one that still generates text,
# so the mistake does not show until you compare against the upstream.

arch qwen3
kind decoder

prologue
  x = embed(token_embd.weight)

block
  h = rms_norm(x, blk.{i}.attn_norm.weight)
  q = matmul(h, blk.{i}.attn_q.weight)
  k = matmul(h, blk.{i}.attn_k.weight)
  v = matmul(h, blk.{i}.attn_v.weight)
  # What qwen3 adds, and the only thing.
  norm_heads(q, blk.{i}.attn_q_norm.weight, head_count)
  norm_heads(k, blk.{i}.attn_k_norm.weight, head_count_kv)
  rope(q, head_count)
  rope(k, head_count_kv)
  a = attention(q, k, v)
  o = matmul(a, blk.{i}.attn_output.weight)
  add(x, o)

  h = rms_norm(x, blk.{i}.ffn_norm.weight)
  g = matmul(h, blk.{i}.ffn_gate.weight)
  silu(g)
  u = matmul(h, blk.{i}.ffn_up.weight)
  mul(g, u)
  d = matmul(g, blk.{i}.ffn_down.weight)
  add(x, d)

epilogue
  x = rms_norm(x, output_norm.weight)
  x = last(x)
  logits = matmul(x, output.weight | token_embd.weight)

The diff between two definitions is what makes the two architectures different — there are no flags and no if arch == … on the other side of a file. qwen2 simply has three bias lines that llama does not.

Rules§

• The file name is the architecture name. qwen3.archdef must declare arch qwen3. It is not bureaucracy: it is the only thing known about a file that does not even parse, and therefore the only way to say which architecture just broke.
• x is the residual — the register that carries state across layers.
• a | b means "this tensor, or that one if the first is absent" (tied embeddings).
• What the GGUF already declares is not repeated. head_count, head_count_kv, embedding_length, the sliding window: all read from the metadata. A param line is only for what the file does not say — today the only one in use is gemma3's window pattern.
• Comments are #; blank lines are free.

The operations§

embed · rms_norm · matmul · add_bias · norm_heads · rope · attention · silu · gelu · gelu_tanh · relu · mul · add · scale · copy · last

dst = op(src, …) assigns to a register; op(dst, …) modifies one in place. That is the entire grammar.

gelu and gelu_tanh are not the same function. gelu_tanh is the tanh approximation (gelu_pytorch_tanh); the two differ by about 1e-3, and a model run with the wrong one drifts instead of failing. Gemma 3 wants gelu_tanh — getting this wrong cost a real investigation, because the output stays readable and only the numbers move (top logit 7.91 against 27.02).

No control flow, and that is the security property§

A definition has no conditionals, no loops, no function calls, and no way to open a file, a socket or the environment. It describes a graph of matrix multiplications.

That is what makes it safe to use a definition written by someone you do not know. Running one does not run their code: the worst a malicious file can do is fail to load, or give wrong numbers with your weights, bounded by the same resource limits as any model. Downloading a .archdef is not like installing a plugin. The obvious alternative — native plugins in .so/.dll — would be remote code execution with extra steps, and is ruled out.

A test enforces it: twenty-one words including if, while, loop, for, exec, import, open, http and env are rejected as operations that do not exist. The day control flow is added, the property is gone. It is not getting added.

Writing one§

1. Read general.architecture and the tensor names out of your GGUF. 2. Copy the closest definition we ship (llama is the plainest) and save it as <arch>.archdef. 3. Change the arch line to match the file name, then adjust the block to the tensors your file actually has. 4. Put it in a directory; set SYNSEMA_INFER_ARCHDEF to it and SYNSEMA_INFER_BACKEND=rust. 5. synsema llm status — your definition should be listed, with its path and its sha. 6. Run a prompt, and compare against ollama run <model> with the same prompt. The tokens should match. If they do not, the order of the steps is wrong far more often than the maths.

That last step is the one to take seriously. An oracle proves two implementations agree, not that either is right: we validated gemma3 against candle, it was green, and candle had the same bug we did. Compare against what the rest of the world runs.

When a definition is not enough§

The format covers the roughly 80% of architectures that are remixes of known blocks. It does not aspire to 100%, and pretending otherwise would turn it into a badly designed programming language. An architecture with genuinely new maths needs Rust, and that is fine — what changed is who can add a model, not that everyone can add every model.

Encoders (ModernBERT, Laya) are written by hand on purpose too: bidirectional attention, two alternating RoPE bases and decision heads share almost nothing with a decoder, and forcing them into the same vocabulary would produce a worse format for both.

Errors§

A definition is foreign data, like a .gguf, so it is held to the same standard as synsema check: fail early, name the line, say the fix. Never a panic halfway through a forward pass, thirty seconds in.

⚠ no cargó — ./archdefs/qwen3.archdef: línea 28: no existe la operación `siluu` — ¿quisiste decir `silu`?

A broken file leaves that architecture unavailable — it never falls back to ours. This matters more than it sounds. The first version did fall back, and a live test showed why that is wrong: with a typo in silu, the model answered perfectly, using our definition. The operator would have sworn their file was running. Now the model refuses to load, and the error names the file:

[local error: no se pudo cargar 'qwen3:0.6b': el modelo declara la arquitectura 'qwen3', que este
binario no conoce.
Conocidas: gemma3, llama, qwen2.
Definiciones que no cargaron:
  - ./archdefs/qwen3.archdef: línea 28: no existe la operación `siluu` — ¿quisiste decir `silu`?]

Who chooses§

The operator chooses the engine, the model and the definitions directory — never the .syn program. A program asks to generate text; it cannot name a model, an architecture or a path, and it cannot make the engine read a file the operator did not enable. Discovering caches offers candidates to whoever writes the configuration; it does not open the disk to the program. Same rule as secrets and hosts: see Capabilities.

Determinism and provenance§

With SYNSEMA_LLM_TEMPERATURE=0 (the default) generation is greedy and repeatable. To say what exactly ran, three things have to travel together — and all three are in synsema llm status --json:

The binary itself is attested separately: see Attestation.