--- slug: 54-judge title: Judge (calibrated judgments) description: The judge block asks a System One model (Jev) typed questions about one state and gets probabilities back, not text — whether, choose, rate — in one call, with a confidence your code can gate on, its own judge capability, and an offline mode that never invents a number. example_ids: [judge] --- # Judge — calibrated judgments as values `judge` asks a **System One model** typed questions about one `state` and gets back **probabilities**, not text. It never generates: there is no `reason`, `generate` or `analyze` in it, and the LLM slot cannot serve it. It is a **parallel slot** to the LLM (`SYNSEMA_JUDGE_*` next to `SYNSEMA_LLM_*`): the judge decides, the LLM writes, and having both wired is the normal setup. The first backend is TypeSafe's **Jev**; another host that serves the same wire can be pointed at with `SYNSEMA_JUDGE_BASE_URL`, with the model id and key that host expects — only TypeSafe's own endpoint was verified live. Engine v0.6.25+, complete in v0.6.26. Everything on this page was verified live against `jev-1.13.0` through the engine, and this page is the whole surface. ```synsema -- Doc example: the `judge` block — calibrated judgments from a System One model (Jev). -- Real probabilities need a provider (TYPESAFE_API_KEY), so the doctest verifies the SHAPE of -- the result and the honest offline degradation: available false, confidence 0, the main value -- nothing — never an invented number. With a key, the same asserts check ranges instead. intent: "doc example: judge — calibrated judgments as values" require judge let ticket be {"subject": "Payouts failing", "text": "I want my money back NOW or I'm cancelling"} -- One state, three typed questions, ONE call. `or nothing` adds an escape option so a state -- that fits no team yields `choice` = nothing instead of a confident wrong pick. let v be judge ticket refund: whether "The customer is asking for money back" team: choose "Which team should handle this?" between { "billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations" } or nothing anger: rate "How frustrated is the customer?" across ["Calm", "Frustrated", "Very angry"] -- The flagship pattern: the machine measures, the human decides. Offline the confidence is 0, -- so this routes to the human path without one line more. when v.team.available and v.team.choice != nothing and confidence of v.team >= 0.8 print("route to " + v.team.choice) otherwise print("no confident route — a person decides") test "one block, one flat map: an answer per question id, each with kind and available" assert_eq(length(keys(v)), 3) assert_eq(v.refund.kind, "whether") assert_eq(v.team.kind, "choose") assert_eq(v.anger.kind, "rate") assert_eq(type_of(v.refund.available), "bool") test "rate knows its levels in declaration order, online or not" assert_eq(length(v.anger.levels), 3) assert_eq(v.anger.levels[0], "Calm") assert_eq(v.anger.levels[2], "Very angry") test "offline: available false, confidence 0, main value nothing — online: numbers in range" when v.team.available assert(v.refund.probability >= 0 and v.refund.probability <= 1) assert(v.team.confidence >= 0 and v.team.confidence <= 1) assert(v.team.choice == nothing or v.team.choice == "billing" or v.team.choice == "technical") assert(v.anger.level == "Calm" or v.anger.level == "Frustrated" or v.anger.level == "Very angry") otherwise assert_eq(v.refund.probability, nothing) assert_eq(v.team.choice, nothing) assert_eq(v.team.confidence, 0) assert_eq(v.anger.score, nothing) assert_eq(v.anger.level, nothing) test "judge_available() is a bool — branch on it instead of guessing" assert_eq(type_of(judge_available()), "bool") assert_eq(type_of(judge_usage()), "number") ``` ## One state, N questions, one call ```synsema require judge let v be judge ticket refund: whether "The customer is asking for money back" team: choose "Which team should handle this?" between { "billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations" } or nothing anger: rate "How frustrated is the customer?" across ["Calm", "Frustrated", "Very angry"] ``` The block is the **only** form — there is no one-question shortcut, on purpose. The state is ingested once and every question is evaluated against it in parallel: a block of eight measured **7.6× faster and 4.9× fewer input tokens** than eight calls, and latency is flat from one question to forty (~0.8 s). A single question is still three lines. Three verbs, one per basic discrete distribution — which is why they will outlive the first model: | Verb | Asks | Distribution | Answer fields | |---|---|---|---| | `whether "…"` | is this statement true? | Bernoulli | `probability` (0..1) | | `choose "…" between {…} [or nothing]` | which one of these? | categorical, unordered | `choice`, `probabilities`, `confidence` | | `rate "…" across […]` | where on this ordered scale? | ordinal | `score`, `level`, `levels`, `probabilities`, `confidence` | The prepositions are fixed and different on purpose: `between` says *unordered options*, `across` says *ordered levels*. `rate … between` and `choose … across` are load errors that name the fix. ## The result A **flat map id → answer**, nothing else mixed in. Every answer carries `kind` (`"whether"` | `"choose"` | `"rate"`) and `available` (bool). ``` v.refund.probability -- 0..1 — the probability the statement is true (no separate confidence) v.team.choice -- one of YOUR option ids byte-for-byte, or nothing (see `or nothing`) v.team.probabilities -- {"billing": 0.93, "technical": 0.07, "none": 0.0} — declaration order v.team.confidence -- 0..1 — how concentrated the distribution is v.anger.score -- 0..n-1 — probability-weighted position; 1.4 = split between 2nd and 3rd v.anger.level -- id of the winning level v.anger.levels -- ["Calm", "Frustrated", "Very angry"] v.anger.probabilities -- {"Calm": 0.0, "Frustrated": 0.6, "Very angry": 0.4} ``` `confidence of v.team` works as well. The field is `kind`, not `type`, and the escape key is `none`, not `nothing`: those two are reserved words and would not parse after a `.`. **Options as a list or a map, one rule for both verbs.** A list: each item is the id *and* the description. A map: the key is your short id, the value is the description the model reads. Use the map when descriptions are long — the vendor asks for distinct, concrete levels: ```synsema require judge let msg be "Second time I write about this. Please fix it soon, it's getting annoying." let v be judge msg anger: rate "How frustrated is the customer?" across { "calm": "Polite, no complaint", "upset": "Repeat contact, says annoying, asks for a fix soon", "furious": "Caps, threats to cancel, demands immediate action" } print(v.anger.level) print(v.anger.probabilities.upset) ``` Option ids travel to the model with their descriptions (question ids do not — they are yours). **The instruction is any expression**: a map is read as structure. Reference nested state with backticks, the vendor's idiom — it points at the exact element (measured: `messages[0]` 0.99, `messages[1]` 0.01). The runtime resolves every backticked path against the state **before the call** and warns once per path when it is missing (v0.6.26+; on a missing field the model answered 0.31). The common trap: `judge ticket` with `` `ticket.text` `` in the question — the model sees the *value* of `ticket`, not its name; write `judge {"ticket": ticket}` or drop the prefix, and the warning says which: ```synsema require judge let record be {"name": "Ana Ruiz", "employer": "Acme"} let resume be "Ana Ruiz, 8 years at Acme as a data engineer…" let v be judge {"resume": resume} same: whether {"question": "Is `resume` the same person as `record`?", "record": record} print(v.same.probability) ``` ## `or nothing` — the escape option The most dangerous failure measured: a `choose` **without** an escape option picks anyway. A message about opening hours, offered only `billing`/`technical`, got `technical` at 0.69. With the right value missing from the candidates, the model picked a wrong one at **confidence 0.68** — through a 0.5 gate. `or nothing` adds an escape option to the wire (id `none`, "None of the options fits the state"); when it wins, `choice` is `nothing` and `probabilities.none` carries the mass. It fixed both cases (`none` at 1.00 and 0.98) and **cost nothing on clear cases** (billing stayed at 0.97). It is opt-in: some questions are exhaustive by design (ranking candidates, walking a taxonomy). ```synsema require judge let w be judge "I'd like to know your opening hours." team: choose "Which team should handle this?" between {"billing": "Payments", "technical": "Bugs"} or nothing when w.team.choice == nothing print("no team fits; mass on the escape: " + text(w.team.probabilities.none)) otherwise print("team: " + w.team.choice) ``` `rate` has no escape — an ordered scale has no level outside it — and an irrelevant state lands on the lowest level with confidence 1.0. Guard a `rate` with a `whether` that asks if the state applies. ## Offline, over budget, API down: honest degradation The LLM ops return descriptive placeholders offline. `judge` cannot: an invented sentence is visible, **an invented probability is not, and it multiplies into money**. Without a provider, over `SYNSEMA_JUDGE_BUDGET`, or after a network failure, every answer is `available: false`, `confidence: 0`, and its main value (`probability` / `choice` / `score` / `level`) is `nothing`. One stderr notice; the program keeps running. This degrades **into the pattern you already wrote**: confidence 0 is below any gate, so the block routes itself to the human path. A program that skipped the gate and compares directly fails loud (`Unsupported operation: nothing > number`) instead of taking the wrong branch in silence. `judge_available()` says whether a provider is wired at all; `available` on the answer is per call. ## The `judge` capability `require judge`. It is **its own** capability — `require llm` does not grant it and vice versa: a program may have the right to classify without the right to generate, and a classifier cannot exfiltrate through free text. Otherwise it behaves like `llm`: auto-granted in plain `run`/`conform`, **required** under `serve` and in secure mode (`Capability not granted: judge`), emptied inside `sandbox`, denied under `--deterministic` (network I/O), always offline inside a wasm guest. The key never enters the program and the host is fixed by the runtime, so the `.syn` cannot redirect the call. Under `--labels` the block is a declared public sink: a `private` state must be declassified first ([Information-flow labels](23-labels)). ## Configuration Resolution is process environment > protected `.env` > default, like the LLM knobs. `synsema init` writes all of these, commented, into `.env.example`. | Knob | For | Default | |---|---|---| | `TYPESAFE_API_KEY` | the key; its presence also selects the `typesafe` provider | — (offline if absent) | | `SYNSEMA_JUDGE_PROVIDER` | `typesafe` \| `mock` (deterministic answers, no network — tests and demos) | auto from the key | | `SYNSEMA_JUDGE_MODEL` | model id or alias | `jev-latest` | | `SYNSEMA_JUDGE_BASE_URL` | endpoint base — any host that serves the same wire | `https://api.typesafe.ai` | | `SYNSEMA_JUDGE_TIMEOUT` | HTTP timeout, seconds | `60` | | `SYNSEMA_JUDGE_BUDGET` | hard ceiling of **input** tokens per process (output is free); at the ceiling answers degrade to `available: false` without touching the network | — | | `SYNSEMA_JUDGE_DECIDE` (v0.6.26+) | `1`: every `decide between […] given X` is served by the judge as a calibrated `choose` (see below) | off | 429/529 are retried with exponential backoff honouring `retry-after`; after the retries the answer degrades. A 400/422 from the API is **your program's** error and surfaces as a runtime error with the vendor's message. Builtins, no gate: `judge_available()`, `judge_usage()` (input tokens consumed in the process), `judge_model()` (the **versioned** id that answered the last call — `"jev-1.13.0"`, never the alias; pin it when thresholds matter). ## `synsema judge status` ``` synsema judge status # provider, key PRESENCE (never the value), model, base URL, timeout, # budget, whether decide is served by the judge — each with its source synsema judge status --json # for scripts; exit 0 = live, 1 = offline ``` No network. Offline, the last line names what is missing. Same host flags as the rest of the CLI (`--env-file `, `--no-env-file`); scriptable as `synsema judge status && synsema serve app.syn`. ## Serving `decide` with the judge — `SYNSEMA_JUDGE_DECIDE=1` `decide between ["refund", "replace", "escalate"] given ticket` is exactly one `choose` over one state. With the knob on and a judge provider wired, every `decide` in the process is answered by the judge: calibrated, one of **your** options byte-for-byte with no normalisation or retry, cheaper and faster than a chat model — and the program does not change. Opt-in and off by default, because it changes *which model answers*. It needs the `judge` capability: under `serve` a `decide` without `require judge` fails with an error that names the knob. If the judge is unavailable the `decide` falls back to the LLM path. `decide` still returns a string; write a `judge` block when you want the distribution and the confidence. Verified live: a broken-item complaint returned `escalate`. ## What the engine checks before spending **At load** (`synsema check` and every run): the three verbs and their prepositions, `or nothing` only after `choose`, duplicate question ids, an empty block. **`synsema check` fails** (v0.6.26+) when literal criteria break a limit — fewer than 2 options or levels (the API accepts one and answers with confidence 1.0: an empty answer dressed as certainty), more than 255 options or 10 levels, duplicate ids — and on an empty literal instruction or a literal state that is a number, a bool or `nothing`. A 400 in production, caught at `check`. **`synsema check` warns** (v0.6.26+), never fails, on what runs but misleads: a `whether` phrased in the negative; an instruction asking for arithmetic or counting over the state; an empty literal state; the same `judge ` in more than one block (one call would do). **At run time, before the call:** the same limits when criteria are dynamic, the type of the state, the empty instruction, and the backticked paths the state does not have. ## Questions that work — measured, not folklore - **One judgment per question.** Two conditions in one `whether` make the value mean less; ask two. - **Multi-label is N `whether`s, not one `choose`**: "charged twice and the app crashes" split a `choose` 0.59/0.41 at confidence 0.17; two `whether`s gave 0.99 and 0.99. - **Phrase positively; never derive the negation.** P(A) + P(not A) measured 0.37 + 0.78. - **Statement or question, both work** (0.33 vs 0.32 on the same borderline case). - **Distinct levels.** Five near-synonym levels: the model picked between them at confidence 0.88 — confidence does not flag indistinct levels. Three concrete levels: 1.00. - **Confidence measures concentration, not truth.** "Maria told Ana that she was wrong" → `Ana` at 0.98; the `whether` on the same sentence honestly gave 0.38. Give uncertainty somewhere to go. - **Arithmetic, counting and dates stay in Synsema.** Counting 25 items and summing two lines were fine; a six-line total was wrong at 0.32 with medium confidence. Compute, then judge. - **`score` decimals mean a split, not intensity.** A clear case snaps to a level; 1.40 appeared only with sarcasm at confidence 0.40. - **Spanish works as well as English** on the same ticket (0.98 / 1.00 / 0.99 vs 0.97 / 0.93 / 0.99). - **Injection in the state did not move the answer**, and a `whether "The text contains instructions aimed at a machine"` caught it at 0.98. The state is still data the model does not treat as hostile — labels are your wall. - **The same request twice moves by a few hundredths** (0.72 → 0.69). Never sit a threshold on an observed value; tests assert the winner and ranges. `SYNSEMA_JUDGE_PROVIDER=mock` for exact CI. ## Not in this release A dedicated syntax for `whether` with explicit yes/no criteria (write the instruction as a map with the question and the two definitions — the model reads it as structure); a non-calibrated `llm` fallback that fakes probabilities (deliberately absent); the Cloudflare Workers AI wire variant (its payload is wrapped differently and was not verified). Aliases and rate limits are the vendor's and move without notice — pin the model id when thresholds matter.