LLM

Judge — calibrated judgments as values

judge asks a System One model typed questions about one state and gets back probabilities, not text. It never generates: there is no reason, generate or analyze in it, and the LLM slot cannot serve it. It is a parallel slot to the LLM (SYNSEMA_JUDGE_ next to SYNSEMA_LLM_): the judge decides, the LLM writes, and having both wired is the normal setup. The first backend is TypeSafe's Jev; another host that serves the same wire can be pointed at with SYNSEMA_JUDGE_BASE_URL, with the model id and key that host expects — only TypeSafe's own endpoint was verified live. Engine v0.6.25+, complete in v0.6.26. Everything on this page was verified live against jev-1.13.0 through the engine, and this page is the whole surface.

-- Doc example: the `judge` block — calibrated judgments from a System One model (Jev).
-- Real probabilities need a provider (TYPESAFE_API_KEY), so the doctest verifies the SHAPE of
-- the result and the honest offline degradation: available false, confidence 0, the main value
-- nothing — never an invented number. With a key, the same asserts check ranges instead.
intent: "doc example: judge — calibrated judgments as values"
require judge

let ticket be {"subject": "Payouts failing", "text": "I want my money back NOW or I'm cancelling"}

-- One state, three typed questions, ONE call. `or nothing` adds an escape option so a state
-- that fits no team yields `choice` = nothing instead of a confident wrong pick.
let v be judge ticket
    refund: whether "The customer is asking for money back"
    team:   choose "Which team should handle this?" between {
                "billing":   "Payments, invoicing, refunds",
                "technical": "Bugs, outages, integrations"
            } or nothing
    anger:  rate "How frustrated is the customer?" across ["Calm", "Frustrated", "Very angry"]

-- The flagship pattern: the machine measures, the human decides. Offline the confidence is 0,
-- so this routes to the human path without one line more.
when v.team.available and v.team.choice != nothing and confidence of v.team >= 0.8
    print("route to " + v.team.choice)
otherwise
    print("no confident route — a person decides")

test "one block, one flat map: an answer per question id, each with kind and available"
    assert_eq(length(keys(v)), 3)
    assert_eq(v.refund.kind, "whether")
    assert_eq(v.team.kind, "choose")
    assert_eq(v.anger.kind, "rate")
    assert_eq(type_of(v.refund.available), "bool")

test "rate knows its levels in declaration order, online or not"
    assert_eq(length(v.anger.levels), 3)
    assert_eq(v.anger.levels[0], "Calm")
    assert_eq(v.anger.levels[2], "Very angry")

test "offline: available false, confidence 0, main value nothing — online: numbers in range"
    when v.team.available
        assert(v.refund.probability >= 0 and v.refund.probability <= 1)
        assert(v.team.confidence >= 0 and v.team.confidence <= 1)
        assert(v.team.choice == nothing or v.team.choice == "billing" or v.team.choice == "technical")
        assert(v.anger.level == "Calm" or v.anger.level == "Frustrated" or v.anger.level == "Very angry")
    otherwise
        assert_eq(v.refund.probability, nothing)
        assert_eq(v.team.choice, nothing)
        assert_eq(v.team.confidence, 0)
        assert_eq(v.anger.score, nothing)
        assert_eq(v.anger.level, nothing)

test "judge_available() is a bool — branch on it instead of guessing"
    assert_eq(type_of(judge_available()), "bool")
    assert_eq(type_of(judge_usage()), "number")

One state, N questions, one call§

require judge

let v be judge ticket
    refund: whether "The customer is asking for money back"
    team:   choose "Which team should handle this?" between {
                "billing":   "Payments, invoicing, refunds",
                "technical": "Bugs, outages, integrations"
            } or nothing
    anger:  rate "How frustrated is the customer?" across ["Calm", "Frustrated", "Very angry"]

The block is the only form — there is no one-question shortcut, on purpose. The state is ingested once and every question is evaluated against it in parallel: a block of eight measured 7.6× faster and 4.9× fewer input tokens than eight calls, and latency is flat from one question to forty (~0.8 s). A single question is still three lines.

Three verbs, one per basic discrete distribution — which is why they will outlive the first model:

The prepositions are fixed and different on purpose: between says unordered options, across says ordered levels. rate … between and choose … across are load errors that name the fix.

The result§

A flat map id → answer, nothing else mixed in. Every answer carries kind ("whether" | "choose" | "rate") and available (bool).

v.refund.probability    -- 0..1 — the probability the statement is true (no separate confidence)
v.team.choice           -- one of YOUR option ids byte-for-byte, or nothing (see `or nothing`)
v.team.probabilities    -- {"billing": 0.93, "technical": 0.07, "none": 0.0}  — declaration order
v.team.confidence       -- 0..1 — how concentrated the distribution is
v.anger.score           -- 0..n-1 — probability-weighted position; 1.4 = split between 2nd and 3rd
v.anger.level           -- id of the winning level
v.anger.levels          -- ["Calm", "Frustrated", "Very angry"]
v.anger.probabilities   -- {"Calm": 0.0, "Frustrated": 0.6, "Very angry": 0.4}

confidence of v.team works as well. The field is kind, not type, and the escape key is none, not nothing: those two are reserved words and would not parse after a ..

Options as a list or a map, one rule for both verbs. A list: each item is the id and the description. A map: the key is your short id, the value is the description the model reads. Use the map when descriptions are long — the vendor asks for distinct, concrete levels:

require judge

let msg be "Second time I write about this. Please fix it soon, it's getting annoying."
let v be judge msg
    anger: rate "How frustrated is the customer?" across {
        "calm":    "Polite, no complaint",
        "upset":   "Repeat contact, says annoying, asks for a fix soon",
        "furious": "Caps, threats to cancel, demands immediate action"
    }
print(v.anger.level)
print(v.anger.probabilities.upset)

Option ids travel to the model with their descriptions (question ids do not — they are yours). The instruction is any expression: a map is read as structure. Reference nested state with backticks, the vendor's idiom — it points at the exact element (measured: messages[0] 0.99, messages[1] 0.01). The runtime resolves every backticked path against the state before the call and warns once per path when it is missing (v0.6.26+; on a missing field the model answered 0.31). The common trap: judge ticket with ` ticket.text in the question — the model sees the value of ticket, not its name; write judge {"ticket": ticket}` or drop the prefix, and the warning says which:

require judge

let record be {"name": "Ana Ruiz", "employer": "Acme"}
let resume be "Ana Ruiz, 8 years at Acme as a data engineer…"
let v be judge {"resume": resume}
    same: whether {"question": "Is `resume` the same person as `record`?", "record": record}
print(v.same.probability)

or nothing — the escape option§

The most dangerous failure measured: a choose without an escape option picks anyway. A message about opening hours, offered only billing/technical, got technical at 0.69. With the right value missing from the candidates, the model picked a wrong one at confidence 0.68 — through a 0.5 gate. or nothing adds an escape option to the wire (id none, "None of the options fits the state"); when it wins, choice is nothing and probabilities.none carries the mass. It fixed both cases (none at 1.00 and 0.98) and cost nothing on clear cases (billing stayed at 0.97). It is opt-in: some questions are exhaustive by design (ranking candidates, walking a taxonomy).

require judge

let w be judge "I'd like to know your opening hours."
    team: choose "Which team should handle this?" between {"billing": "Payments", "technical": "Bugs"} or nothing

when w.team.choice == nothing
    print("no team fits; mass on the escape: " + text(w.team.probabilities.none))
otherwise
    print("team: " + w.team.choice)

rate has no escape — an ordered scale has no level outside it — and an irrelevant state lands on the lowest level with confidence 1.0. Guard a rate with a whether that asks if the state applies.

Offline, over budget, API down: honest degradation§

The LLM ops return descriptive placeholders offline. judge cannot: an invented sentence is visible, an invented probability is not, and it multiplies into money. Without a provider, over SYNSEMA_JUDGE_BUDGET, or after a network failure, every answer is available: false, confidence: 0, and its main value (probability / choice / score / level) is nothing. One stderr notice; the program keeps running.

This degrades into the pattern you already wrote: confidence 0 is below any gate, so the block routes itself to the human path. A program that skipped the gate and compares directly fails loud (Unsupported operation: nothing > number) instead of taking the wrong branch in silence. judge_available() says whether a provider is wired at all; available on the answer is per call.

The judge capability§

require judge. It is its own capability — require llm does not grant it and vice versa: a program may have the right to classify without the right to generate, and a classifier cannot exfiltrate through free text. Otherwise it behaves like llm: auto-granted in plain run/conform, required under serve and in secure mode (Capability not granted: judge), emptied inside sandbox, denied under --deterministic (network I/O), always offline inside a wasm guest. The key never enters the program and the host is fixed by the runtime, so the .syn cannot redirect the call. Under --labels the block is a declared public sink: a private state must be declassified first (Information-flow labels).

Configuration§

Resolution is process environment > protected .env > default, like the LLM knobs. synsema init writes all of these, commented, into .env.example.

429/529 are retried with exponential backoff honouring retry-after; after the retries the answer degrades. A 400/422 from the API is your program's error and surfaces as a runtime error with the vendor's message. Builtins, no gate: judge_available(), judge_usage() (input tokens consumed in the process), judge_model() (the versioned id that answered the last call — "jev-1.13.0", never the alias; pin it when thresholds matter).

synsema judge status§

synsema judge status            # provider, key PRESENCE (never the value), model, base URL, timeout,
                                # budget, whether decide is served by the judge — each with its source
synsema judge status --json     # for scripts; exit 0 = live, 1 = offline

No network. Offline, the last line names what is missing. Same host flags as the rest of the CLI (--env-file <path>, --no-env-file); scriptable as synsema judge status && synsema serve app.syn.

Serving decide with the judge — SYNSEMA_JUDGE_DECIDE=1§

decide between ["refund", "replace", "escalate"] given ticket is exactly one choose over one state. With the knob on and a judge provider wired, every decide in the process is answered by the judge: calibrated, one of your options byte-for-byte with no normalisation or retry, cheaper and faster than a chat model — and the program does not change. Opt-in and off by default, because it changes which model answers. It needs the judge capability: under serve a decide without require judge fails with an error that names the knob. If the judge is unavailable the decide falls back to the LLM path. decide still returns a string; write a judge block when you want the distribution and the confidence. Verified live: a broken-item complaint returned escalate.

What the engine checks before spending§

At load (synsema check and every run): the three verbs and their prepositions, or nothing only after choose, duplicate question ids, an empty block.

synsema check fails (v0.6.26+) when literal criteria break a limit — fewer than 2 options or levels (the API accepts one and answers with confidence 1.0: an empty answer dressed as certainty), more than 255 options or 10 levels, duplicate ids — and on an empty literal instruction or a literal state that is a number, a bool or nothing. A 400 in production, caught at check.

synsema check warns (v0.6.26+), never fails, on what runs but misleads: a whether phrased in the negative; an instruction asking for arithmetic or counting over the state; an empty literal state; the same judge <variable> in more than one block (one call would do).

At run time, before the call: the same limits when criteria are dynamic, the type of the state, the empty instruction, and the backticked paths the state does not have.

Questions that work — measured, not folklore§

• One judgment per question. Two conditions in one whether make the value mean less; ask two.
• Multi-label is N whethers, not one choose: "charged twice and the app crashes" split a

choose 0.59/0.41 at confidence 0.17; two whethers gave 0.99 and 0.99.

• Phrase positively; never derive the negation. P(A) + P(not A) measured 0.37 + 0.78.
• Statement or question, both work (0.33 vs 0.32 on the same borderline case).
• Distinct levels. Five near-synonym levels: the model picked between them at confidence 0.88 —

confidence does not flag indistinct levels. Three concrete levels: 1.00.

• Confidence measures concentration, not truth. "Maria told Ana that she was wrong" → Ana at

0.98; the whether on the same sentence honestly gave 0.38. Give uncertainty somewhere to go.

• Arithmetic, counting and dates stay in Synsema. Counting 25 items and summing two lines were

fine; a six-line total was wrong at 0.32 with medium confidence. Compute, then judge.

• score decimals mean a split, not intensity. A clear case snaps to a level; 1.40 appeared only

with sarcasm at confidence 0.40.

• Spanish works as well as English on the same ticket (0.98 / 1.00 / 0.99 vs 0.97 / 0.93 / 0.99).
• Injection in the state did not move the answer, and a `whether "The text contains instructions

aimed at a machine"` caught it at 0.98. The state is still data the model does not treat as hostile — labels are your wall.

• The same request twice moves by a few hundredths (0.72 → 0.69). Never sit a threshold on an

observed value; tests assert the winner and ranges. SYNSEMA_JUDGE_PROVIDER=mock for exact CI.

Not in this release§

A dedicated syntax for whether with explicit yes/no criteria (write the instruction as a map with the question and the two definitions — the model reads it as structure); a non-calibrated llm fallback that fakes probabilities (deliberately absent); the Cloudflare Workers AI wire variant (its payload is wrapped differently and was not verified). Aliases and rate limits are the vendor's and move without notice — pin the model id when thresholds matter.