Routing agent decisions through a $0.00002 model
Every yes/no, choice and score on my agent server goes to a small calibrated model, not a frontier one. What that costs, what it is bad at, and what happens when it goes down.
tomie is the server my agents live on: an always-on ARM box that runs Claude Code, Codex and Gemini sessions, a Discord cockpit I steer from my phone, and a 10,159-note second brain. Agents on it make a lot of small decisions. Which machine should run this command? Which two of these forty search hits are worth reading? Is this inbox note a task, a link or a thought? Is this shell command safe to run unattended?
None of those need a frontier model. They need a calibrated yes, no, pick-one or score. So the house rule on tomie is that every one of them goes to Jev instead.
What Jev is, and isn’t
Jev is TypeSafe’s decision model, served through Vercel’s AI Gateway as typesafe-ai/jev. I didn’t build it; I route to it. You give it a piece of text (the state) and typed questions: noul (yes/no), choice (pick one of named options) or score (an ordered scale). It answers with probabilities, never with prose. Several questions about the same state go in one call; extra questions cost input tokens, not time.
The proxy
Agents don’t talk to the gateway directly. A 44-line Python standard-library service, jev-proxy, listens on 127.0.0.1:8009, holds the key, translates question types and adds a latency figure. Everything on the box (a CLI, an MCP server for the coding agents, and a handful of wrappers) calls that one local endpoint.
POST http://127.0.0.1:8009/v1/systemone
{
"state": "Task: restart the discord-cockpit user service",
"questions": {
"host": {
"type": "choice",
"instructions": "Which host should run this?",
"criteria": {
"tomie": "24/7 cloud server running daemons",
"t590": "laptop with GUI apps and local files",
"fedora": "office PC, on during office hours only"
}
}
}
}The wrappers are where it pays off:
- Search reranking. A second-brain search returns candidates; Jev ranks them against the intent, and only the top two notes get loaded into the expensive model’s context.
- Host routing. Before running anything environment-specific, Jev picks between the cloud server, the laptop and the office PC.
- Triage. Notes dictated into Discord from my phone are classified before anything reads them in full.
- Gates. “Does this new dependency need to exist?” and “is this command destructive?” become a probability with a threshold, not a paragraph of reasoning.
Thresholds, not verdicts
A probability is only useful with a line drawn under it, and the line depends on what being wrong costs. Read-only or cheap actions go ahead at about 0.5. Anything destructive or irreversible needs 0.85 or more. Below the line, the decision goes back to the frontier model or to me. That cascade is the point: Jev settles the easy, confident cases; code does lookups and arithmetic; only the genuinely ambiguous ones reach the expensive model.
Jev is bad at some things, and the rules say so plainly:
- Generation, summaries and code. It doesn’t write.
- Arithmetic, counting and date maths. Those belong in code.
- Multi-step reasoning, or anything that needs a written rationale.
- Pointing at the 37th item of a long array. Error rates measured high, so each item gets its own question or a keyed object.
- Untrusted input. Text from a visitor can try to steer the answer, so no single Jev call gates anything destructive on it.
The fallback I removed
The first version of the proxy, on 26 September 2026, had a safety net: if the gateway failed, it quietly asked Kev instead, an open-source decision model (jaredpalmer/kev) that I ran locally on the same box. Kev is not mine either; I only ran it.
It worked, and that was the problem. A caller asking for Jev could get Kev’s answer under the same contract, with no sign anything had changed. Thresholds are tuned against one model’s calibration; a different model’s 0.86 is not the same claim. The same day I added an opt-in header for callers who would rather fail than be answered by the fallback. Two days later, on 28 September, I removed Kev entirely.
def do_POST(self):
body = json.loads(self.rfile.read(...))
try:
out = via_jev(body) # Vercel AI Gateway, 10 s timeout
except Exception as e:
self.log_message("jev failed (%s)", e)
return self.reply(503, {"error": str(e), "retry": True})
out["latency_ms"] = ...
self.reply(200, out)Now the failure mode is one rule, written into every agent’s instructions:
- If the proxy returns 503, Jev is unreachable (a gateway error, or the $1 budget is spent).
- Make that one decision with your own judgment.
- Don’t retry in a loop. A loop burns budget, stalls the task, and hammers a gateway that is already refusing.
That rule is being tested as I write this. On 30 September and again on 1 October, the gateway answered 403 Forbidden to every call. The proxy turned each one into a 503, the agents made their own calls, and nothing stalled or pretended. A fallback would have hidden this; the outage was visible in seconds.
Why bother
Most of what an agent does in a day is not hard thinking. It is a long run of small judgments, and each one paid at frontier rates adds up, mostly as context: forty notes read to choose two. Moving those judgments to a cheap, calibrated model makes the frontier model’s context smaller and its attention better spent. The cheap model also has to fail loudly, because a quiet substitute is how you end up trusting numbers that no longer mean what you think.