Detection Is Not Generation: Why a Polling Loop Runs at the Speed of Its Most Expensive Call

Tue, 22 Sep 2026 12:47:29 +0800

someone sent me an install command for a skill called typesafe and i assumed from the name it was a typescript linting thing. it is not. it is a model that only answers multiple choice. reading its docs sent me somewhere more interesting than the tool itself, which is the shape of the loop it wants to sit in, so i had my AI pull the specs and help me work the maths out. i have not run it, and the note says so where it matters. the personal note at the bottom is mine to write after.

tl;dr

a “System One” model returns typed answers (one of these options, yes or no, a number on this scale) instead of prose, in about 100ms. the interesting part is not that it is cheap. it is that most real-time LLM loops fuse two jobs into one call: noticing something is there, and writing about it. the noticing is cheap and must run constantly; the writing is expensive and should run rarely. fuse them and the cheap job inherits the expensive job’s cadence, which is why so many “real-time” features are secretly a thirty-second timer. split them and the saving is $1/(k+p)$, where $k$ is how cheap your detector is and $p$ is how often a tick actually has something worth saying. the punchline is that $p$ dominates: if your trigger rate is high, splitting buys you almost nothing.

the question

i got handed a one-line install command for something called the “typesafe” skill. the name suggested static typing. what it actually installs is a wrapper around a hosted model called Jev, and Jev does not write.

that was the interesting bit. we have spent three years treating “call an LLM” as one primitive. here is a thing that deliberately gives up the generation half, keeps the understanding half, and gets two orders of magnitude cheaper and roughly an order of magnitude faster in exchange. so: what is that trade actually good for, and where does it fall over?

what it returns, which is the whole design

TypeSafe calls this class of thing a System One model, after Kahneman’s fast-and-intuitive System 1. you send it a state (a string, or structured JSON: a conversation, some records, a policy) and a map of questions. it answers every question against that state, in parallel, in one request.

there are exactly three question types, and that constraint is the product:

type	returns	example
`noul`	probability the answer is yes	`0.95`
`choice`	one of your options, plus the full distribution over them	`"billing"`
`score`	a probability-weighted value across your levels	`1.4`

no free text, no code, no explanation of its reasoning. you define the answer space; it points at a spot inside it.

two consequences fall straight out of that. first, it cannot hallucinate an option you did not supply, because the output is structurally constrained to your list. that is a stronger guarantee than prompting a general model to “only answer from these categories” and parsing what comes back. second, because it returns a distribution rather than a single token, you get calibrated confidence for free: a peaked distribution means it is sure, a flat one means it is not, and choice and score collapse that shape into a single confidence number between 0 and 1 so you can threshold on it without doing the maths yourself.

that second one matters more than it sounds. their docs put it well: a system that cannot express honest uncertainty cannot be trusted. the usual prompt-and-parse pattern gives you an answer with no honest way to ask how sure it was, so you end up either trusting everything or building a second LLM call to grade the first.

the thing i actually learnt

here is where the tool stopped being the interesting part.

think about any feature that watches a stream and occasionally says something useful. a live transcript that surfaces a prompt. a log tail that raises an incident. an inbox that flags the one message that needs a human. they all have the same shape:

1every Δ seconds:
2 look at the recent window
3 decide whether anything is worth acting on
4 if so, produce the thing to show

the ordinary way to build that is one LLM call per tick, doing both steps at once. ask it “is there a gap here, and if so what should we say?” and parse the answer. it works, and it is one call, and it feels efficient.

it is not, and the reason is that those two steps have opposite requirements:

	deciding whether	producing what
how often it must run	every tick, continuously	rarely, only when there is something
latency budget	must be sub-second or the tick rate collapses	seconds are fine
output	a boolean, or a number	prose
cost per call	wants to be near zero	is allowed to be expensive

when one call does both, the tick interval is set by the expensive half. you cannot poll every three seconds if each poll costs a full generation, so you poll every thirty, and the feature that was supposed to be real-time quietly becomes a timer. and you are paying the generation price on every tick, including the overwhelming majority of ticks where the honest answer was “nothing here”.

split them and the detector sets the cadence instead. at roughly 100ms per detection you can look every few seconds; the generator only wakes when the detector says something is actually there.

the maths, and the caveat hiding in it

worth doing properly, because it tells you when not to bother.

symbol	what it means
$c_d$	cost of one detection call
$c_g$	cost of one generation call
$p$	trigger rate: the fraction of ticks that genuinely have something worth generating
$k$	$c_d/c_g$, the detector’s cost relative to the generator

fused: every tick pays for a generation, so cost per tick is $c_g$.

split: every tick pays for a detection, and a $p$ fraction of ticks additionally pay for a generation, so cost per tick is $c_d + p,c_g$.

the ratio between them is the saving:

$$ R = \frac{c_g}{c_d + p,c_g} = \frac{1}{k + p} $$

that little formula is the whole decision. two things read straight off it.

first, your saving is capped at $1/p$ no matter how cheap the detector gets. driving $k$ to zero leaves you with $1/p$. if a quarter of your ticks have something worth saying, a free detector still only saves you 4x.

second, $p$ dominates $k$ almost immediately. Jev is priced at 4.2 cents per million input tokens, against roughly a dollar for the cheapest sensible general model and five dollars for a frontier one, so $k$ lands somewhere between 0.042 and 0.008. both are small enough that once $p$ climbs past about 0.1, which detector you chose barely moves the answer:

The two curves are a 24x cheaper detector and a 119x cheaper detector. They separate only at the far left, where triggers are rare. Everywhere else the trigger rate sets the answer and the detector's price is noise.

and there is a break-even, in the same way the build-versus-buy crossover has one. $R$ falls through 1 at $p = 1 - k$, around 96%. past that you are paying for a detector to tell you something you were going to do anyway, and the split is a net loss. that is the honest boundary: this pattern is for needles in haystacks. if your stream is mostly needles, keep the one call.

where people get this wrong

thinking it replaces the LLM. it does not, and the pitch is not “swap your model for a cheaper one”. Jev cannot write your output. it re-scopes the generator to fire rarely instead of constantly. anyone who hears “cheaper model” has heard the wrong thing.

reaching for it to extract values. there is no open-ended extraction. you cannot ask it to pull a figure out of a paragraph. what you can do is have ordinary code find the candidate spans, then have the model pick which one you meant. select, never generate. that inversion is most of the cookbook, and it is also why the output cannot be invented.

leading with the cost saving. it is real and it is usually irrelevant. at these prices a personal-scale workload costs cents a month, so “it is cheaper” invites the obvious reply that the current bill is already small. the argument that survives is the cadence one: you could not afford to look every three seconds before, and now you can.

stuffing the corpus into the state. the context is 64k per request and 32k for the state plus the longest single question. a document library does not fit. shortlist in code, judge the shortlist.

assuming calibration means correctness. their docs are explicit that calibration is measured across groups of predictions, not guaranteed per answer. so thresholds have to be fitted against your own labelled data, and building that labelled set is the actual project. the integration is the part that merely looks like the work.

when it applies, and when it does not

it applies when the judgment is closed (you can enumerate the answers), frequent (you want it running constantly), and mostly negative (the honest answer is usually “nothing here”). routing, triage, screening, reranking a shortlist, verifying a claim against its evidence, deciding whether to wake something expensive.

it does not apply when you need words out, when the input is an image or audio (text only, today), when the answer space is genuinely open, or when nearly every tick is a hit.

two constraints worth knowing before anyone builds on it: the published rate limit is 1,200 requests per minute, which is twenty a second and a real ceiling for anything multi-tenant, and the docs carry an explicit warning that limits may change without notice while they scale. zero data retention is an enterprise-plan feature. english is the primary training language and their own docs say other scripts, CJK included, are currently less accurate. i have not benchmarked any of this; it is what the documentation states, and it is the sort of thing that looks fine in a demo and matters in production.

key takeaways

a System One model answers multiple choice, not essays: noul, choice, score, in about 100ms, with calibrated confidence attached.
the output cannot be an option you did not supply, which is a structural guarantee rather than a prompting convention.
most real-time LLM features fuse detection and generation, and then the cheap job inherits the expensive job’s cadence. that is why so many of them are secretly a timer.
splitting saves $1/(k+p)$. the trigger rate $p$ dominates; the detector’s price is almost noise.
there is a break-even near a 96% trigger rate. needles in haystacks, not haystacks of needles.
lead with cadence, not cost. cost is the weaker half of a true argument.
the labelled set you need to fit the thresholds is the real project cost.

a personal note

wip …

sources and further reading

TypeSafe documentation index, and the pages under it for System One, primitives, state, confidence, models and pricing and the HTTP API. all specification figures in this note (price, latency, context, rate limits, language support) are read off those pages on 22 September 2026, not measured by me.
speculative fan-out and composite scoring, the two patterns that make the one-request-many-questions shape worth building around.
Anthropic model pricing, for the generation-side anchor used in $k$.
Daniel Kahneman, Thinking, Fast and Slow (2011), where the System 1 and System 2 framing comes from.
my earlier note on what is actually inside an agent harness, which is the same instinct applied one layer up: most of the magic turns out to be plumbing.
the marginal cost note for why a per-tick cost only matters once you multiply it by how many ticks you actually run.

Cost on ENKR's Blog | Jing Hui PANG