Sthānika AI
30 MODELS · 5,405 QUESTIONS · 2,522 CONTEXTS · 11 CATEGORIES
~/research/decisionnativebench

DecisionNativeBench

Twenty-nine open-weight models and hosted Jev 1.13, every one of the 5,405 questions in the evaluation set, each model returning a probability for every option. Jev scores 90.5%; the best open-weight model, Winnow-12B, 86.9%.

Our evaluation set for decision-native models · 29 open-weight models and hosted Jev 1.13 evaluated on all 5,405 questions · Oct 2026

DecisionNativeBench is our own dataset for evaluating decision-native models: models that take a structured case and a set of typed questions and return a probability for every option the caller supplies, instead of generating an answer. Models of this kind — TypeSafe's Jev and a fast-growing set of open alternatives — are being built into software to route requests, approve or hold transactions, rate severity and pick the next tool, where a wrong decision is acted on directly. Public decision suites are mostly assembled from existing classification and question-answering datasets; this one was written to test the thing these models are used for: reading a policy, spotting the exception that applies, and applying the rule to the facts of one case.

DecisionNativeBench contains 5,405 questions over 2,522 synthetic cases in 11 business domains, from customer support and finance to security incidents, evidence assessment and tool selection. Every question is a yes/no judgement, a choice among 4 to 512 options, or a level on a numbered rubric, and every one has a single deterministic gold answer fixed when the case was written, so results are scored by comparison, not by a model acting as judge.

This report publishes the dataset together with its results: 29 open-weight models, including six Sieve models from our own lab, and hosted Jev 1.13, each run with its own serving code and shipped calibration through one common harness and scored the same way.

the task

DecisionNativeBench tests decision-native models: models that do not write an answer but put a probability on each option the caller supplies, the way TypeSafe's Jev and its open alternatives work. Each item is one shared context — a policy, its rules and exceptions, a rating scale, and the facts of one case, as a JSON object — and one to three questions about it, each of a fixed type:

noul · 2,106 questions
A yes/no judgement against the context, answered with P(true). Options are always True / False.
choice · 1,733 questions
One label from an unordered set: an action, a queue, a tool, a defect code. The set has 4 to 512 options (median 8).
score · 1,566 questions
A level on an ordered numeric scale of 3 to 5 levels. The context itself defines what each level means.
one context, several decisions
988 contexts carry three questions, 907 carry two and 627 carry one, so a model has to get several decisions about the same case right at once.

Every model receives the same input: the case's context, unchanged, and its questions, each with the options it may choose from.

Every question has one deterministic gold answer, fixed when the case was written, so scoring is a comparison against it rather than a model-judged rubric.

metrics

accuracy
The model's most probable option equals the gold answer. A question the model could not answer — almost always because it has more options than the model's server accepts — counts as wrong, so this rewards coverage as well as judgement.
NLL
−log of the probability the model gave the gold answer, averaged. It punishes a confident wrong answer far more than a hesitant one; lower is better.
ECE
Expected calibration error: does a model that says 80% turn out right 80% of the time? Lower is better. It matters whenever an application acts on the probability, for example "escalate when p < 0.9".
throughput
Questions answered per second when the 5,405 questions are sent once, one at a time, to the model on its own runtime on one NVIDIA A100 80GB.
GPU cost
What that full run cost on the GPU: its run time at $2.10 per GPU-hour (E2E Networks A100 80GB on-demand, before tax). An estimate, not a bill.
score MAE / within 1
For score questions only: how many levels off the top prediction is, on average, and the share of answers within one level of gold.

the dataset

The evaluation set is 2,522 contexts and 5,405 questions, all synthetic and all in the test split, across 11 business categories and 213 subcategories. Each context is written from a generation specification that fixes the schema and the question mix: a policy with its exceptions, a case with the facts that trigger or defeat them, and a scale that defines every score level. Gold answers are deterministic; 2,034 questions also carry a soft gold distribution, which is used for calibration diagnostics, never for accuracy.

questions by category

categorycontextsquestionssub-catsnoulchoicescore
nli evidence41384715392309146
tool selection2496904247238205
ecommerce orders24057224232172168
customer support23157126228186157
routing classification21553918212186141
finance transactions21552425207161156
security incidents20351124197164150
risk priority severity18645115182139130
quality assurance219349240178171
policy compliance2092092520900
tool action selection1421421300142

choice questions by number of options

choice set sizequestions
4 options623
8 options360
16 options266
32 options200
64 options113
128 options87
256 options57
512 options27
example 1 · ecommerce orders · delivery issue
the context ecommerce_6002 · exactly as every model receives it
{ "slot_rule": "Where a customer is not present for a booked slot, the first missed slot is rebooked at no charge. A second missed slot in the same order carries the failed delivery fee.", "evidence_rule": "A slot counts as missed only where the driver was at the address within the booked window. Arrival outside the window does not count against the customer.", "state_scale": { "1": "One missed slot recorded.", "2": "Two missed slots recorded.", "3": "Attempts exhausted." }, "case": { "slot_1": "Booked 10:00 to 12:00; driver scan at 12:41", "slot_2": "Booked 14:00 to 16:00 on a later day; driver scan at 15:10, no answer", "customer_account": "Says they waited in for the first slot until 12:30", "fee_applied": "Failed delivery fee charged after slot 2", "attempts_permitted": 3 } }
the questions options as supplied; green = gold answer
noul2 options

Did the first attempt fall inside its booked window?

gold: False
True
False
choice4 options

What should happen to the failed delivery fee?

gold: Reverse the Fee
Reverse the Fee
Halve the Fee
Apply the Fee to the Next Attempt
Uphold the Fee
score3 options

What attempt state applies?

gold: 1
1
2
3
example 2 · tool selection · tool constraint
the context tool_selection_10045 · exactly as every model receives it
{ "user_request": "The nightly reconciliation job needs its schedule moved an hour earlier from tomorrow.", "policy": { "schedule_changes": "Reversible and permitted, but the job's owning service is the scheduler rather than the application.", "approval": "Not required for a reversible timing change within the existing window.", "window": "The new time falls within the agreed batch window." }, "state": { "window_check": "Completed; the new time is inside the window.", "owner_notified": "Yes." }, "tool_catalogue": { "naming": "Each tool is named as the service followed by the operation, joined by a space hyphen space.", "services": { "Identity": "User accounts, groups and entitlements.", "Endpoint": "Managed workstations and their agents.", "Network": "Switching, routing, wireless and firewalling.", "Database": "Database instances, schemas and jobs.", "Storage": "File shares, object stores and backups.", "Application": "Deployed business applications.", "Messaging": "Mail and collaboration services.", "Scheduler": "Batch job definitions and executions." }, "operations": { "Read Configuration": "Returns how the object is currently configured.", "Read Audit History": "Returns the record of changes made to the object and by whom.", "Read Health Metrics": "Returns current and recent operational measurements.", "List Entitlements": "Returns who holds what access to the object.", "Create Record": "Creates a new object of that type.", "Update Record": "Changes an existing object, reversibly.", "Restart or Recycle": "Restarts the running instance.", "Request Approval": "Raises an approval request against that service's owner." } }, "notes": { "objective": "To change an existing scheduled definition.", "reversibility": "A schedule can be changed back." }, "risk_policy": { "1": "Retrieval only; nothing is created, changed or sent.", "2": "Retrieval of restricted or personal data, with no change made.", "3": "A change to a record that can be undone.", "4": "A change that is hard to undo, or one that needs approval before it is made." } }
the questions green = gold answer; the choice list is shortened
noul2 options

Does the new time fall within the agreed batch window?

gold: True
True
False
choice64 options

Which tool should be used to handle this request?

gold: Scheduler - Update Record
Database - Read Audit History
Endpoint - Read Configuration
Network - Request Approval
Endpoint - Request Approval
Database - Create Record
Scheduler - Create Record
Scheduler - Update Record
… 57 more
score4 options

What is the action risk score of what this request requires?

gold: 3
1
2
3
4

leaderboard: every run, overall

All 5,405 questions, one run per model — the 29 open-weight models and hosted Jev 1.13 — each at the calibration temperature its authors ship. Accuracy counts an unanswered question as wrong. NLL and ECE score the probabilities a model returns rather than its top answer; lower is better for both.

model size  
#runaccuracy
all 5,405		accuracy by question typeNLLECEnot
answered		throughput
req/s		GPU cost
$
yes/nochoicescore
01Jev 1.13 (hosted)
jev-1.13 · closed
90.5%97.2%84.3%88.3%0.2470.024841.90.16
02Winnow-12B
EldanRing/Winnow-12B · 12B
86.9%96.3%78.3%83.9%0.3350.05217113.10.23
03JevK5-9B
alibiserikbay/JevK5-9B · 9B
86.4%95.9%80.2%80.7%0.4230.01507.20.44
04Sieve-9B-v2 Sthānika
sthanika-ai/Sieve-9B-v2 · 9B
86.1%96.6%78.4%80.6%0.3700.0288416.10.19
05JPT-9B
kirp/jpt-9b · 9B
86.0%95.7%77.5%82.1%0.3980.0208418.00.17
06Sieve-9B Sthānika
sthanika-ai/Sieve-9B · 9B
85.1%95.7%75.0%81.9%0.4200.0388414.90.21
07Kev-9B
jaredpalmer/kev-9b · 9B
84.9%95.7%75.5%80.8%0.4720.0738425.80.12
08JPT-4B
kirp/jpt-4b · 4B
84.7%96.0%75.6%79.6%0.4230.0258418.70.17
09Nimble-9B
bespokelabs/Bespoke-Nimble-9B · 9B
84.7%95.7%77.1%78.2%0.4740.047847.40.42
10Qwen3.5-9B + valence LoRA
joshycodes/Qwen3.5-9B-valence-setpoint-plus5-lora · 9B
83.3%93.7%74.8%78.5%0.4980.02405.10.62
11Sieve-4B Sthānika
sthanika-ai/Sieve-4B · 4B
82.7%95.1%73.7%75.8%0.4650.0428410.60.29
12Lux-9B
llm-semantic-router/Decision-1.0-Lux-9B · 9B
82.0%95.6%72.7%74.1%0.4860.0288413.70.23
13decider-4b
Mapika/decider-4b · 4B
82.0%93.5%74.8%74.5%0.4640.0228416.90.18
14Kev-4B
jaredpalmer/kev-4b · 4B
81.4%95.4%68.8%76.6%0.5650.0958434.30.09
15Lev1-3.7B
theaviv/Lev1-3.7B · 3.7B
80.3%94.0%72.1%71.0%0.4880.0268414.90.21
16Jev-Style 2B v3
chaoliangUNSW/Jev-Style-2B-Decision-v3 · 2B
75.3%90.9%66.4%64.0%0.6550.043848.80.35
17Nox-4B
llm-semantic-router/Decision-1.0-Nox-4B · 4B
71.7%88.7%59.7%62.0%0.7750.0648414.80.21
18Sieve-2B-v2 Sthānika
sthanika-ai/Sieve-2B-v2 · 2B
70.9%88.3%61.5%57.9%0.8140.03009.30.34
19Strands-decider 2B
StrandsAgents/strands-decider-2B-hobson-v19 · 2B
70.0%87.8%61.0%56.1%0.8240.1028410.30.30
20this-that-model-1.2
flock-io/this-that-model-1.2 · 2B
68.5%87.8%59.2%52.9%1.1150.1348413.20.24
21Sieve-2B Sthānika
sthanika-ai/sieve-2b · 2B
68.5%86.9%57.6%55.6%0.8560.028010.20.31
22Sieve-2B-v3 Sthānika
sthanika-ai/Sieve-2B-v3 · 2B
67.9%85.0%56.7%57.4%0.8150.0228425.70.12
23EXAONE-4.0-1.2B-JEV
carrtesy/EXAONE-4.0-1.2B-JEV · 1.2B
67.8%86.1%54.6%58.0%0.8120.0288425.80.12
24Jiwo-0.8B
eljiwo/jiwo-0.8b · 0.8B
67.0%84.0%57.8%54.3%0.8650.0378420.80.15
25Kodiak-v0.2-1B
cortex-agent-llc/kodiak-v0.2-1b · 1B
60.3%81.6%49.1%44.1%1.0880.096022.70.14
26Decision-2.0-Sol-2B
vllm-sr/Decision-2.0-Sol-2B · 2B
59.3%82.2%36.7%53.7%1.1110.0708420.10.15
27Bosun 3.1 1.7B
Hanno-Labs/bosun-v3.1-1.7b · 1.7B
53.2%67.5%47.7%40.0%1.2300.1458416.50.19
28CLM-v0.1-8B
Contrastive-LM/CLM-v0.1-8B · 8B
47.8%84.3%18.2%31.6%1.9730.19109.60.33
29LFM2.5-2.6B-RLCD
monotykamary/LFM2.5-2.6B-RLCD · 2.6B
47.4%61.8%31.0%46.0%1.4670.1588416.60.19
30Laya typed-decisions
convaiinnovations/laya-typed-decisions · 0.4B
44.9%63.1%31.6%35.1%1.2290.02717128.60.11
Unanswered questions have more options than a model accepts and are counted as wrong.

accuracy against speed and cost

One dot per model, from the same run of all 5,405 questions behind the leaderboard. The open-weight models ran one request at a time on one NVIDIA A100 80GB. Jev 1.13 (square) is a hosted API called through OpenRouter, so its numbers include the network: its throughput and latency are timed one request at a time like the others, but over the internet, and its cost is the API bill rather than GPU time. The latency chart uses a log scale so that Jev's 499 ms fits beside the open models. Up and to the right is better on the first chart, up and to the left on the other two. Hover a dot for its exact numbers.

open-weight modelSthānika modelhosted API (Jev 1.13)

accuracy vs throughput

40%50%60%70%80%90%010203040throughput (questions per second, higher is better) →accuracyWinnow-12B — accuracy 86.9% · throughput 13.1 req/s · GPU cost $0.23 · median latency 83 msJevK5-9B — accuracy 86.4% · throughput 7.2 req/s · GPU cost $0.44 · median latency 59 msJPT-9B — accuracy 86.0% · throughput 18.0 req/s · GPU cost $0.17 · median latency 46 msJPT-4B — accuracy 84.7% · throughput 18.7 req/s · GPU cost $0.17 · median latency 45 msKev-9B — accuracy 84.9% · throughput 25.8 req/s · GPU cost $0.12 · median latency 38 msNimble-9B — accuracy 84.7% · throughput 7.4 req/s · GPU cost $0.42 · median latency 108 msQwen3.5-9B + valence LoRA — accuracy 83.3% · throughput 5.1 req/s · GPU cost $0.62 · median latency 134 msLux-9B — accuracy 82.0% · throughput 13.7 req/s · GPU cost $0.23 · median latency 59 msdecider-4b — accuracy 82.0% · throughput 16.9 req/s · GPU cost $0.18 · median latency 41 msKev-4B — accuracy 81.4% · throughput 34.3 req/s · GPU cost $0.09 · median latency 27 msLev1-3.7B — accuracy 80.3% · throughput 14.9 req/s · GPU cost $0.21 · median latency 47 msJev-Style 2B v3 — accuracy 75.3% · throughput 8.8 req/s · GPU cost $0.35 · median latency 93 msNox-4B — accuracy 71.7% · throughput 14.8 req/s · GPU cost $0.21 · median latency 58 msStrands-decider 2B — accuracy 70.0% · throughput 10.3 req/s · GPU cost $0.30 · median latency 93 msthis-that-model-1.2 — accuracy 68.5% · throughput 13.2 req/s · GPU cost $0.24 · median latency 68 msEXAONE-4.0-1.2B-JEV — accuracy 67.8% · throughput 25.8 req/s · GPU cost $0.12 · median latency 31 msJiwo-0.8B — accuracy 67.0% · throughput 20.8 req/s · GPU cost $0.15 · median latency 47 msKodiak-v0.2-1B — accuracy 60.3% · throughput 22.7 req/s · GPU cost $0.14 · median latency 23 msDecision-2.0-Sol-2B — accuracy 59.3% · throughput 20.1 req/s · GPU cost $0.15 · median latency 43 msBosun 3.1 1.7B — accuracy 53.2% · throughput 16.5 req/s · GPU cost $0.19 · median latency 56 msCLM-v0.1-8B — accuracy 47.8% · throughput 9.6 req/s · GPU cost $0.33 · median latency 84 msLFM2.5-2.6B-RLCD — accuracy 47.4% · throughput 16.6 req/s · GPU cost $0.19 · median latency 54 msLaya typed-decisions — accuracy 44.9% · throughput 28.6 req/s · GPU cost $0.11 · median latency 28 msSieve-9B-v2 — accuracy 86.1% · throughput 16.1 req/s · GPU cost $0.19 · median latency 78 msSieve-9B — accuracy 85.1% · throughput 14.9 req/s · GPU cost $0.21 · median latency 81 msSieve-4B — accuracy 82.7% · throughput 10.6 req/s · GPU cost $0.29 · median latency 75 msSieve-2B-v2 — accuracy 70.9% · throughput 9.3 req/s · GPU cost $0.34 · median latency 90 msSieve-2B — accuracy 68.5% · throughput 10.2 req/s · GPU cost $0.31 · median latency 82 msSieve-2B-v3 — accuracy 67.9% · throughput 25.7 req/s · GPU cost $0.12 · median latency 35 msJev 1.13 (hosted) — accuracy 90.5% · throughput 1.9 req/s (one request at a time, network included) · API cost $0.16 (OpenRouter bill) · median latency 499 ms incl. networkSieve-9B-v2Sieve-9Bdecider-4bLux-9BLev1-3.7BStrands 2BJPT-9BJPT-4BSieve-2B-v2this-that-1.2Sieve-2BWinnow-12BJevK5-9BNimble-9Bvalence LoRAJev-Style 2BNox-4BEXAONE-1.2BKodiak 1BDecision-2.0-SolBosun 1.7BLFM 2.6BSieve-2B-v3Kev-9BSieve-4BKev-4BJiwo-0.8BCLM-8BLayaJev 1.13 (hosted)

accuracy vs GPU cost

40%50%60%70%80%90%$0.0$0.2$0.4$0.6$0.8GPU cost of the full benchmark run (lower is better) ←accuracyWinnow-12B — accuracy 86.9% · throughput 13.1 req/s · GPU cost $0.23 · median latency 83 msJevK5-9B — accuracy 86.4% · throughput 7.2 req/s · GPU cost $0.44 · median latency 59 msJPT-9B — accuracy 86.0% · throughput 18.0 req/s · GPU cost $0.17 · median latency 46 msJPT-4B — accuracy 84.7% · throughput 18.7 req/s · GPU cost $0.17 · median latency 45 msKev-9B — accuracy 84.9% · throughput 25.8 req/s · GPU cost $0.12 · median latency 38 msNimble-9B — accuracy 84.7% · throughput 7.4 req/s · GPU cost $0.42 · median latency 108 msQwen3.5-9B + valence LoRA — accuracy 83.3% · throughput 5.1 req/s · GPU cost $0.62 · median latency 134 msLux-9B — accuracy 82.0% · throughput 13.7 req/s · GPU cost $0.23 · median latency 59 msdecider-4b — accuracy 82.0% · throughput 16.9 req/s · GPU cost $0.18 · median latency 41 msKev-4B — accuracy 81.4% · throughput 34.3 req/s · GPU cost $0.09 · median latency 27 msLev1-3.7B — accuracy 80.3% · throughput 14.9 req/s · GPU cost $0.21 · median latency 47 msJev-Style 2B v3 — accuracy 75.3% · throughput 8.8 req/s · GPU cost $0.35 · median latency 93 msNox-4B — accuracy 71.7% · throughput 14.8 req/s · GPU cost $0.21 · median latency 58 msStrands-decider 2B — accuracy 70.0% · throughput 10.3 req/s · GPU cost $0.30 · median latency 93 msthis-that-model-1.2 — accuracy 68.5% · throughput 13.2 req/s · GPU cost $0.24 · median latency 68 msEXAONE-4.0-1.2B-JEV — accuracy 67.8% · throughput 25.8 req/s · GPU cost $0.12 · median latency 31 msJiwo-0.8B — accuracy 67.0% · throughput 20.8 req/s · GPU cost $0.15 · median latency 47 msKodiak-v0.2-1B — accuracy 60.3% · throughput 22.7 req/s · GPU cost $0.14 · median latency 23 msDecision-2.0-Sol-2B — accuracy 59.3% · throughput 20.1 req/s · GPU cost $0.15 · median latency 43 msBosun 3.1 1.7B — accuracy 53.2% · throughput 16.5 req/s · GPU cost $0.19 · median latency 56 msCLM-v0.1-8B — accuracy 47.8% · throughput 9.6 req/s · GPU cost $0.33 · median latency 84 msLFM2.5-2.6B-RLCD — accuracy 47.4% · throughput 16.6 req/s · GPU cost $0.19 · median latency 54 msLaya typed-decisions — accuracy 44.9% · throughput 28.6 req/s · GPU cost $0.11 · median latency 28 msSieve-9B-v2 — accuracy 86.1% · throughput 16.1 req/s · GPU cost $0.19 · median latency 78 msSieve-9B — accuracy 85.1% · throughput 14.9 req/s · GPU cost $0.21 · median latency 81 msSieve-4B — accuracy 82.7% · throughput 10.6 req/s · GPU cost $0.29 · median latency 75 msSieve-2B-v2 — accuracy 70.9% · throughput 9.3 req/s · GPU cost $0.34 · median latency 90 msSieve-2B — accuracy 68.5% · throughput 10.2 req/s · GPU cost $0.31 · median latency 82 msSieve-2B-v3 — accuracy 67.9% · throughput 25.7 req/s · GPU cost $0.12 · median latency 35 msJev 1.13 (hosted) — accuracy 90.5% · throughput 1.9 req/s (one request at a time, network included) · API cost $0.16 (OpenRouter bill) · median latency 499 ms incl. networkSieve-9B-v2JPT-9BSieve-9BJPT-4Bdecider-4bLux-9BLev1-3.7BWinnow-12BKev-9BSieve-2B-v2Jiwo-0.8BJev 1.13 (hosted)Strands 2BSieve-2BEXAONE-1.2BKodiak 1BSieve-2B-v3JevK5-9BNimble-9BKev-4BJev-Style 2BNox-4Bthis-that-1.2Decision-2.0-SolBosun 1.7BLFM 2.6Bvalence LoRASieve-4BCLM-8BLaya

accuracy vs latency

40%50%60%70%80%90%2050100200500median latency per question, ms, log scale (lower is better) ←accuracyWinnow-12B — accuracy 86.9% · throughput 13.1 req/s · GPU cost $0.23 · median latency 83 msJevK5-9B — accuracy 86.4% · throughput 7.2 req/s · GPU cost $0.44 · median latency 59 msJPT-9B — accuracy 86.0% · throughput 18.0 req/s · GPU cost $0.17 · median latency 46 msJPT-4B — accuracy 84.7% · throughput 18.7 req/s · GPU cost $0.17 · median latency 45 msKev-9B — accuracy 84.9% · throughput 25.8 req/s · GPU cost $0.12 · median latency 38 msNimble-9B — accuracy 84.7% · throughput 7.4 req/s · GPU cost $0.42 · median latency 108 msQwen3.5-9B + valence LoRA — accuracy 83.3% · throughput 5.1 req/s · GPU cost $0.62 · median latency 134 msLux-9B — accuracy 82.0% · throughput 13.7 req/s · GPU cost $0.23 · median latency 59 msdecider-4b — accuracy 82.0% · throughput 16.9 req/s · GPU cost $0.18 · median latency 41 msKev-4B — accuracy 81.4% · throughput 34.3 req/s · GPU cost $0.09 · median latency 27 msLev1-3.7B — accuracy 80.3% · throughput 14.9 req/s · GPU cost $0.21 · median latency 47 msJev-Style 2B v3 — accuracy 75.3% · throughput 8.8 req/s · GPU cost $0.35 · median latency 93 msNox-4B — accuracy 71.7% · throughput 14.8 req/s · GPU cost $0.21 · median latency 58 msStrands-decider 2B — accuracy 70.0% · throughput 10.3 req/s · GPU cost $0.30 · median latency 93 msthis-that-model-1.2 — accuracy 68.5% · throughput 13.2 req/s · GPU cost $0.24 · median latency 68 msEXAONE-4.0-1.2B-JEV — accuracy 67.8% · throughput 25.8 req/s · GPU cost $0.12 · median latency 31 msJiwo-0.8B — accuracy 67.0% · throughput 20.8 req/s · GPU cost $0.15 · median latency 47 msKodiak-v0.2-1B — accuracy 60.3% · throughput 22.7 req/s · GPU cost $0.14 · median latency 23 msDecision-2.0-Sol-2B — accuracy 59.3% · throughput 20.1 req/s · GPU cost $0.15 · median latency 43 msBosun 3.1 1.7B — accuracy 53.2% · throughput 16.5 req/s · GPU cost $0.19 · median latency 56 msCLM-v0.1-8B — accuracy 47.8% · throughput 9.6 req/s · GPU cost $0.33 · median latency 84 msLFM2.5-2.6B-RLCD — accuracy 47.4% · throughput 16.6 req/s · GPU cost $0.19 · median latency 54 msLaya typed-decisions — accuracy 44.9% · throughput 28.6 req/s · GPU cost $0.11 · median latency 28 msSieve-9B-v2 — accuracy 86.1% · throughput 16.1 req/s · GPU cost $0.19 · median latency 78 msSieve-9B — accuracy 85.1% · throughput 14.9 req/s · GPU cost $0.21 · median latency 81 msSieve-4B — accuracy 82.7% · throughput 10.6 req/s · GPU cost $0.29 · median latency 75 msSieve-2B-v2 — accuracy 70.9% · throughput 9.3 req/s · GPU cost $0.34 · median latency 90 msSieve-2B — accuracy 68.5% · throughput 10.2 req/s · GPU cost $0.31 · median latency 82 msSieve-2B-v3 — accuracy 67.9% · throughput 25.7 req/s · GPU cost $0.12 · median latency 35 msJev 1.13 (hosted) — accuracy 90.5% · throughput 1.9 req/s (one request at a time, network included) · API cost $0.16 (OpenRouter bill) · median latency 499 ms incl. networkJPT-9BJPT-4BSieve-4Bdecider-4bLev1-3.7BWinnow-12BSieve-9B-v2Sieve-9BKev-9BLux-9BSieve-2B-v2Strands 2BSieve-2BJevK5-9BJev-Style 2Bthis-that-1.2Nimble-9Bvalence LoRANox-4BEXAONE-1.2BBosun 1.7BLFM 2.6BSieve-2B-v3Kev-4BJiwo-0.8BKodiak 1BDecision-2.0-SolCLM-8BLayaJev 1.13 (hosted)
Jev 1.13 (the square at top right) is the only model reached over the internet. Its 499 ms median is what a caller sees, and much of it is network and service overhead (opening a connection, the trip to OpenRouter and on to TypeSafe's servers, and back) rather than the model's own compute. Jev's compute time is not published and cannot be measured from outside, so its position here is not a measure of how fast the model itself is.

yes/no against choice

Accuracy on the 2,106 yes/no questions and the 1,733 choice questions, which share the same contexts. Each pair of dots is one run; the bar between them is how much harder picking the right option is than confirming a single condition in the same case.

yes/no (noul)choice
0.00.20.40.60.81.0accuracy on the same 2,522 contextsJev 1.13 (hosted)Jev 1.13 (hosted) — choice 0.843Jev 1.13 (hosted) — yes/no 0.9720.8430.972Winnow-12BWinnow-12B — choice 0.783Winnow-12B — yes/no 0.9630.7830.963JevK5-9BJevK5-9B — choice 0.802JevK5-9B — yes/no 0.9590.8020.959Sieve-9B-v2Sieve-9B-v2 — choice 0.784Sieve-9B-v2 — yes/no 0.9660.7840.966JPT-9BJPT-9B — choice 0.775JPT-9B — yes/no 0.9570.7750.957Sieve-9BSieve-9B — choice 0.750Sieve-9B — yes/no 0.9570.7500.957Kev-9BKev-9B — choice 0.755Kev-9B — yes/no 0.9570.7550.957JPT-4BJPT-4B — choice 0.756JPT-4B — yes/no 0.9600.7560.960Nimble-9BNimble-9B — choice 0.771Nimble-9B — yes/no 0.9570.7710.957Qwen3.5-9B + valence LoRAQwen3.5-9B + valence LoRA — choice 0.748Qwen3.5-9B + valence LoRA — yes/no 0.9370.7480.937Sieve-4BSieve-4B — choice 0.737Sieve-4B — yes/no 0.9510.7370.951Lux-9BLux-9B — choice 0.727Lux-9B — yes/no 0.9560.7270.956decider-4bdecider-4b — choice 0.748decider-4b — yes/no 0.9350.7480.935Kev-4BKev-4B — choice 0.688Kev-4B — yes/no 0.9540.6880.954Lev1-3.7BLev1-3.7B — choice 0.721Lev1-3.7B — yes/no 0.9400.7210.940Jev-Style 2B v3Jev-Style 2B v3 — choice 0.664Jev-Style 2B v3 — yes/no 0.9090.6640.909Nox-4BNox-4B — choice 0.597Nox-4B — yes/no 0.8870.5970.887Sieve-2B-v2Sieve-2B-v2 — choice 0.615Sieve-2B-v2 — yes/no 0.8830.6150.883Strands-decider 2BStrands-decider 2B — choice 0.610Strands-decider 2B — yes/no 0.8780.6100.878this-that-model-1.2this-that-model-1.2 — choice 0.592this-that-model-1.2 — yes/no 0.8780.5920.878Sieve-2BSieve-2B — choice 0.577Sieve-2B — yes/no 0.8690.5770.869Sieve-2B-v3Sieve-2B-v3 — choice 0.567Sieve-2B-v3 — yes/no 0.8500.5670.850EXAONE-4.0-1.2B-JEVEXAONE-4.0-1.2B-JEV — choice 0.546EXAONE-4.0-1.2B-JEV — yes/no 0.8610.5460.861Jiwo-0.8BJiwo-0.8B — choice 0.578Jiwo-0.8B — yes/no 0.8400.5780.840Kodiak-v0.2-1BKodiak-v0.2-1B — choice 0.491Kodiak-v0.2-1B — yes/no 0.8160.4910.816Decision-2.0-Sol-2BDecision-2.0-Sol-2B — choice 0.367Decision-2.0-Sol-2B — yes/no 0.8220.3670.822Bosun 3.1 1.7BBosun 3.1 1.7B — choice 0.477Bosun 3.1 1.7B — yes/no 0.6750.4770.675CLM-v0.1-8BCLM-v0.1-8B — choice 0.182CLM-v0.1-8B — yes/no 0.8430.1820.843LFM2.5-2.6B-RLCDLFM2.5-2.6B-RLCD — choice 0.310LFM2.5-2.6B-RLCD — yes/no 0.6180.3100.618Laya typed-decisionsLaya typed-decisions — choice 0.316Laya typed-decisions — yes/no 0.6310.3160.631

credit & links

Every model in this report is the work of its authors. The six Sieve models — Sieve-9B-v2, Sieve-9B, Sieve-4B, Sieve-2B-v3, Sieve-2B-v2 and Sieve-2B — are from our lab, Sthānika AI, and were evaluated exactly like the others: the same harness, the same inputs and the same scoring, with no setting specific to them.

The other models and their authors: Jev 1.13 (TypeSafe AI), Winnow-12B (EldanRing), JevK5-9B (alibiserikbay), JPT-4B and JPT-9B (kirp), Kev-4B and Kev-9B (jaredpalmer), Bespoke-Nimble-9B (Bespoke Labs), the Qwen3.5-9B valence LoRA (joshycodes), Decision-1.0 Lux-9B and Nox-4B and Decision-2.0 Sol-2B (vLLM Semantic Router), decider-4b (Mapika), Lev1-3.7B (theaviv), Jev-Style 2B (chaoliangUNSW), Strands-decider 2B (Strands Agents), this-that-model-1.2 (FLock.io), EXAONE-4.0-1.2B-JEV (carrtesy, built on LG AI Research's EXAONE), Jiwo-0.8B (eljiwo), Kodiak-v0.2-1B (Cortex Agent), Bosun 1.7B (Hanno Labs), CLM-v0.1-8B (Contrastive-LM), LFM2.5-2.6B-RLCD (monotykamary, built on Liquid AI's LFM2.5) and Laya (convai innovations). Each was run with its authors' own runtime and shipped calibration, at a fixed published checkpoint. Lux-9B and Nox-4B ship a runtime profile validated on AMD GPUs; they ran here on NVIDIA with that profile step skipped and everything else as shipped.

Jev 1.13 is a hosted model, accessed through OpenRouter; the typed-decision interface used throughout follows Jev's. GPU costs use E2E Networks' public on-demand rate for an NVIDIA A100 80GB. The DecisionNativeBench dataset, the evaluation harness, and every run, timing and cost in this report are by Sthānika AI.

SthānikaSieve-9B-v2 ↗Sieve-9B ↗Sieve-4B ↗Sieve-2B-v3 ↗Sieve-2B-v2 ↗Sieve-2B ↗

Jev 1.13 ↗Winnow-12B ↗JevK5-9B ↗JPT-9B ↗JPT-4B ↗Kev-9B ↗Kev-4B ↗Nimble-9B ↗valence LoRA ↗Lux-9B ↗Nox-4B ↗Decision-2.0-Sol-2B ↗decider-4b ↗Lev1-3.7B ↗Jev-Style 2B ↗Strands-decider 2B ↗this-that-1.2 ↗EXAONE-4.0-1.2B-JEV ↗Jiwo-0.8B ↗Kodiak-v0.2-1B ↗Bosun 1.7B ↗CLM-v0.1-8B ↗LFM2.5-2.6B-RLCD ↗Laya ↗

Sthānika AI · an initiative of PurpleTalk · HyderabadDecisionNativeBench — typed-decision evaluation dataset, Oct 2026