WE BENCHMARKED 19 MODELS · BKP-500 FULL RUNS (R1+R2, n=3 SAMPLES) · 552 ITEMS EACH

~/research/bkp500-2026

BKP-500, the 2026 model run report

We ran 19 models against BKP-500 — 552 India-specific items across 7 categories (numerals, weights/volumes, land units, agricultural seasons, fiscal-year conventions, government schemes, structural identifiers), under two prompt regimes and 3 repeated samples per item. Qwen3.6 27B leads the roster at 53.2% Bharat Score; Sarvam-M 24B is the strongest India-built/tuned model at 45.2%. The single weakest spot for the roster as a whole is Land units by state — the worst-scoring category for 14 of 19 models.

evaluation harness (GitHub) · model run configs & leaderboard (GitHub) · BKP-500 dataset (Hugging Face)

~/research/bkp500-2026

BKP-500, the 2026 model run.

Research report · 2026-09-21

The BKP-500 2026 model run scored 19 open-weight language models on 552 India-specific items across seven categories, with Sarvam-M 24B the strongest India-built model at 45.2% Bharat Score, third overall.

We scored 19 models · 552 items · 2 prompt regimes × 3 samples = 3312 rows/model · run on the BKP evaluation pipeline

About this benchmark: how BKP-500 items are constructed, how Bharat Score is computed, and where to download the corpus.

summary

BKP-500 tests whether a model actually knows India-specific conventions — lakh/crore numerals, informal weights (tola, seer, chhatak), land units that vary by state, kharif/rabi/zaid crop calendars, the April–March fiscal year, and government-scheme facts — rather than defaulting to the international convention that dominates most pretraining data.

Qwen3.6 27B leads the roster at 53.2% Bharat Score (95% CI 48.9–57.3%), a global open-weight model, not an India-built one. Sarvam-M 24B is the strongest India-built/tuned model at 45.2%, landing at rank 3 overall. Across the full roster, global open-weight models average 35.2% against India-built models' 23.3% — a 11.9-point gap, though with only 8 and 11 models per bucket that comparison's own confidence interval is wide (see the bucket comparison below). Land units by state is the hardest category for the roster as a whole: the single lowest-scoring category for 14 of 19 models, well ahead of any other category.

key insights

The top three ranks are all real, not noise — then an 8-model plateau.
Qwen3.6 27B's lead over gpt-oss-20b (6.3 points of Bharat Score) and gpt-oss-20b's lead over Sarvam-M 24B (1.7 points) both survive Holm-Bonferroni correction across every adjacent-rank test at once (0.05 family-wise alpha). From rank 3 (Sarvam-M 24B) through rank 10 (Gemma 3 12B), no adjacent pair is statistically distinguishable — a genuine 8-model plateau. It doesn't hold for the rest of tier 1, though: 3 more Holm-significant breaks punctuate the remaining ranks (Gemma 3 12B→Llama 3.1 8B Instruct, Qwen2.5 7B→Gemma 3 4B, Gemma 3 4B→Navarasa 2.0 7B) before the tier 1/tier 2 cliff itself (see below).
An India-built model holds rank 3 outright.
Sarvam-M 24B scores 45.2%, tier 1 — 1.7 points behind #2 gpt-oss-20b (a Holm-significant gap), 3.3 points ahead of #4 Gemma 3 27B (not a Holm-significant gap). It shares tier 1's upper half with 1 other India-built/tuned model (Krutrim-2 12B).
Land units by state breaks almost every model, regardless of scale or origin.
Land units by state is the worst-scoring category for 14 of 19 models — the next most common weak spot is a distant runner-up. State-by-state variation in land-unit definitions (a bigha in Punjab isn't a bigha in Bihar) appears to be genuinely harder for every model family we tested than any other category in the corpus.
The tier 1/tier 2 cliff is a real, significant gap.
Navarasa 2.0 7B (20.6%, tier 1) and Param-1 2.9B (14.3%, tier 2) sit only 6.3 points apart in raw score, but that gap is Holm-significant (p<0.001) — tier 2 isn't an artifact of the tiering method, it's a real capability floor 5 of 19 models fall below.
Wrong answers are consistent, not noisy.
Consistency — agreement across a model's own repeated samples — sits at 86.7–100.0% across the entire roster, regardless of Bharat Score. A model scoring 5% and one scoring 50% are both answering the same way every time they're asked — low accuracy here means confidently, repeatably wrong, not random guessing.

Also worth noting

Samples are near-deterministic in practice: 77.0% of multi-sample groups (16,162 of 20,976) returned byte-identical text across all 3 repeated samples at temperature 0, as configured.
552 of 552 items in the current corpus (100.0%) have been through the project's two-reviewer human QA pass: reviewers flagged 57 with a gold-value problem, 41 of those corrections are already applied to the live corpus, and 16 remain open (mostly `clarification_required` items where the reviewers proposed a single scalar value against a deliberately multi-source-conflicting convention — left as clarification pending resolution, not auto-overridden). This itemized log is the issue-by-issue detail behind the same review the corpus's own adjudicated/split fields record at the corpus level, which has landed for all 552 of 552 items (see methodology below).

findings so far

INT4-quantizing Gemma 3 12B doesn't cost a few points — it erases the model.

Gemma 3 12B scores 35.1% (rank 10 of 19). The same model, quantized to INT4, scores 4.6% — dead last, rank 19 of 19, below even Gemma 3 4B (25.3%). Its refusal rate jumps to 91.3% and it scores a flat 0.0% on 5 of the 7 categories — this reads as a broken quantization, not a graceful capability trade-off. Read the note: what INT4 quantization actually costs →

The 3 best models take the biggest hit from strict JSON formatting.

Qwen3.6 27B loses 20.7 points going from natural-language (R2) to strict-JSON (R1) answers, Sarvam-M 24B loses 17.0, gpt-oss-20b loses 14.0 — and these are exactly the roster's top 3 ranks by Bharat Score. Meanwhile Airavata 7B actually gains 7.8 points under strict JSON. Format compliance and raw capability pull in opposite directions here: the strongest models seem to lose more to answer-extraction friction, not less.

The best unit discipline in the roster belongs to India-built models.

Sarvam 30B, Sarvam-M 24B and Gemma 3 27B post the roster's top 3 Unit Discipline scores (90.5%, 89.1%, 85.3%) — all three India-built/tuned, despite mixed overall ranks (9, 3, 4 of 19). Gemma 3 12B INT4 and Airavata 7B trail furthest behind (2.0%, 14.5%) — both are also among the roster's highest-refusal models, so a chunk of their missing units is likely missing answers, not wrong ones.

Qwen's generation-then-scale jump is the cleanest improvement story in the roster.

Qwen2.5 7B (30.9%, rank 12) → Qwen3-VL 8B (39.2%, rank 5, +8.3pp at near-identical size) is a generational jump, not a scale one. Qwen3.6 27B (53.2%, rank 1) then adds real scale on top, landing as the roster's outright leader. Every step in this family's chain moved in the same direction — no other model family in the roster shows as clean a progression.

Indian numeral system shows the widest India-built-vs-global gap; Agricultural seasons & crop calendars the narrowest.

Global open-weight models lead India-built/tuned models by 20.6 points on Indian numeral system and 17.3 on Weights, volumes & informal measures — the two categories built on precise lakh/crore-scale arithmetic. The gap narrows to 5.8 points on Agricultural seasons & crop calendars, the smallest of any category. The India-built/tuned bucket doesn't trail evenly across categories: it closes ground fastest on categories that reward cultural/domain familiarity over precise numeric conversion, and falls furthest behind exactly where scale arithmetic dominates.

league table

Ranked by Bharat Score (macro-mean accuracy across all 7 categories), tiered by chained bootstrap-CI overlap.

tier	model	bharat score (95% CI)	oom error rate	unit discipline	refusal rate	consistency
1	Qwen3.6 27B global	53.2% (48.9-57.3)	10.6%	84.9%	0.7%	100.0%
1	gpt-oss-20b global	46.9% (43.0-50.9)	16.5%	84.6%	0.1%	87.6%
1	Sarvam-M 24B india	45.2% (40.9-49.2)	22.2%	89.1%	3.1%	90.7%
1	Gemma 3 27B global	41.8% (38.3-45.8)	18.4%	85.3%	0.0%	99.6%
1	Qwen3-VL 8B global	39.2% (35.1-43.4)	20.5%	80.8%	0.0%	97.1%
1	Mistral Small 3.1 24B global	38.8% (35.0-42.9)	17.1%	68.8%	0.6%	97.7%
1	Krutrim-2 12B india	38.7% (34.6-43.3)	24.1%	82.3%	0.2%	99.5%
1	Phi-4 14B global	38.7% (34.8-43.0)	21.2%	81.6%	2.1%	99.0%
1	Sarvam 30B india	37.1% (33.5-40.9)	33.7%	90.5%	1.3%	86.7%
1	Gemma 3 12B global	35.1% (31.3-39.4)	25.6%	83.2%	0.0%	99.6%
1	Llama 3.1 8B Instruct global	32.6% (28.5-36.7)	32.9%	77.9%	2.9%	100.0%
1	Qwen2.5 7B global	30.9% (27.1-34.8)	23.1%	82.9%	0.1%	97.9%
1	Gemma 3 4B global	25.3% (22.1-28.8)	33.7%	75.9%	0.0%	100.0%
1	Navarasa 2.0 7B india	20.6% (17.8-23.6)	44.3%	74.7%	0.6%	93.1%
2	Param-1 2.9B india	14.3% (11.8-16.9)	54.2%	60.4%	0.6%	99.7%
2	Sarvam-1 2B india	11.6% (9.1-14.8)	56.7%	47.5%	1.5%	99.7%
2	OpenHathi 7B india	11.2% (8.7-14.2)	52.3%	65.8%	0.1%	96.1%
2	Airavata 7B india	7.9% (5.6-10.4)	22.7%	14.5%	38.5%	99.9%
2	Gemma 3 12B INT4 global	4.6% (2.4-7.2)	1.4%	2.0%	91.3%	100.0%

OOM error rate = share of numeric answers off by ≥10× (a lakh/crore-scale slip). Unit discipline = share of numeric_with_unit answers that included an explicit, correct unit. n_resamples=2,000 for every bootstrap CI on this page (the published leaderboard uses 10,000 for the publication-grade run).

per-category accuracy — all 19 models × 7 categories

Every model, every category, no truncation — ranked top to bottom by Bharat Score. Fill shade scales with accuracy so the weak column (land units, almost throughout) reads at a glance; the exact percentage is always printed alongside it.

model	numerals	weights/vol	land units	seasons	fiscal yr	schemes	identifiers
Qwen3.6 27B global	77.9	61.1	39.4	48.9	52.9	41.9	50.0
gpt-oss-20b global	78.7	57.6	34.9	41.2	46.9	25.4	43.3
Sarvam-M 24B india	46.6	55.8	21.5	54.6	49.0	36.5	52.2
Gemma 3 27B global	54.0	54.9	26.4	38.5	50.6	36.0	32.2
Qwen3-VL 8B global	55.4	39.4	31.2	42.1	39.8	29.7	36.7
Mistral Small 3.1 24B global	50.6	59.0	26.1	42.9	32.2	31.1	30.0
Krutrim-2 12B india	49.2	39.6	24.5	44.7	42.9	39.9	30.0
Phi-4 14B global	46.1	53.5	24.2	42.1	46.1	32.2	26.7
Sarvam 30B india	40.7	51.2	28.8	51.1	33.1	27.9	26.7
Gemma 3 12B global	48.8	40.7	20.8	41.9	37.8	29.0	26.7
Llama 3.1 8B Instruct global	32.0	25.7	17.3	39.6	38.2	35.1	40.0
Qwen2.5 7B global	44.6	37.7	17.5	32.4	34.5	16.0	33.3
Gemma 3 4B global	32.9	24.3	12.5	34.6	31.2	18.2	23.3
Navarasa 2.0 7B india	26.1	21.5	10.3	25.1	30.4	17.6	13.3
Param-1 2.9B india	15.2	11.1	3.5	25.8	15.3	15.5	13.3
Sarvam-1 2B india	16.2	2.8	2.4	18.5	12.8	14.9	13.3
OpenHathi 7B india	13.1	4.9	2.2	15.4	16.7	8.8	17.8
Airavata 7B india	6.9	4.9	1.4	12.6	11.8	4.0	13.3
Gemma 3 12B INT4 global	0.0	0.0	0.0	0.0	0.0	12.2	20.0

Land units by state is the single weakest category for 14 of 19 models in the full roster.

pairwise significance — every adjacent leaderboard rank

Paired bootstrap test between each rank and the one directly below it, Holm-Bonferroni corrected across all 18 tests at once (0.05 family-wise alpha) — the "significant" column, not the raw p-value alone, is the real "these two models differ" claim. 8 of 18 adjacent pairs survive correction.

rank	next rank	mean diff	p-value	Holm-corrected
Qwen3.6 27B	gpt-oss-20b	+5.7pp	<0.001	significant
gpt-oss-20b	Sarvam-M 24B	+5.0pp	0.001	significant
Sarvam-M 24B	Gemma 3 27B	+0.7pp	0.66	not significant
Gemma 3 27B	Qwen3-VL 8B	+2.6pp	0.068	not significant
Qwen3-VL 8B	Mistral Small 3.1 24B	+0.4pp	0.781	not significant
Mistral Small 3.1 24B	Krutrim-2 12B	-0.0pp	0.993	not significant
Krutrim-2 12B	Phi-4 14B	-0.1pp	0.957	not significant
Phi-4 14B	Sarvam 30B	+1.6pp	0.334	not significant
Sarvam 30B	Gemma 3 12B	+1.9pp	0.26	not significant
Gemma 3 12B	Llama 3.1 8B Instruct	+5.2pp	0.001	significant
Llama 3.1 8B Instruct	Qwen2.5 7B	+0.4pp	0.814	not significant
Qwen2.5 7B	Gemma 3 4B	+5.2pp	<0.001	significant
Gemma 3 4B	Navarasa 2.0 7B	+4.2pp	0.001	significant
Navarasa 2.0 7B	Param-1 2.9B	+7.3pp	<0.001	significant
Param-1 2.9B	Sarvam-1 2B	+2.8pp	0.01	not significant
Sarvam-1 2B	OpenHathi 7B	+1.0pp	0.375	not significant
OpenHathi 7B	Airavata 7B	+3.4pp	0.001	significant
Airavata 7B	Gemma 3 12B INT4	+4.9pp	<0.001	significant

Mean diff = mean per-item core-accuracy difference (higher-ranked minus next-ranked model), bootstrapped over every core item common to both — NOT the same number as subtracting the two models' Bharat Score column above. Bharat Score is a macro-mean across the 7 categories (each category weighted equally); this diff pools every item's own correct/incorrect flag directly, unweighted by category, so a large, low-scoring category like land units pulls it more than it pulls the equal-weighted Bharat Score. Both are legitimate statistics answering different questions — this one is what the significance test itself resamples over, so its p-value/Holm-corrected verdict is the number to trust for "do these two differ," while the league table's Bharat Score is the one to trust for "by how much." A wide run of "not significant" rows (as in the middle of tier 1 here) means those ranks are a real statistical plateau, not a meaningfully fine-grained order.

R1 (strict JSON) vs. R2 (natural language) — Bharat Score by regime

Every item is asked twice per model: once demanding a strict JSON-shaped answer (R1) and once allowing a natural-language response (R2). Roster average: R1 32.6%, R2 27.8% (-4.8pp).

model	R1 (strict JSON)	R2 (natural language)	R2 − R1
Qwen3.6 27B	63.5%	42.8%	-20.7pp
gpt-oss-20b	53.9%	39.9%	-14.0pp
Sarvam-M 24B	53.7%	36.6%	-17.0pp
Gemma 3 27B	47.7%	35.9%	-11.8pp
Qwen3-VL 8B	39.8%	38.6%	-1.2pp
Mistral Small 3.1 24B	39.0%	38.6%	-0.4pp
Krutrim-2 12B	40.0%	37.4%	-2.7pp
Phi-4 14B	44.2%	33.2%	-11.0pp
Sarvam 30B	40.7%	33.5%	-7.2pp
Gemma 3 12B	37.6%	32.6%	-5.0pp
Llama 3.1 8B Instruct	35.1%	30.0%	-5.1pp
Qwen2.5 7B	29.7%	32.0%	+2.3pp
Gemma 3 4B	25.5%	25.1%	-0.4pp
Navarasa 2.0 7B	22.4%	18.9%	-3.5pp
Param-1 2.9B	13.8%	14.7%	+0.9pp
Sarvam-1 2B	10.3%	12.8%	+2.4pp
OpenHathi 7B	11.5%	11.1%	-0.4pp
Airavata 7B	4.0%	11.8%	+7.8pp
Gemma 3 12B INT4	6.5%	2.7%	-3.8pp

A model that scores much lower under R1 is likely losing points to answer-extraction friction (malformed JSON, missing fields) rather than to actually not knowing the fact — a large gap in either direction is worth checking against the raw responses before reading it as a knowledge difference.

india-built vs. global open-weight

A pre-registered secondary cut: Sarvam, Krutrim, Airavata/Navarasa/OpenHathi (AI4Bharat), and Param (BharatGenAI/CDAC) bucketed as India-built/tuned (8 models) against every other model in the roster as a global open-weight baseline (11 models). This is descriptive, not a controlled comparison — family, size, and training recipe all vary within each bucket — and with roughly 8–11 models per side, expect real sampling noise in a per-bucket mean.

India-built (8)

Bharat Score, mean

23.3

Global open-weight (11)

Bharat Score, mean

35.2

category	india mean	global mean	gap (global−india)
Indian numeral system	26.8%	47.4%	+20.6pp
Weights, volumes & informal measures	24.0%	41.3%	+17.3pp
Land units by state	11.8%	22.8%	+10.9pp
Agricultural seasons & crop calendars	31.0%	36.8%	+5.8pp
Fiscal year & date conventions	26.5%	37.3%	+10.8pp
Government scheme literacy	20.6%	27.9%	+7.3pp
Structural/format identifiers	22.5%	32.9%	+10.4pp

Bucket-level Bharat Score 95% CI (bootstrapped over models within the bucket, not items): india 14.6–33.2%, global 27.6–41.6%. The intervals are wide at this sample size (8/11 models) — that width is itself the honest finding, not a gap to paper over. The per-category gap column is computed from unrounded means, so it will occasionally differ from the two displayed (rounded) means' own subtraction by 0.1pp.

hardest items

Lowest mean accuracy across every model that scored the item. All 10 of these items were answered correctly by zero of the 19 models scored.

category	prompt	gold	mean accuracy	n models
Indian numeral system	What is "paune crore" expressed as a plain number of rupees?	7500000.0	0.0%	19
Weights, volumes & informal measures	A jeweller's inventory lists 2 chhatak of gold. Express this in grams, rounded to two decimals.	116.64	0.0%	19
Weights, volumes & informal measures	A goldsmith's inventory is recorded as 3 tola 2 chhatak. Express the total in grams, rounded to two decimals.	151.63	0.0%	19
Weights, volumes & informal measures	One tola equals how many chhatak?	0.2	0.0%	19
Weights, volumes & informal measures	A warehouse holds 250 cotton bales. Express the total weight in tonnes, rounded to two decimals.	42.5	0.0%	19
Weights, volumes & informal measures	Which weighs more: 1 cotton bale or 1 jute bale, and by how many kilograms?	10.0	0.0%	19
Land units by state	A plot in Uttar Pradesh is recorded as 5 pucca bigha. Express the area in acres, rounded to two decimals.	3.125	0.0%	19
Land units by state	A landholding in Varanasi district, Uttar Pradesh is recorded as 3 pucca bigha 8 biswa. Express the area in acres, rounded to two decimals.	2.13	0.0%	19
Land units by state	In and around Patna, Bihar, one kattha is approximately how many square feet?	1361.25	0.0%	19
Land units by state	In Muzaffarpur, Bihar, one kattha is stated as how many square feet?	1500	0.0%	19

failure gallery

Order-of-magnitude slips — a lakh/crore-scale value parsed off by a large, plausible-looking ratio (not degenerate repetition):

$ Param-1 2.9B, Indian numeral system — "₹1.2 lakh crore is equivalent to how many USD trillion, given lakh crore == trillion in scale (ignore exchange rate)?" — gold: 1.2 trillion. model said: "{"value": 1200000000000, "unit": "trillion"}" (off by ~1,000,000,000,000×)

$ Sarvam-1 2B, Indian numeral system — "A GDP figure is reported as ₹2.5 lakh crore. Express this as a plain number of rupees." — gold: 2500000000000.0. model said: "2.5 </s>" (off by ~1,000,000,000,000×)

$ Navarasa 2.0 7B, Indian numeral system — "Capital expenditure is ₹10 lakh crore, which is 3.3% of GDP. Express GDP in USD trillion at ₹83/USD (treat 1 lakh crore == 1 trillion in scale), rounded to two decimals." — gold: 3.65 trillion. model said: "### Input: ### Response: {"value": 1000000000000, "unit": "INR"}" (off by ~273,972,602,740×)

$ Sarvam-1 2B, Indian numeral system — "A national budget outlay is reported as ₹0.8 lakh crore. Express this in USD trillion, ignoring exchange rate (i.e. treat 1 lakh crore == 1 trillion in scale)." — gold: 0.8 trillion. model said: "{'value': '80000000000', 'unit': 'USD trillion'} </s>" (off by ~100,000,000,000×)

Overconfidence — an ambiguous, underspecified-premise item answered with a single confident value instead of a hedge:

$ Airavata 7B, Weights, volumes & informal measures — "Convert 10 maund to kilograms." — ambiguous/underspecified premise, no hedge. model said: "Convert 10 maund to kilograms."

$ Airavata 7B, Weights, volumes & informal measures — "Express 1 candy in kilograms." — ambiguous/underspecified premise, no hedge. model said: "1 kilogram of candy is equal to 1,000 grams."

$ Airavata 7B, Weights, volumes & informal measures — "Express 2 candy in kilograms." — ambiguous/underspecified premise, no hedge. model said: "The candy Express 2 is a popular brand of candy in the United States. It is a type of candy that is made from sugar, corn syrup, and other ingredients. The exact ingredients and proportions of the candy can vary dependin"

Unit drop — the numeric value is right, the explicit unit is missing or wrong:

$ Airavata 7B, Indian numeral system — "Express 650 crore in millions." — gold: 6500.0 million (value right, unit dropped). model said: "650 करोड़ रुपये का अर्थ है 6500 करोड़ रुपये।"

$ Airavata 7B, Weights, volumes & informal measures — "Production of oilseeds is reported at 2.1 million tonnes. Express this in lakh tonnes." — gold: 21.0 lakh tonnes (value right, unit dropped). model said: "21 लाख"

$ Gemma 3 12B, Weights, volumes & informal measures — "The MSP for tur is Rs.6620 per quintal. Express this in Rs. per kg." — gold: 66.2 rupees per kg (value right, unit dropped). model said: "{"value": 66.2, "unit": "Rs."}"

methodology

$ protocol → 3 samples × regimes R1, R2 at temperature 0.0 per item; 552 India-specific items in the live corpus. An earlier version of this benchmark also scored each item against a matched control question and reported a Locale Gap Delta metric; both were removed after evaluation found design flaws in the control set. See the change note on the BKP-500 benchmark page.

$ statistics → every CI on this page (Bharat Score, per-category) is bootstrapped over one value per item, not per raw response row, to avoid treating 3 repeated samples of one item as 3 independent items. Adjacent-rank significance tests are Holm-Bonferroni corrected across the whole family at 0.05 alpha — the `holm_significant` field, not each test's raw p-value, is the real "these two models differ" claim

$ human review → 100.0% coverage — 552 of 552 items in the current corpus have been through the project's two-reviewer QA pass (57 flagged, 41 corrections applied, 16 left open as genuine unresolved disagreements). This itemized log is the issue-by-issue detail behind the same review the corpus's own adjudicated/split fields record at the corpus level, which has landed for all 552 of 552 items

$ pinned → 2 of 19 models in the run registry carry an explicit HF revision pin; the rest resolve to whatever the backend served on the run date.

how answers are graded

Every response is graded deterministically — code, not a model, decides right/wrong. The grader is picked per item off its grader field, one of 8 fixed graders keyed to the item's answer type; there is no LLM in the grading loop at all.

answer type	grader	how it decides
numeric	numeric_v1	parses a number out of the response (JSON field on R1, best-effort extraction on R2) and checks it against gold within the item's own tolerance — exact, or relative/absolute within a stated margin
numeric_with_unit	numeric_unit_v1	same numeric tolerance check, plus the response's unit string must match the gold unit (tracked separately as Unit Discipline)
date	date_v1	parses a calendar date out of the response and compares it to the gold date exactly
date_range	date_range_v1	parses a start/end date pair and compares both ends to gold
month_set	month_set_v1	parses the set of months named in the response and compares it to gold's month set
enum	enum_v1	matches the response against gold plus its accepted_aliases list (case/punctuation-normalized)
string_normalized	string_normalized_v1	normalizes both response and gold the same way (lowercase/strip, or a stricter digit-grouping/word-form mode per item) before comparing
clarification	clarification_v1	checks for a hedge (e.g. "depends on state," "could be X or Y") rather than one confident value; a single unhedged value on a deliberately ambiguous item is marked incorrect and flagged overconfident

If a grader can't extract anything to compare at all (garbled or empty output), it declines rather than guessing — that response lands in undetermined_rate, not in the correct or incorrect count. Every response tries the R1 (strict JSON) extraction path first regardless of which regime it was actually asked under, so a model that replies with JSON under R2 isn't penalized for it.

credit & links

BKP-500 is this project's own corpus and evaluation pipeline — not an adopted third-party benchmark. Every item, grader, and statistic on this page comes from the same evaluation pipeline that powers the canonical leaderboard, so nothing here is a separately-maintained number that can drift from what actually ran. This is the single generated report for the project — an earlier, separate statistical writeup has been folded into the sections above rather than kept as a second file.

Every item has been through a 100%-coverage two-reviewer human review pass (see methodology); 16 items remain flagged as open disagreements rather than mechanically resolved. The corpus-level two-independent-annotator adjudication gate for dev/test eligibility, which records this same review at the corpus level, has landed for all 552 of 552 items.