BKP-500: the Bharat Knowledge Probe, explained

BKP-500, the Bharat Knowledge Probe, is a 552-item benchmark measuring whether a language model knows the everyday conventions of Indian life — lakh and crore arithmetic, state-specific land units, crop calendars, the April to March fiscal year, government schemes and Indian identifier formats.

// what BKP-500 is

Ask a language model to convert ₹1.2 lakh crore into dollars and it will usually answer with total confidence. Often it will be off by a factor of a hundred, and nothing in the answer will warn you.

BKP-500, the Bharat Knowledge Probe, measures exactly that kind of failure. It tests whether a model knows the conventions that every Indian resident takes for granted and that most pretraining data barely contains: lakh and crore arithmetic, informal weights, land units that change from state to state, crop calendars, the April to March fiscal year, government scheme facts, and the structure of Indian identifiers.

The corpus holds 552 items across seven categories. A model passes an item by producing the right value, in the right unit, under the convention the question is asking about. Code grades every answer. No language model is involved in scoring.

// why it exists

Most benchmarks covering India test Indian languages. Knowing Hindi and knowing India are different things.

A model can write fluent Hindi and still convert a lakh incorrectly, treat a bigha as the same area in Punjab and Bihar, place the fiscal year in January, or answer a kharif question with a rabi calendar. None of those are language failures. They are knowledge failures, and they surface in exactly the places where they cost the most: finance, agriculture, land records and government services.

These errors are quiet. A lakh and crore slip does not look like a refusal or a garbled answer. It looks like a confident, well-formatted number that happens to be wrong by two orders of magnitude.

// the seven categories

Land units deserve a note. A single unit name maps to different areas depending on where the question is asked, which almost no international unit does. This is the category no model handles well, and it is why items that depend on a jurisdiction are tagged with the state they apply to.

// metrics

Bharat Score

Bharat Score is the macro-mean accuracy across all seven categories. Each category's accuracy is computed first, then the seven are averaged with equal weight.

It is macro-averaged on purpose. The categories hold different numbers of items, and a raw item mean would let the largest one dominate the headline. Under macro-averaging, a model that is strong on numerals and blind on land units cannot hide the blind spot behind volume.

Every Bharat Score is reported with a 95% confidence interval. Full definition →

Secondary measures

Out-of-magnitude error rate. The share of numeric answers off by a factor of ten or more. On lakh and crore scales this is the characteristic failure, so it is tracked on its own.

Unit discipline. The share of answers requiring a unit that included an explicit, correct one. A right number with a dropped unit is scored wrong but counted here separately, because it is a different mistake from a wrong number.

Refusal rate. The share of items a model declines to answer.

Consistency. Agreement across a model's own repeated samples of the same item. It separates a model that is reliably wrong from one that is guessing, and in practice most models turn out to be reliably wrong.

Tiers and significance

Models are grouped into tiers by chained overlap of their confidence intervals. Differences between adjacent ranks are tested with a paired bootstrap, Holm-Bonferroni corrected across the whole family of comparisons at a 0.05 family-wise alpha. The corrected result, not a raw p-value and not rank order alone, is what supports any claim that two models differ.

// how answers are graded

Every response is graded deterministically. The grader is chosen per item from its declared answer type, from a fixed set of eight:

Clarification items test whether a model recognizes that a question cannot be answered with one value. A single confident answer on one of these is marked incorrect.

If a grader cannot extract anything comparable from a response, it declines rather than guessing. That response is recorded as undetermined and counts in neither the correct nor the incorrect column.

// run protocol

Each item is asked under two prompt regimes, strict JSON and natural language, with three repeated samples per item at temperature zero. The strict-JSON extraction path is tried first on every response, so a model that answers in JSON when asked in natural language is not penalized for it.

Confidence intervals are bootstrapped over one value per item rather than per response, so three samples of one item are never treated as three independent observations.

// corpus, license and review status

The corpus, the item design, the graders and the pipeline are all our own work. BKP-500 is not an adopted third-party benchmark.

Review status. All 552 items have been through a two-reviewer quality pass, and the corpus-level adjudication that governs test eligibility is complete for all 552. Every item ships as split: test. The reviewers flagged 57 items with a gold-value problem. 41 of those corrections are applied, and the remaining 16 are kept as clarification items: cases where official sources conflict on a convention, so the correct answer is to recognize the ambiguity rather than commit to one value.

Some items, state land units particularly, carry provenance gaps where an authoritative source was not confirmed at authoring time. Every item records its source, supporting quote and a confidence rating, so a reader can weigh it.

Gold answers are visible. Nothing is held out yet, so a score should not be read as evidence of uncontaminated competence without checking the model's training cutoff. A held-out test split is planned. Every item carries a unique canary string so anyone building a pretraining corpus can filter it out.

License. The data is CC BY-NC 4.0. The evaluation harness is Apache 2.0.

The corpus sits behind a Hugging Face access agreement to protect it as a benchmark. The dataset card, including the full method and category counts, is readable without signing in.

// runs to date

// changes to this benchmark

September 21, 2026. An earlier version of BKP-500 paired every item with an internationally framed control question and reported a Locale Gap Delta metric from the difference. Evaluation surfaced systematic design flaws in the control set, so the controls and the metric were both removed. The corpus now holds the 552 India-specific items only, and all scores have been recomputed on that basis. The 2026 roster was updated from 21 models to 19 at the same time.

We publish changes like this one here rather than editing quietly, because a benchmark that changes without saying so cannot be trusted.

// how to cite

Sthānika AI (2026). BKP-500: the Bharat Knowledge Probe, 2026 model run. https://sthanika.ai/research/bkp500-2026

Cite Bharat Score with its confidence interval. Intervals overlap across much of the roster, so a difference in rank is not by itself a claim that two models differ.