BKP-500: the Bharat Knowledge Probe, explained

// what BKP-500 is

BKP-500, the Bharat Knowledge Probe, is a benchmark that tests whether a language model actually knows India-specific conventions, or whether it defaults to the international convention that dominates most pretraining data.

The corpus holds 1,086 items. 552 of them are India-specific core items across seven categories. The rest are matched international control items, each one a twin of a core item asking the equivalent question under a self-contained or international convention.

A model passes an item by producing the right value, in the right unit, under the convention the question is actually asking about. Answers are graded by code, not by another model.

// why it exists

Most benchmarks that cover India test Indian languages. That is a different question from whether a model knows India.

A model can produce fluent Hindi and still convert a lakh incorrectly, treat a bigha as a fixed nationwide area, place the fiscal year in January, or answer a kharif question with a rabi calendar. Those failures are not language failures. They are knowledge failures about conventions that hold in India and nowhere else, and they show up in exactly the applications where they cost the most: finance, agriculture, land records, and government scheme delivery.

BKP-500 measures that gap directly, and the paired-item design is what lets it separate a genuine India-specific gap from a model simply being weak at the underlying task.

// the seven categories

Indian numeral system. Lakh and crore arithmetic, digit grouping, and conversion between the Indian and international scales. This is where a factor-of-ten slip does the most damage, so numeric answers here are also tracked separately for scale errors.

Weights, volumes and informal measures. Tola, seer, chhatak and the other informal units still in daily commercial use, alongside quintal-to-kilogram conversions of the kind that appear in minimum support price arithmetic.

Land units by state. The definition of a bigha in Punjab is not the definition of a bigha in Bihar. Katha, guntha, cent and their state-level variants behave the same way. This category exists because a single unit name maps to different areas depending on where the question is being asked, which is a property almost no international unit has.

Agricultural seasons and crop calendars. Kharif, rabi and zaid sowing and harvest windows, and the crops that belong to each.

Fiscal-year conventions. The April to March financial year, quarter boundaries derived from it, and the assessment-year offset that follows.

Government schemes. Eligibility rules, benefit amounts, and administering bodies for central schemes.

Structural identifiers. The format and composition rules of the identifiers used across Indian administrative and financial systems.

// paired-item design

Every core item, with 18 deliberate exceptions, has a matched control twin. The control asks the same underlying question under an international or self-contained convention, at comparable difficulty.

This matters because a low score on India items, taken alone, tells you nothing. It could mean the model does not know Indian conventions, or it could mean the model is bad at arithmetic, bad at dates, or bad at following the answer format. The control twin holds all of that constant. When a model does well on the control and poorly on its paired core item, the difference is attributable to the India-specific convention rather than to general capability.

The 18 unpaired core items are clarification-type items, deliberately underspecified questions with no international twin, where the correct behavior is to hedge rather than commit to one value. Those are scored on Ambiguity Handling and Overconfidence instead of on accuracy.

// metrics

Bharat Score

Bharat Score is the macro-mean accuracy across all seven categories: each category's accuracy is computed first, then the seven category scores are averaged with equal weight.

It is macro-averaged rather than a raw item mean on purpose. The categories do not hold equal numbers of items, and a raw mean would let the largest category dominate the headline number. Under macro-averaging, a model that is strong on numerals and blind on land units cannot hide the blind spot behind volume.

Read it as a percentage of the seven-category average, not as a share of items answered correctly.

Locale Gap Delta

Locale Gap Delta is control-item accuracy minus India-item accuracy, computed over matched pairs only.

A positive value means the international framing was easier for that model, which is the expected direction and the reading the benchmark was built to detect. A negative value means the India-framed item was easier, which does happen and is worth examining rather than dismissing.

One caution when reading it: an extreme Locale Gap Delta from a model with a very low Bharat Score is usually a symptom of that model failing broadly, not a clean locale signal. The metric is most informative on models that are otherwise working.

Secondary measures

OOM error rate. The share of numeric answers off by a factor of ten or more, which on lakh and crore scales is the characteristic failure.

Unit discipline. The share of answers requiring a unit that included an explicit, correct one. A right value with a dropped unit is scored incorrect but tracked here separately, because it is a different kind of mistake from a wrong number.

Refusal rate, consistency, ambiguity handling, overconfidence. Consistency is agreement across a model's own repeated samples, which distinguishes a model that is repeatably wrong from one that is guessing.

// scoring

Every response is graded deterministically. Code decides right and wrong, and there is no model in the grading loop.

The grader is selected per item from its declared answer type, from a fixed set of eight: numeric, numeric with unit, date, date range, month set, enum, normalized string, and clarification. Each applies its own comparison rule, with per-item tolerances on numeric types and an accepted-alias list on enums. The clarification grader checks for a hedge rather than a single confident value, and marks an unhedged answer on a deliberately ambiguous item as incorrect and overconfident.

If a grader cannot extract anything comparable from a response, it declines rather than guessing. That response is recorded as undetermined and is counted in neither the correct nor the incorrect column.

// run protocol

Each item is asked under two prompt regimes, strict JSON and natural language, with three repeated samples per item at temperature zero. The strict-JSON extraction path is attempted first on every response regardless of which regime produced it, so a model that answers in JSON when asked in natural language is not penalized for it.

Confidence intervals are bootstrapped over one value per item rather than per raw response row, so three repeated samples of the same item are never treated as three independent observations. Adjacent-rank significance tests are Holm-Bonferroni corrected across the whole family of comparisons at a 0.05 family-wise alpha, which means the corrected result, not an individual raw p-value, is what supports any claim that two models differ.

// corpus, license and review status

The corpus, the item design, the graders and the evaluation pipeline are all this project's own work. BKP-500 is not an adopted third-party benchmark.

Review status, stated plainly. All 1,086 items in the current corpus have been through a single-reviewer quality pass. That reviewer flagged 123 items with a gold-value problem. 104 of those corrections are applied to the live corpus and 19 remain open, most of them cases where a single scalar value was proposed against a convention that genuinely conflicts across sources, which we have left as an open question rather than resolving mechanically.

This is a lighter bar than the project's stricter gate of two independent annotators with adjudication, which governs dev and test eligibility and has not yet run. Under that stricter definition every item is still marked draft. Any figure produced from this corpus should be read as reviewed but not yet fully adjudicated.

// runs to date

Two control items were added to the corpus after the 2026 run completed and were never sent to any model, which is why the corpus now holds 1,086 items and the 2026 run was scored on 1,084.

// how to cite

Sthānika AI (2026). BKP-500: the Bharat Knowledge Probe, 2026 model run. https://sthanika.ai/research/bkp500-2026

When citing a score, include the confidence interval. Bharat Score figures carry a 95% interval and the intervals overlap across much of the roster, so a rank difference is not by itself a claim that two models differ. The report marks which adjacent-rank gaps survive correction.