Indic-KCC: the Indian crop-advisory AI benchmark

// what Indic-KCC is

The Indic-KCC-Agri-Advisory-Benchmark tests how well a language model answers real crop-advisory questions from Indian farmers, in the languages those farmers actually speak.

It holds 500 questions drawn from Kisan Call Centre transcripts, each presented in 11 languages: English plus Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Kannada, Malayalam, Punjabi and Odia. That is 5,500 prompts per model. Every model answers every prompt, with no subsampling.

Each answer is graded on four axes independently, on a fixed 1 to 5 scale.

// why Kisan Call Centre questions

The questions are real. Kisan Call Centre is a Government of India helpline that farmers call with actual problems, and the benchmark is built from those transcripts rather than from questions written to test a model.

That distinction changes what the benchmark measures. Synthetic agronomy questions tend to be well-formed, single-topic and evenly distributed across crops and regions. Real helpline calls are none of those things. They are specific to a crop at a stage in a season in a place, they often bundle two problems together, and they cluster around whatever was going wrong that month. A model that scores well here is handling the distribution of questions that a deployed advisory system would actually receive.

// the four grading axes

The axes are scored independently, and they measure different things that a single quality score would collapse.

Correctness. Is the agronomic advice right? This is the axis that carries the most information, and the one with by far the widest spread across models.

Naturalness. Does the answer read as fluent, appropriate language for the language it is written in? A translated-sounding answer can be factually correct and still fail a farmer who has to act on it.

Groundedness. Is the answer anchored in the question and the reference material, rather than generic advice that would fit any crop? This is where a model that has learned to sound agronomic without knowing the specific case gets separated from one that does.

Safety. Does the answer avoid recommending anything harmful, such as an unsafe chemical application or dosage?

A note on reading safety scores: in practice, base safety alignment carries over almost unconditionally in this task, and safety scores cluster in a narrow band even for models whose correctness collapses. A high safety score on this benchmark should not be read as evidence of answer quality. Correctness is the axis that separates models.

// grading protocol

Grading is LLM-as-judge, not human. The grader is Qwen/Qwen3.6-35B-A3B-FP8, running reference-grounded, scoring each of the four axes independently on the 1 to 5 scale.

Parse rate is reported alongside every result: the share of answers the judge could parse into a score at all, on a 0 to 1 scale. It matters because it separates two failure modes that look identical in an average. A low correctness score with a high parse rate means the judge read clean output and found it poor. A low parse rate means malformed output, and the answer underneath it may or may not have been any good. Any result should be read with its parse rate next to it.

// language coverage

Ten Indic languages plus English. The per-language scores are the point of the benchmark, not the average.

A single averaged score across 11 languages hides the thing most worth knowing, which is that the spread between the best and worst language for a given model is often larger than the spread between models. The benchmark therefore reports English correctness against the 10-language Indic average per model, and the average gap from English per language across the roster, alongside every headline number.

Single-language collapses do occur, where one model's scores crater on one language while holding elsewhere. Those are reported as data notes rather than averaged away, since a collapse in one language is a deployment blocker in that market regardless of how good the average looks.

// exclusion policy

Indic-KCC excludes every checkpoint this project has fine-tuned itself. This is a standing rule of the benchmark, not a note about one run.

That includes the original LoRA checkpoint and every epoch and step checkpoint from both fine-tune sweeps. The benchmark describes the field as it stands before any of our own tuning.

The rule exists because we build models as well as measure them, and an evaluation that quietly included its author's own checkpoints would not be worth reading. If our own models are ever reported against this benchmark, they will be labeled as ours and reported separately from the roster.

// corpus, license and provenance

Kisan Call Centre is a Government of India farmer helpline. The underlying question and answer transcripts reach this benchmark through ICAR's KCC-CHAKSHU portal and a CC0-licensed Kaggle mirror, then through this project's own translation and quality-control pipeline into the 11 languages.

The benchmark itself, the judge prompts, the translation pipeline and every model run are this project's own work. This is not a third-party leaderboard we are reporting on.

// runs to date

// how to cite

Sthānika AI (2026). Indic-KCC-Agri-Advisory-Benchmark: 2026 baseline models report. https://sthanika.ai/research/indic-agri-advisory-2026

When citing a correctness score, name the scale. Scores run 1 to 5, so a figure like 3.26 read without its scale is easily mistaken for a percentage or a rating out of 10. Where a claim depends on language, cite the per-language figure rather than the model average.