MILU, re-run on the 2026 models.

We ran the full 11-language MILU test set — 79,608 questions each — on 17 models available through mid-2026 (16 league-table rows: 15 open-weight models, with Qwen3.6-27B appearing twice, thinking on and off — the other 2 models were API-only and are covered separately below). We matched the scoring protocol to each model: loglikelihood 5-shot where the harness fits, generative 0-shot where it doesn't, and paired every result with a real or reference-derived cost per 1,000 answered questions.

Qwen3.6-27B with thinking on leads the open-weight league table at 83.84% overall, ahead of Sarvam-M 24B's 82.44%. Its own thinking-off configuration scores 16.84 points lower (67.00%), the widest swing from a single setting change in the roster. Two models in our roster — Qwen3.8-Max (89.67%, ₹187.45/1K) and DeepSeek V4-Flash (79.51%, ₹2.45/1K) — were accessed only via API with no local checkpoint, so we compare them separately against closed frontier models rather than ranking them alongside models we can actually re-host ourselves. Beyond the rankings, the data show that scoring protocol is the main confounder for reasoning models, matched-size generations still gain meaningfully (Gemma 3→4, Qwen2.5→Qwen3-VL), and the low-resource gap — especially Odia — persists across almost every model regardless of scale.

We normalized to a per-question unit cost rather than total run cost, so a huge dense model and a tiny sparse one are comparable on the same axis. Every model on this chart is open-weight and hosted locally, so cost is GPU-hours × an illustrative reference rate ($1.80/hr, one on-demand A100 80GB PCIe instance) — not a real bill, since we ran these on a box that's privately provisioned and shared. We converted to ₹ at ₹95.29/$1, then divided by thousands of successfully-answered questions across all 11 languages (79,608 total per model). For loglikelihood models, we measured this GPU-hour figure as the wall-clock time of the full run, which already includes all 4 per-option forward passes per question — dividing that real total by the number of questions (not by 4× as many requests) is what makes a loglikelihood row and a generative row comparable on the same axis. The ₹ conversion itself is a real rate; the underlying $ cost basis is still not — treat every ₹ figure here as directional, not an invoice. The two models we accessed only via API (Qwen3.8-Max, DeepSeek V4-Flash) use real metered spend instead of this reference rate, so we break them out separately in the frontier & API comparison above rather than mixing cost bases on one axis.

We numbered each point by league-table rank (matches the # column in the table further down this page) and connected its model name with a thin leader line, routed into open space rather than sitting on top of the dot. A few points (#09-#11) sit close together in both accuracy and cost — that's genuine proximity in the underlying data; follow the leader line or hover for the exact number. The x-axis extends to ₹200 to fit the roster's two priciest points, Qwen3.6-27B, thinking on (#01, 83.84%) at ₹161.35/1K and Sarvam-30B (#03, 72.99%) at ₹165.03/1K — near-coincidentally close in cost (both are 27B+-class models served via llama.cpp across the full 79,608-item set at similar GPU-hour totals) despite an 11-point accuracy gap between them. A log-scale x-axis handles that stretch without compressing the cheaper points on the left.

This section mixes two different kinds of sourcing, kept clearly separate in the table below. Eleven closed frontier models come from three system cards' own MILU comparison tables — Claude Opus 4.7, Claude Sonnet 5, and Claude Sonnet 4.6's own cards, plus every model each of those cards benchmarked itself against. We did not run any of these eleven — no local checkpoint exists for any of them, and API access wasn't in scope for this phase. We took their accuracy verbatim from each system card's own MILU comparison table — self-reported for that card's own Claude model, and run by that card's authors (Anthropic) for every non-Anthropic model shown alongside it, not self-reported by Google or OpenAI. We estimated their cost ourselves — not a bill — per the model's own tokenizer family: tiktoken's o200k_base (Anthropic and OpenAI models — exact for OpenAI, an approximation for Claude) or a local Gemma tokenizer (Google models — Gemini and Gemma share the same SentencePiece tokenizer family, a materially closer proxy than tiktoken, especially on Indic scripts). Qwen3.8-Max and DeepSeek V4-Flash are different — the two models in our own roster we accessed only via API, with no local checkpoint to host ourselves. Unlike the eleven above, we ran both of these ourselves, on the full 79,608-item MILU test set, with real metered $ spend, not an estimate — see the league table note for why they're grouped here with the API-accessed frontier models rather than the open-weight roster. The scatter chart below plots only the eleven vendor-cited frontier models, on their own scale, since that cluster (87-94% accuracy) sits well above our own roster; Qwen3.8-Max and DeepSeek appear in the table beneath it instead.

This table shows accuracy on our four foregrounded product languages — Telugu, Hindi, Marathi, Kannada — side by side across nine selected models (all 17 in our roster completed full 11-language runs; these nine are shown here for space). For the per-domain (subject-area) pattern instead of per-language, see "the domain gap" section below.

We compare English against the 10-language Indic average, per model, across all 17 of our models — and, since every model shows some gap, which specific languages close it and which widen it.

"Social Sciences" weakness is partly a logic-puzzle problem, not a knowledge gap. A large share of misses under it come from MILU's subject: Sociology bucket, which is mostly encoded family-relation logic puzzles: