What It Actually Said.

Parameter count does not predict transcription accuracy — model architecture does.

The best-performing model in this benchmark is a 600M-parameter CTC conformer (IndicConformer, 22.02% WER, pooled across all 22 languages). A 430M-parameter model (SraVaani 1.0, 22.91% WER) trails by half a point despite covering only 19 of the 22 languages. Both outperform every autoregressive or LLM-decoder model evaluated, including models an order of magnitude larger. Parameter count was not the determining factor; whether the model was purpose-built for transcription was.

This distinction has practical consequences. Four additional models — all autoregressive, each marketed with some form of multilingual speech or speech-understanding capability — loaded and executed without error, producing fluent output that was not, in fact, a transcription of the input audio. A WER score computed against such output is technically valid but substantively meaningless, since it scores the wrong task. Section 2 documents all four cases, with source audio and model output.

Parameter count versus outcome, for the five models with confirmed published parameter counts:

Increasing parameter count from 430M to 1.7B yields negligible improvement. At 7.35B, the model fails the task altogether: Dhwani-1's own repository ships an explicit transcription prompt template, yet the model translates the input regardless.

The IndicVoices paper (arXiv 2403.01926) reports its own IndicASR — a 130M-parameter conformer trained only on IndicVoices — at 25.65% macro WER, in its Table 7. We ran the newer, public 600M checkpoint of the same lineage (IndicConformer) and scored it the same way.

An improvement of 2.83 points on 20 of 22 languages, consistent with a larger, later-generation checkpoint. Two languages moved in the opposite direction — Kannada (+4.8pt) and Odia (+5.4pt) — the only anomaly in this comparison, not yet root-caused. Exact reproduction of the paper's results is not possible in either direction: the paper does not disclose its normalization scheme, decoding mode, or which transcript level (verbatim or standardized) it scores against.

Hindi, full split, all three vendors:

Sarvam ranks third, effectively tied with the two leading open-weights models. Soniox ranks fifth, ahead of every Hindi-only purpose-built open-weights model evaluated (Qwen3-ASR, Nemotron, Voxtral). Deepgram ranks last among the nine. All three commercial results reflect full-scale evaluation runs.

Additional Languages: Soniox vs. Deepgram vs. Sarvam (Bengali, Gujarati, Kannada, Marathi, Tamil, Telugu)

Evaluated using identical audio and scoring methodology against all three vendors' full valid splits. Odia was initially included in this comparison but excluded, as neither Soniox nor Deepgram supports it: Soniox returns a 400 "Invalid language hint" error for the Odia language code, confirmed directly against the live API and consistent with its absence from Soniox's published language list; Deepgram returns the same error code (see coverage table below).

→ scroll to see all three vendors on small screens

^*For Soniox, the Tamil sample size is n=5,275 rather than 5,276: one utterance failed due to a request timeout unrelated to the Odia issue, and was not re-attempted given its negligible effect on the aggregate result.

Sarvam wins every language in this set, several by a wide margin — its micro WER (22.85%) approaches the pooled scores of the open-weights leaders (IndicConformer 22.02%, SraVaani 22.91%, each across a wider language set), and it costs less than Deepgram while beating it on every single language. Between Soniox and Deepgram alone, neither is uniformly superior: Soniox performs better on Bengali and Marathi; Deepgram performs better on Telugu, Kannada, and particularly Tamil. Kannada and Tamil remain Soniox's weakest results in this set by a substantial margin — both Dravidian-script languages, both near 50% WER, while its results on Indo-Aryan-script languages stay in the low-to-mid twenties, a pattern consistent with a genuine script or language-family effect rather than noise. Combined with its Hindi result (Section 6, above), Sarvam is now the strongest commercial API in this benchmark on every language tested.

Deepgram Language Coverage: 11 of 22 IndicVoices Languages

Confirmed by direct testing against the live API — each unsupported language code returns a 400 error — since Deepgram's documentation does not enumerate IndicVoices-specific coverage.

Malayalam and Odia are notable omissions given their relatively high resource availability; overall coverage caps at exactly half of the IndicVoices language set. The remaining unsupported languages form the same lower-resource cluster that presents difficulty across vendors generally.

Ground truth. IndicVoices provides two transcript levels per utterance: verbatim (phonetic, as spoken) and standardized (normalized orthography). This benchmark scores against the standardized level, consistent with the expectation that a transcription model should output normalized text rather than colloquial spelling.

Normalization. Text is normalized by stripping Unicode punctuation categories (covering sentence-final marks across all scripts, including the Devanagari danda ।) and casefolding Latin-script segments. Numeral expansion is not applied.

Metric. Word error rate and character error rate are computed using the jiwer library. Both micro-averaged (pooled) and macro-averaged (per-language mean) results are reported, as the IndicVoices paper does not specify which convention it uses. Character error rate counts whitespace as a character — jiwer's default behavior, computing Levenshtein distance over the full normalized string rather than word-delimited text. This is stated explicitly because it affects the reported value: see the Section 8 Telugu example, where CER is 16.67% excluding spaces and 20.00% (the value this report uses) including them.

Split. Only the train and valid splits of IndicVoices are publicly available; no held-out test split exists. All results in this report are computed against the full valid split.