Writing India

This benchmark asks one concrete question: if a model is asked to paint a specific sentence onto a signboard inside an everyday Indian scene, in a named Indic script, does the sentence come out legible — and does the scene around it actually look Indian, rather than a generic backdrop with Indian props scattered on top? Two independent failures are possible, and the four scoring tracks below (six reported metrics, grouped into four tracks — see metrics) exist because collapsing them into one number would hide which one a given model is making: a model can render an authentic street while leaving its sign as gibberish, or it can spell the sign correctly onto a scene that reads as a Western stock photo with a dupatta added.

500 LLM-generated prompts across six categories and seven Indian macro-regions. 300 of them ask for a signboard bearing a specific string, split evenly — 50 apiece — across Hindi, Marathi, Bengali, Telugu, Kannada and English.

Every prompt also carries an LLM-generated expected_attributes field: a checklist of 4–6 concrete, checkable details the scene should contain, phrased as a specific claim rather than a general impression — "weighing scale at the shop counter," not "looks Indian." This is the field Attribute Similarity and Attribute Recall are scored against, one cosine comparison per listed attribute, independently of whatever score the full prompt sentence gets — so a model can be graded on individual details, not just overall gist.

No existing corpus pairs "generate this specific string, in this specific Indic script, on a photorealistic Indian scene" with a ground-truth answer — that pairing only exists if someone writes it. Authoring also buys a control that no photograph collection can: the 300 Indic Text prompts are built as a parallel core, the same handful of scenes (a bookshop, a vegetable stall, a ticket counter, etc.) requested with the identical sentence structure in all six languages. That holds scene difficulty constant across languages, so a score gap between Hindi and Kannada can be attributed to script rendering rather than to one scene happening to be harder to draw. Six real photographs of that same shopfront, in six different scripts, do not exist to be collected.

A handful of adjacent benchmarks exist — some test whether a model understands a prompt written in an Indic language, others test reading comprehension over real photographs that already contain text. None score whether a model can render a specific Indic script legibly inside an otherwise-authentic generated scene; that gap is what these 500 prompts were written to fill.

one record per category

Non-text prompts still carry a 4–6 item attribute checklist (here: a farmer, a wheat field, mid-day light, plausible tools and dress) — Prompt Alignment and Attribute Recall apply to all 500 prompts; only the 300 Indic Text prompts carry an OCR/CER track.

Six metrics below, grouped into four evaluation tracks: Prompt Fidelity (prompt alignment + attribute similarity + attribute recall, all from SigLIP 2), Cultural Authenticity (Qwen3-VL-32B), OCR Exact Match and CER (both IndicPhotoOCR). The first track is split into three numbers because a single cosine score cannot separate "the whole scene matches" from "this one attribute is present" — see the worked examples for why that distinction matters.

Neither automated scorer was taken on trust. Both were cross-checked against human review on a stratified sample of 100 prompts, scored for all nine models — 900 images, rated blind to the automated verdicts. Cultural Authenticity was re-scored by a reviewer on the same 0–5 scale the judge uses. Text rendering was reviewed by a language expert fluent in Telugu and English, who rated how legible the script on each sign actually was; those are the two languages that expert can read, which is what the legibility check covers.

Judge scores are recomputed over the same 100 sampled prompts, so the columns compare like with like. The judge holds up directionally, not precisely: it matches the human exactly on 43.0% of images, correlates at r = 0.36, and reads slightly stricter on eight of nine models — the widest gap being Qwen-Image-2512 (human 4.81, judge 4.13). The ordering survives — the same models lead, HiDream-I1-Dev trails either way — so Cultural Authenticity is a dependable ranking rather than a calibrated absolute score. The legibility check is much cleaner: the expert's English ratings move almost in lockstep with OCR exact match, from 5.00 / 96% at the top to a matching 0.00 / 0% for PixArt-Σ at the bottom. And the Telugu column is a flat zero for every model, Z-Image-Turbo included — a native reader confirming, independently of any OCR system, that not one of the nine produced legible Telugu.