Indic Script Reading Gap

This benchmark asks a vision-language model to read Indian public signage the way it is actually photographed: a hand-painted shop board at an angle, a fare list behind a grille, a hoarding half in shadow, multiple languages on one signboard. Each model is shown one full, unedited test-set image from BSTD — a scene-text dataset of real photographs from across India, annotated at word level with a polygon, the exact transcription, and a script-language label — and asked to find and read everything legible in it.

Every model is given the identical prompt, verbatim:

No bounding boxes are given, no script is named and no example is shown, so the model has to both find the text and read it.

Because BSTD supplies the gold strings, scoring is a string comparison rather than a model-judged rubric: the sign says what it says.

BSTD is a scene-text dataset of 6,582 photographs taken across India — 5,263 in its train split and the 1,319 of its test split scored here — each annotated by hand at word level: a polygon around every piece of text, the exact string inside that polygon, and the language it is written in. Those annotations are the ground truth; a transcription is graded by string comparison against them, with no model judging the answer.

Every item in this benchmark looks like the three below: one full, unedited photograph, and a set of human-drawn polygons each carrying the exact string inside it and the language it is written in. Nothing is cropped for the model and nothing is straightened — it is handed the whole frame and asked to transcribe every piece of legible text it can find, in the original script, one per line. The yellow outlines below are BSTD's ground truth, not model output.

Fuzzy recall on the 9,512 English and 11,785 Indic-script annotations that appear on the same 914 images. Each pair of dots is one model; the bar between them is that model's script gap, with image quality held constant by construction.

Of the 1,319 test images, 914 carry both at least one English annotation and at least one Indic-script annotation.

BSTD's full test split, one run per model, less the one or two images per model that failed every retry. exact is a normalized substring match; fuzzy additionally accepts a match at 0.8 similarity, so it is always the higher of the pair.

One table per metric, because the three answer different questions: a model can score well on precision by writing very little, and that looks nothing like scoring well on recall by reading accurately. Rows are the eight models. Columns group the images by their dominant language label — the most common one among that image's annotations. These are BSTD's own labels rather than scripts: Hindi and Marathi share Devanagari, Assamese and Bengali share Eastern Nagari, so the thirteen columns span eleven writing systems. All three tables are the fuzzy variant, the more generous of the two matchers; the exact variant is in the leaderboard above. An outlined cell is the best model in that column, except in Meitei — at 130 gold strings, less than half the next-smallest column, a few hits move it enough that a "best" badge there would mark a coin flip, not a result, so the outline is suppressed for that column only.

recall, fuzzy — of the annotated text, how much was found

precision, fuzzy — of what the model wrote, how much is backed by an annotation

F1, fuzzy — the two combined