Alternative-Route accessibility benchmark: gemini-3.1-flash-image
Summary
gemini-3.1-flash-image was scored on a 200-item open-ended benchmark, judged independently by claude-sonnet-4.5 (primary) and gpt-4o-mini (secondary) against the three-criterion rubric (2-of-3). The two judges agree on 89.0% of items (Cohen's κ = 0.718, “substantial”). Human-label validation gives claude-sonnet-4.5 the higher agreement (κ=0.857 "almost perfect" vs κ=0.483 "moderate" for gpt-4o-mini, n=20), so the primary judge is the judge of record for this model. The reported headline is 72.5% (145/200), with 68.0%–74.5% as the full judge-dependent range.
Accuracy by judge configuration
The same 200 items scored three ways: Primary (claude-sonnet-4.5 alone), Secondary (gpt-4o-mini alone), and AND (both judges must independently mark an item correct — the most conservative reading).
| Category | Samples | Primary | Secondary | AND |
|---|---|---|---|---|
| Aged person | 52 | 43 (82.7%) | 46 (88.5%) | 43 (82.7%) |
| General | 71 | 50 (70.4%) | 48 (67.6%) | 45 (63.4%) |
| Physical mobility | 43 | 29 (67.4%) | 33 (76.7%) | 29 (67.4%) |
| Sensory impairments | 34 | 23 (67.6%) | 22 (64.7%) | 19 (55.9%) |
| OVERALL | 200 | 145 (72.5%) | 149 (74.5%) | 136 (68.0%) |
Physical mobility shows the largest primary/secondary split of any category — 67.44% vs. 76.74%, a 9.3-point gap. Physical mobility is the weakest category under the reported configuration, at 67.44% (29/43).
Judge reliability
Every item is scored twice — once by claude-sonnet-4.5 (primary) and once by gpt-4o-mini (secondary), each applying the same rubric independently. Raw agreement is 89.0% and Cohen's κ = 0.718— “substantial” (Landis & Koch: <0 poor, 0–0.2 slight, 0.2–0.4 fair, 0.4–0.6 moderate, 0.6–0.8 substantial, 0.8–1.0 almost perfect).
Human-label validation (n = 20)
| Judge | Agreement w/ human | Cohen's κ | Read |
|---|---|---|---|
| anthropic/claude-sonnet-4.5 (primary) | 95.0% | 0.857 | almost perfect |
| openai/gpt-4o-mini (secondary) | 85.0% | 0.483 | moderate |
Human-label validation gives claude-sonnet-4.5 the higher agreement (κ=0.857 "almost perfect" vs κ=0.483 "moderate" for gpt-4o-mini, n=20), so the primary judge is the judge of record for this model.
Failure taxonomy
Every item scored 0 by the primaryjudge, bucketed into a failure sub-category by keyword match against its ground-truth cues, intent and the model's answer. 55 failures total.
| Main category | Sub-category | Failures | Failed IDs |
|---|---|---|---|
| aged_person | Amenities/Accessibility | 1 | aged_035 |
| aged_person | General/Other | 2 | aged_008, general_062 |
| aged_person | Landmarks/Wayfinding | 1 | aged_029 |
| aged_person | Route Planning/Choice | 3 | aged_012, aged_027, aged_042 |
| aged_person | Turns/Intersections | 2 | aged_034, aged_052 |
| general | General/Other | 5 | general_010, general_040, general_050, general_063, general_065 |
| general | Landmarks/Wayfinding | 2 | general_022, general_034 |
| general | Obstructions/Hazards | 3 | general_032, general_053, general_060 |
| general | Route Planning/Choice | 6 | general_013, general_023, general_037, general_044, general_048, general_068 |
| general | Traffic/Signals | 3 | general_019, general_031, general_045 |
| general | Turns/Intersections | 2 | general_035, general_058 |
| physical_mobility | Amenities/Accessibility | 4 | physical_018, physical_026, physical_033, physical_034 |
| physical_mobility | General/Other | 1 | physical_035 |
| physical_mobility | Landmarks/Wayfinding | 1 | physical_042 |
| physical_mobility | Obstructions/Hazards | 2 | physical_027, physical_039 |
| physical_mobility | Route Planning/Choice | 2 | physical_029, physical_036 |
| physical_mobility | Traffic/Signals | 2 | physical_010, physical_031 |
| physical_mobility | Turns/Intersections | 2 | physical_013, physical_019 |
| sensory_impairments | Amenities/Accessibility | 2 | sensory_010, sensory_031 |
| sensory_impairments | General/Other | 2 | sensory_006, sensory_032 |
| sensory_impairments | Landmarks/Wayfinding | 5 | sensory_011, sensory_021, sensory_027, sensory_026, sensory_030 |
| sensory_impairments | Obstructions/Hazards | 1 | sensory_037 |
| sensory_impairments | Traffic/Signals | 1 | sensory_005 |
Takeaways
Human-label validation gives claude-sonnet-4.5 the higher agreement (κ=0.857 "almost perfect" vs κ=0.483 "moderate" for gpt-4o-mini, n=20), so the primary judge is the judge of record for this model.
67.44% under the primary judge vs. 76.74% under the secondary judge — a 9.3-point swing, the largest of any category. Worth a manual read of the Physical mobility disagreements before quoting either extreme.
11 of 55 failures (20%), concentrated in: 3 aged_person, 6 general, 2 physical_mobility.
At κ = 0.483 against human labels ("moderate"), this judge is meaningfully less reliable than the other — treat its category-level figures with more caution, especially where it diverges sharply from the other judge.