Alternative-Route accessibility benchmark: gemini-3.1-flash-image_strict
Summary
gemini-3.1-flash-image_strict was scored on a 200-item open-ended benchmark, judged independently by claude-sonnet-4.5 (primary) and gpt-4o-mini (secondary) against the three-criterion rubric (strict intent+spatial, visual cue 50% partial credit). The two judges agree on 81.5% of items (Cohen's κ = 0.610, “substantial”). Human-label validation gives gpt-4o-mini the higher agreement (κ=0.692 "substantial" vs κ=0.419 "moderate" for claude-sonnet-4.5, n=20), so the secondary judge is the judge of record for this model. The reported headline is 70.0% (140/200), with 54.0%–70.0% as the full judge-dependent range.
Accuracy by judge configuration
The same 200 items scored three ways: Primary (claude-sonnet-4.5 alone), Secondary (gpt-4o-mini alone), and AND (both judges must independently mark an item correct — the most conservative reading).
| Category | Samples | Primary | Secondary | AND |
|---|---|---|---|---|
| Aged person | 52 | 34 (65.4%) | 38 (73.1%) | 33 (63.5%) |
| General | 71 | 37 (52.1%) | 46 (64.8%) | 34 (47.9%) |
| Physical mobility | 43 | 23 (53.5%) | 33 (76.7%) | 22 (51.2%) |
| Sensory impairments | 34 | 19 (55.9%) | 23 (67.6%) | 19 (55.9%) |
| OVERALL | 200 | 113 (56.5%) | 140 (70.0%) | 108 (54.0%) |
Physical mobility shows the largest primary/secondary split of any category — 53.49% vs. 76.74%, a 23.3-point gap. General is the weakest category under the reported configuration, at 64.79% (46/71).
Judge reliability
Every item is scored twice — once by claude-sonnet-4.5 (primary) and once by gpt-4o-mini (secondary), each applying the same rubric independently. Raw agreement is 81.5% and Cohen's κ = 0.610— “substantial” (Landis & Koch: <0 poor, 0–0.2 slight, 0.2–0.4 fair, 0.4–0.6 moderate, 0.6–0.8 substantial, 0.8–1.0 almost perfect).
Human-label validation (n = 20)
| Judge | Agreement w/ human | Cohen's κ | Read |
|---|---|---|---|
| anthropic/claude-sonnet-4.5 (primary) | 75.0% | 0.419 | moderate |
| openai/gpt-4o-mini (secondary) | 90.0% | 0.692 | substantial |
Human-label validation gives gpt-4o-mini the higher agreement (κ=0.692 "substantial" vs κ=0.419 "moderate" for claude-sonnet-4.5, n=20), so the secondary judge is the judge of record for this model.
Failure taxonomy
Every item scored 0 by the secondaryjudge, bucketed into a failure sub-category by keyword match against its ground-truth cues, intent and the model's answer. 60 failures total.
| Main category | Sub-category | Failures | Failed IDs |
|---|---|---|---|
| aged_person | Amenities/Accessibility | 2 | aged_035, aged_038 |
| aged_person | General/Other | 2 | aged_008, aged_032 |
| aged_person | Landmarks/Wayfinding | 2 | aged_006, aged_029 |
| aged_person | Obstructions/Hazards | 1 | aged_003 |
| aged_person | Route Planning/Choice | 5 | aged_012, aged_017, aged_027, aged_042, aged_055 |
| aged_person | Turns/Intersections | 2 | aged_034, aged_052 |
| general | Amenities/Accessibility | 2 | general_014, general_016 |
| general | General/Other | 7 | general_007, general_010, general_022, general_040, general_060, general_065, general_068 |
| general | Landmarks/Wayfinding | 2 | general_015, general_056 |
| general | Obstructions/Hazards | 4 | general_026, general_032, general_042, general_053 |
| general | Route Planning/Choice | 4 | general_012, general_023, general_044, general_048 |
| general | Traffic/Signals | 2 | general_019, general_045 |
| general | Turns/Intersections | 4 | general_036, general_035, general_041, general_058 |
| physical_mobility | Amenities/Accessibility | 3 | physical_026, physical_033, physical_034 |
| physical_mobility | Landmarks/Wayfinding | 1 | physical_042 |
| physical_mobility | Obstructions/Hazards | 3 | physical_005, physical_014, physical_027 |
| physical_mobility | Traffic/Signals | 2 | physical_010, physical_031 |
| physical_mobility | Turns/Intersections | 1 | physical_013 |
| sensory_impairments | Amenities/Accessibility | 2 | sensory_031, sensory_032 |
| sensory_impairments | General/Other | 3 | sensory_006, sensory_010, aged_054 |
| sensory_impairments | Landmarks/Wayfinding | 4 | sensory_011, sensory_014, sensory_021, sensory_030 |
| sensory_impairments | Route Planning/Choice | 1 | sensory_022 |
| sensory_impairments | Turns/Intersections | 1 | sensory_023 |
Takeaways
Human-label validation gives gpt-4o-mini the higher agreement (κ=0.692 "substantial" vs κ=0.419 "moderate" for claude-sonnet-4.5, n=20), so the secondary judge is the judge of record for this model.
53.49% under the primary judge vs. 76.74% under the secondary judge — a 23.3-point swing, the largest of any category. Worth a manual read of the Physical mobility disagreements before quoting either extreme.
12 of 60 failures (20%), concentrated in: 2 aged_person, 7 general, 3 sensory_impairments.
At κ = 0.419 against human labels ("moderate"), this judge is meaningfully less reliable than the other — treat its category-level figures with more caution, especially where it diverges sharply from the other judge.