Last month I was building a feature that asked an LLM to describe a path through a grid. Nothing fancy \u2014 parse a dungeon map, plan a route, output natural language directions. I was using Claude Haiku because this would run a lot and I didn’t want to burn through tokens on a bigger model.<\/p>\n
It worked. Reasonably well, anyway. Then I changed the input format.<\/p>\n
Instead of feeding the model a text description of adjacency relationships (“cell A connects to B and D, B connects to A and C…”), I switched to a visual ASCII grid. Same maze. Same model. Same prompt. The quality fell off a cliff. The model started giving directions that would walk straight through walls.<\/p>\n
I assumed I had a prompt engineering problem and spent two days messing with the prompt. Added chain-of-thought steps. Described the coordinate system in detail. Gave example paths. Nothing helped.<\/p>\n
A paper out this week \u2014 researchers testing Gemini 2.5 Flash, GPT-5-mini, Claude Haiku 4.5, and DeepSeek on structured maze tasks \u2014 confirmed exactly what I ran into: LLMs don’t have representation-invariant spatial understanding. The same model that gets 80-86% accuracy on a 5\u00d75 maze when you feed it text adjacency data drops to 16-34% on the identical maze in visual grid format. That’s not a minor regression. That’s a completely different capability.<\/p>\n
The paper (arXiv:2604.10690<\/a>) is worth reading if you’re building anything that requires LLMs to reason about physical space or multi-step planning in structured environments.<\/p>\n The setup was simple: give models mazes of different sizes (5\u00d75 to 11\u00d711) in different formats, and see how they find valid paths. The key finding isn’t that LLMs are bad at mazes \u2014 some models did fine on small mazes in text format. The finding is that performance is representation-dependent<\/em>, not format-invariant.<\/p>\n In other words, these models don’t build a spatial map and reason from it. They pattern-match on whatever representation you give them. If that format matches what they saw in training data, they do well. If it doesn’t, they fail even when the underlying problem is identical.<\/p>\n The researchers also tried chain-of-thought prompting. Models got up to 96-99% semantic coverage in their reasoning traces \u2014 correctly naming adjacent cells, describing the layout accurately. But they still made navigation errors. The model can describe the maze’s structure correctly and still give you directions that walk through walls.<\/p>\n That one stuck with me. The model knows<\/em> the structure of the maze. It just can’t use that knowledge reliably to plan a path.<\/p>\n If you’re using LLMs for anything spatial, this is a problem you’re likely already hitting, not a future concern.<\/p>\n Navigation features are the obvious case. Any time you ask an LLM to plan a path, suggest directions, or reason about what’s adjacent to what, the format you feed it matters more than you’d expect. JSON adjacency lists consistently outperform ASCII grids. Coordinate systems described in prose outperform raw numbers. I’ve found this holds across model families.<\/p>\n Multi-step planning has the same representation sensitivity. I’ve seen it with dependency resolution too: give an LLM a dependency tree as JSON and it handles it fine; describe the same tree differently and it misorders steps or loops. This isn’t just a spatial problem \u2014 it’s a general pattern that spatial tasks happen to make very visible.<\/p>\n Map and diagram understanding hits this too. If your feature involves images of maps, floorplans, or spatial diagrams, don’t assume the model sees it spatially. The paper specifically found that visual formats underperform text formats even when the information content is identical. A model looking at an ASCII grid may be worse at navigating it than a model reading a text description of the same grid.<\/p>\n My first assumption was that this was a capability gap that model upgrades would close. If Haiku fails on visual grids, surely GPT-5 or Gemini Pro handles it fine.<\/p>\n Not what the research shows. The representation sensitivity exists across frontier models. Gemini 2.5 Flash, GPT-5-mini, Claude Haiku 4.5 all show the same directional pattern: text adjacency outperforms visual grids. The gap narrows a bit with bigger models, but it doesn’t disappear.<\/p>\n This is worth knowing before you spend time and money on model upgrades hoping to solve what’s actually a representation problem. Check the format before you reach for a bigger model.<\/p>\n Control the representation explicitly. Don’t pass through whatever format your data happens to be in \u2014 convert it first. For graph-like spatial data, JSON adjacency lists work better than ASCII art or image inputs. Be specific about coordinates, directions, and connectivity in the prompt, even if it feels redundant.<\/p>\n Validate the output structurally. If the LLM says “go from A to B to C,” check programmatically that this path is actually valid in your data. Don’t treat the model’s spatial reasoning as authoritative \u2014 treat it as a suggestion that needs verification. Extra work, but the only reliable approach I’ve found.<\/p>\n Break spatial problems into smaller steps. Performance degrades quickly with grid size. If you need a full path through a large space, ask for 2-3 steps at a time, validate them, then continue. Asking for the whole plan at once on anything larger than a 7\u00d77 grid is asking for trouble.<\/p>\n And: consider whether you need LLMs for the spatial part at all. A* pathfinding<\/a> is a solved problem. If you need path planning, use a traditional algorithm and then use an LLM to describe the result in natural language. Use each tool for what it’s actually good at.<\/p>\n That’s what I ended up doing. The LLM generates readable directions for routes that a deterministic algorithm already figured out and verified. It’s good at that. It’s not good at doing the routing itself.<\/p>\n The paper tests base models with standard prompting. Whether fine-tuning specifically on spatial tasks changes this picture, I genuinely don’t know. My guess is a spatially fine-tuned model would improve on its training distribution but still show representation sensitivity on formats it hasn’t seen \u2014 but that’s speculation.<\/p>\n I’ve also tried very explicit coordinate descriptions in the prompt and gotten some improvement. Not enough to fully compensate for the format sensitivity the research documents, but meaningful enough that prompt engineering isn’t a complete dead end here.<\/p>\n If spatial reasoning failures fit into a wider pattern you’re hitting with reasoning models, I looked at this more broadly earlier \u2014 reasoning models are more fragile than the benchmarks suggest<\/a>, and the spatial case is one version of a general problem: confident-sounding output that’s structurally wrong.<\/p>\n The practical move, for now: when you hit a spatial reasoning failure, check your input representation before you blame the model. It’s probably the format.<\/p>\nWhere this shows up in practice<\/h2>\n
The representation gap isn’t fixed by a bigger model<\/h2>\n
What I changed after running into this<\/h2>\n
A few things I’m still not sure about<\/h2>\n
\n