Assistant: let a model that reads images look at the map and the layout (`view_map`)

Open
#2,517 1 comment 1 reaction 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
rust

Research direction

Start with crates/assistant/src/tools.rs, crates/assistant/src/provider.rs, and crates/assistant/src/compact.rs; compare the existing MCP view_map in crates/assistant/src/mcp.rs and PNG rendering in crates/engine/src/cmd/share.rs. Trace how each of the four provider formats sends tool results and how history is sized. Done means the assistant can return map images through each format, retain only the two newest images, and document the image data sent when view_map is called.

Written by the indexing model from the issue text.

Description

What's missing

The in-app assistant (Analyze › AI Assistant, and rustgis-cli ask) changes the map but can't see it. Its tools (tools::defs(), crates/assistant/src/tools.rs:28) answer in text only, and a tool result is a String (ToolResult.content, crates/assistant/src/provider.rs:578), sent as text in all four wire formats (provider.rs:1052 Chat Completions, :1152 Responses, :1378 Anthropic, :1508 Gemini).

The MCP server already has view_map and snapshot, which return a PNG (crates/assistant/src/mcp.rs:467, :550). So an outside agent driving RustGIS over MCP can check what it did, while the built-in assistant, often running the same model, can't.

What only a picture shows, after "move the legend off the data" or "label the districts":

  • a label covered by a legend or a text box, or cut at the map frame's edge;
  • a box or a north arrow sitting on the data;
  • extruded features standing in the sea, or a frame that cuts the city off;
  • text too small to read at the page's size.

clipped_text (crates/render/src/layout.rs:188) catches text cut by its own box. It doesn't catch text covered by another element, or a map label drawn across the frame's edge.

An example from a script that makes the same 3D density map (extruded census tracts in a tilted map frame) for any Brazilian city: the layout is placed by measuring a preview, and the cases above came up across four cities. Each was found by looking at the exported PNG, and each fix was a command the assistant could have run itself.

Proposal

Part 1: images in tool results, and view_map for the assistant
  • ToolResult gets an optional list of images (PNG bytes). Its text stays as it is.
  • Each wire format carries them its own way:
    • Anthropic: tool_result.content becomes blocks: the text, then {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": …}}.
    • Gemini: an inlineData part (mimeType, data) after the functionResponse, in the same user content.
    • OpenAI Chat Completions: a tool message carries text only, so the image goes in a user message right after the tool messages, as an image_url with a data: URL.
    • Responses (ChatGPT sign-in): the function_call_output keeps its text, followed by a user message with an input_image.
  • view_map joins tools::defs() with the parameters and the code the MCP tool has (width, height, rustgis_engine::cmd::share::map_png, share.rs:179), so both servers offer one tool. It only looks, so it needs no approval.
  • History. compact.rs:42 sizes results by their text. Only the last two images are kept; an older one is replaced by its caption ("map image, 1024 × 640, no longer attached"), so a long conversation doesn't send old pictures again.
  • A model that doesn't read images. If the provider refuses the image (an OpenAI-compatible server or an Ollama model without vision), the turn is sent again once with the caption in its place, and for the rest of the conversation view_map answers that this model doesn't read images.
  • Privacy. docs/user-guide/assistant.md:301 lists what requests carry. Add: "and, when the assistant calls view_map, a picture of the map, basemap tiles included".
Part 2: the layout page, and a closer look
  • view_map takes layout (a layout's name): the page drawn by rustgis_render::layout::render (crates/render/src/layout.rs:85) at the dpi that fits width.
  • It takes region: [x, y, w, h] in mm on a layout, or [west, south, east, north] on the map. The crop is drawn at full detail, since small text read off a whole page is where a model guesses.
  • The MCP's view_map gains the same two parameters.

Choices in this proposal

  • One tool, view_map, with the same name and parameters in the assistant and in MCP, rather than a new screenshot tool: a skill written for one works with the other.
  • Drawn offscreen, not captured from the window: it works in rustgis-cli ask and with the window hidden. Capturing the window stays with the MCP relay's snapshot.
  • 1024 × 640 by default, as in MCP: about 870 input tokens on Claude (width × height / 750), close to one medium text result.
  • Two images kept in the conversation; older ones become their captions.
  • No on/off setting: the tool is read-only and runs only when the model calls it; the Privacy section says what it sends.
  • Two pull requests: part 1 (images in results, view_map of the map view for the assistant), then part 2 (layout page and region, in both servers).

Not in this proposal

  • Pictures the user attaches to a message.
  • Window capture for the in-app assistant.
  • In the web app the browser calls the provider; whether map_png builds for it is to be checked in part 1.
Dominant language
Rust
Stars
1.9k
Forks
268
Avg merge
11h 1m
Merged PRs (30d)
942

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from opengeos/RustGIS

All issues in opengeos/RustGIS

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.