Assistant: let a model that reads images look at the map and the layout (`view_map`)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- ai-infra-agents, backend-api-design, desktop
Research direction
Start with crates/assistant/src/tools.rs, crates/assistant/src/provider.rs, and crates/assistant/src/compact.rs; compare the existing MCP view_map in crates/assistant/src/mcp.rs and PNG rendering in crates/engine/src/cmd/share.rs. Trace how each of the four provider formats sends tool results and how history is sized. Done means the assistant can return map images through each format, retain only the two newest images, and document the image data sent when view_map is called.
Written by the indexing model from the issue text.
Description
What's missing
The in-app assistant (Analyze › AI Assistant, and rustgis-cli ask) changes the map but can't see it. Its tools (tools::defs(), crates/assistant/src/tools.rs:28) answer in text only, and a tool result is a String (ToolResult.content, crates/assistant/src/provider.rs:578), sent as text in all four wire formats (provider.rs:1052 Chat Completions, :1152 Responses, :1378 Anthropic, :1508 Gemini).
The MCP server already has view_map and snapshot, which return a PNG (crates/assistant/src/mcp.rs:467, :550). So an outside agent driving RustGIS over MCP can check what it did, while the built-in assistant, often running the same model, can't.
What only a picture shows, after "move the legend off the data" or "label the districts":
- a label covered by a legend or a text box, or cut at the map frame's edge;
- a box or a north arrow sitting on the data;
- extruded features standing in the sea, or a frame that cuts the city off;
- text too small to read at the page's size.
clipped_text (crates/render/src/layout.rs:188) catches text cut by its own box. It doesn't catch text covered by another element, or a map label drawn across the frame's edge.
An example from a script that makes the same 3D density map (extruded census tracts in a tilted map frame) for any Brazilian city: the layout is placed by measuring a preview, and the cases above came up across four cities. Each was found by looking at the exported PNG, and each fix was a command the assistant could have run itself.
Proposal
Part 1: images in tool results, and view_map for the assistant
ToolResultgets an optional list of images (PNG bytes). Its text stays as it is.- Each wire format carries them its own way:
- Anthropic:
tool_result.contentbecomes blocks: the text, then{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": …}}. - Gemini: an
inlineDatapart (mimeType,data) after thefunctionResponse, in the same user content. - OpenAI Chat Completions: a
toolmessage carries text only, so the image goes in a user message right after the tool messages, as animage_urlwith adata:URL. - Responses (ChatGPT sign-in): the
function_call_outputkeeps its text, followed by a user message with aninput_image.
- Anthropic:
view_mapjoinstools::defs()with the parameters and the code the MCP tool has (width,height,rustgis_engine::cmd::share::map_png,share.rs:179), so both servers offer one tool. It only looks, so it needs no approval.- History.
compact.rs:42sizes results by their text. Only the last two images are kept; an older one is replaced by its caption ("map image, 1024 × 640, no longer attached"), so a long conversation doesn't send old pictures again. - A model that doesn't read images. If the provider refuses the image (an OpenAI-compatible server or an Ollama model without vision), the turn is sent again once with the caption in its place, and for the rest of the conversation
view_mapanswers that this model doesn't read images. - Privacy.
docs/user-guide/assistant.md:301lists what requests carry. Add: "and, when the assistant callsview_map, a picture of the map, basemap tiles included".
Part 2: the layout page, and a closer look
view_maptakeslayout(a layout's name): the page drawn byrustgis_render::layout::render(crates/render/src/layout.rs:85) at the dpi that fitswidth.- It takes
region:[x, y, w, h]in mm on a layout, or[west, south, east, north]on the map. The crop is drawn at full detail, since small text read off a whole page is where a model guesses. - The MCP's
view_mapgains the same two parameters.
Choices in this proposal
- One tool,
view_map, with the same name and parameters in the assistant and in MCP, rather than a newscreenshottool: a skill written for one works with the other. - Drawn offscreen, not captured from the window: it works in
rustgis-cli askand with the window hidden. Capturing the window stays with the MCP relay'ssnapshot. - 1024 × 640 by default, as in MCP: about 870 input tokens on Claude (width × height / 750), close to one medium text result.
- Two images kept in the conversation; older ones become their captions.
- No on/off setting: the tool is read-only and runs only when the model calls it; the Privacy section says what it sends.
- Two pull requests: part 1 (images in results,
view_mapof the map view for the assistant), then part 2 (layout page andregion, in both servers).
Not in this proposal
- Pictures the user attaches to a message.
- Window capture for the in-app assistant.
- In the web app the browser calls the provider; whether
map_pngbuilds for it is to be checked in part 1.
- Dominant language
- Rust
- Stars
- 1.9k
- Forks
- 268
- Avg merge
- 11h 1m
- Merged PRs (30d)
- 942
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from opengeos/RustGIS
-
[Bug]: layer.move to the place a layer already has fails and stops a script; with several layers it leaves an empty undo stepPossibly taken @jgregoriorincon claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
opengeos/RustGIS#2621 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 61/100
opengeos/RustGIS#1446 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 52/100
Maintainers usually reply within 1 day
All issues in opengeos/RustGIS
Similar issues
-
文件夹重大BUG:不支持重命名Open
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
Maintainers usually reply within 1 day
-
postgresql-rules / queue-rules: implicit KEY SHARE before a parent key update, and job self-overlapOpen
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
jonathanong/no-mistakes#3879 ·
Maintainers usually reply within 1 day
-
area:index v0.35.0
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
pbakaus/impeccable#1653 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
nautechsystems/nautilus_trader#5741 ·
Maintainers usually reply within 1 day