Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Optimize]: Resolution budgets, crop/bbox coordinate tools, and image-workflow guidance for vision tools

Open
#2,248 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
38/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
rust

Research direction

Start by tracing the existing analyze_image, view_image, optimize_image_with_size_limit, and ImageAnalyzer paths, including ImageLimits and coordinate_note handling. Then review how built-in SKILL.md files are registered. Done means budget presets, crop_image and draw_bbox, budget-path upscaling, and the image-workflow skill work together while legacy behavior remains unchanged.

Written by the indexing model from the issue text.

Description

enhancement

Summary

Upgrade BitFun's built-in vision tools with four capabilities borrowed from Qwen-MM-Plugins' design:

  1. Resolution budget presets for analyze_image / view_image (small / normal / large)
  2. Coordinate closed loop: new crop_image and draw_bbox tools using 0-1000 normalized coordinates
  3. Small-image upscaling to a minimum pixel floor (capped at 2x linear) so tiny crops stay legible for VLMs
  4. Built-in decision skill (image-workflow) teaching the agent when to view vs analyze vs crop, and how to pick a budget

Background

The current vision path is two fixed-limit tools plus a batch pre-analysis path:

  • analyze_image sends the image + prompt to the configured image-understanding model. Every call pays full image tokens, and the only size control is the hard provider cap (1MB tool cap / provider dimension limits). There is no way for the agent to trade detail against cost.
  • view_image attaches an image to the primary model context, also with no resolution control.
  • optimize_image_with_size_limit never upscales, so a tiny crop (e.g. a 64x64 region) is sent as-is and OCR/detail suffers.
  • The only coordinate handling is a prose coordinate_note warning the model that positions in the analysis are estimates. There is no tool to actually crop or annotate a static image, so the estimate cannot be turned into an action loop.
  • Tool descriptions are one-liners; the model has no guidance on which tool to use when.

Qwen-MM-Plugins solves these with token-budget-driven resolution presets (256/1024/2048 visual tokens), a crop/draw_bbox pair that consumes grounding output in the same 0-1000 normalized coordinate system, an upscale floor, and declarative SKILL.md decision documents.

Proposed changes

1. Budget presets (optional parameter, legacy behavior unchanged)
  • Add ImageBudget (small/normal/large) mapping to pixel targets via token budgets (256/1024/2048 x 32^2 = ~512^2 / ~1024^2 / ~1448^2), clamped by the provider ImageLimits ceiling.
  • Add an optional budget parameter to analyze_image and view_image. When omitted, the existing path runs unchanged (no behavior change for existing callers/configs).
  • Fix the resize_note wording which currently hard-codes "downscaled" and would misreport upscaling.
2. Coordinate closed loop: crop_image + draw_bbox
  • crop_image(path, box[x1,y1,x2,y2 in 0-1000], output_path?): validates and clamps the box, saves the cropped region next to the source by default, and returns pixel-mapped coordinates. Attaches a preview when the primary model supports multimodal tool output; degrades to a text summary otherwise.
  • draw_bbox(path, boxes[{bbox,label?}], output_path?): draws rectangles with an adaptive line width and a fixed palette. v1 draws boxes only (no text labels, no new dependencies).
  • Both support remote workspace paths (read + write through workspace filesystem services).
  • Closed loop: the model reports a region in normalized coordinates, crop_image cuts it, then analyze_image(crop_path, budget="large") re-inspects the region at high resolution.
3. Small-image upscaling
  • In the budget path only: images below the minimum pixel floor (~512^2) are upscaled, capped at 2x linear so tiny images do not burn tokens pointlessly (larger regions should be re-cropped instead).
4. Built-in decision skill
  • New builtin skill image-workflow (SKILL.md): tool-selection decision table, budget guidance (small preview / normal default / large detail; large on crops), coordinate discipline (normalized 0-1000, model coordinates are estimates, use reported width/height/was_resized to convert), and cost tips (image tokens dominate; crop before repeated full-image calls).

Non-goals (follow-ups)

  • 32px patch-grid snapping for Qwen-style providers
  • Text label rendering in draw_bbox (needs font rasterization)
  • Budget support for the ImageAnalyzer batch pre-analysis path
Dominant language
Rust
Stars
2.3k
Forks
236
Avg merge
2h 55m
Merged PRs (30d)
611

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from GCWing/OpenBitFun

All issues in GCWing/OpenBitFun

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.