[Optimize]: Resolution budgets, crop/bbox coordinate tools, and image-workflow guidance for vision tools
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 38/100
- Tipo di issue
- Funzionalità
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Tranquilla
- Stack tecnologico
- rust
- Ambito
- ai, computer-vision, tooling
Direzione di ricerca
Inizia tracciando i percorsi esistenti di analyze_image, view_image, optimize_image_with_size_limit e ImageAnalyzer, inclusa la gestione di ImageLimits e coordinate_note. Poi verifica come vengono registrati i file SKILL.md integrati. Il lavoro è completato quando le impostazioni predefinite del budget, crop_image e draw_bbox, l’upscaling tramite budget-path e la skill image-workflow funzionano insieme, mentre il comportamento legacy rimane invariato.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
Upgrade BitFun's built-in vision tools with four capabilities borrowed from Qwen-MM-Plugins' design:
- Resolution budget presets for
analyze_image/view_image(small/normal/large) - Coordinate closed loop: new
crop_imageanddraw_bboxtools using 0-1000 normalized coordinates - Small-image upscaling to a minimum pixel floor (capped at 2x linear) so tiny crops stay legible for VLMs
- Built-in decision skill (
image-workflow) teaching the agent when to view vs analyze vs crop, and how to pick a budget
Background
The current vision path is two fixed-limit tools plus a batch pre-analysis path:
analyze_imagesends the image + prompt to the configured image-understanding model. Every call pays full image tokens, and the only size control is the hard provider cap (1MB tool cap / provider dimension limits). There is no way for the agent to trade detail against cost.view_imageattaches an image to the primary model context, also with no resolution control.optimize_image_with_size_limitnever upscales, so a tiny crop (e.g. a 64x64 region) is sent as-is and OCR/detail suffers.- The only coordinate handling is a prose
coordinate_notewarning the model that positions in the analysis are estimates. There is no tool to actually crop or annotate a static image, so the estimate cannot be turned into an action loop. - Tool descriptions are one-liners; the model has no guidance on which tool to use when.
Qwen-MM-Plugins solves these with token-budget-driven resolution presets (256/1024/2048 visual tokens), a crop/draw_bbox pair that consumes grounding output in the same 0-1000 normalized coordinate system, an upscale floor, and declarative SKILL.md decision documents.
Proposed changes
1. Budget presets (optional parameter, legacy behavior unchanged)
- Add
ImageBudget(small/normal/large) mapping to pixel targets via token budgets (256/1024/2048 x 32^2 = ~512^2 / ~1024^2 / ~1448^2), clamped by the providerImageLimitsceiling. - Add an optional
budgetparameter toanalyze_imageandview_image. When omitted, the existing path runs unchanged (no behavior change for existing callers/configs). - Fix the
resize_notewording which currently hard-codes "downscaled" and would misreport upscaling.
2. Coordinate closed loop: crop_image + draw_bbox
crop_image(path, box[x1,y1,x2,y2 in 0-1000], output_path?): validates and clamps the box, saves the cropped region next to the source by default, and returns pixel-mapped coordinates. Attaches a preview when the primary model supports multimodal tool output; degrades to a text summary otherwise.draw_bbox(path, boxes[{bbox,label?}], output_path?): draws rectangles with an adaptive line width and a fixed palette. v1 draws boxes only (no text labels, no new dependencies).- Both support remote workspace paths (read + write through workspace filesystem services).
- Closed loop: the model reports a region in normalized coordinates,
crop_imagecuts it, thenanalyze_image(crop_path, budget="large")re-inspects the region at high resolution.
3. Small-image upscaling
- In the budget path only: images below the minimum pixel floor (~512^2) are upscaled, capped at 2x linear so tiny images do not burn tokens pointlessly (larger regions should be re-cropped instead).
4. Built-in decision skill
- New builtin skill
image-workflow(SKILL.md): tool-selection decision table, budget guidance (small preview / normal default / large detail; large on crops), coordinate discipline (normalized 0-1000, model coordinates are estimates, use reported width/height/was_resized to convert), and cost tips (image tokens dominate; crop before repeated full-image calls).
Non-goals (follow-ups)
- 32px patch-grid snapping for Qwen-style providers
- Text label rendering in
draw_bbox(needs font rasterization) - Budget support for the
ImageAnalyzerbatch pre-analysis path
- Lingua principale
- Rust
- Stelle
- 2.3k
- Fork
- 236
- Merge medio
- 2h 56m
- PR unite (30g)
- 619
Guida per i contributori
Apri la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di GCWing/OpenBitFun
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
GCWing/OpenBitFun#3213 · 1 commento ·
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
GCWing/OpenBitFun#2363 ·
-
question
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 78/100
GCWing/OpenBitFun#2340 ·
-
Difficoltà 3/5 1-2 giorni Idoneità per principianti 65/100
GCWing/OpenBitFun#3210 ·
-
GCWing/OpenBitFun#3195 · 2 commenti · 1 assegnatario ·
Tutte le issue di GCWing/OpenBitFun
Issue simili
-
Replayed reasoning items send "content": null, which the Responses API schema does not permit Apertabug CLI custom-model
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
rust-bitcoin/rust-bitcoin#6930 · 1 commento ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
fulcrumgenomics/ferro-hgvs#2251 ·
-
A-allocators A-docs C-enhancement T-libs
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100