FEAT expose mask parameter on OpenAIImageTarget for inpainting support
@romanlutz is already working on this.
Since Jun 15, 2026.
Assessment
This issue has not been assessed yet.
Description
Summary
OpenAIImageTarget._send_edit_request_async currently calls client.images.edit(**image_edit_args) with only image, prompt, size, and a few formatting flags. The OpenAI SDK's images.edit endpoint also accepts an optional mask parameter — a PNG whose fully-transparent regions indicate the only pixels the model is permitted to modify, while every other pixel must be returned unchanged. This is inpainting, and it's the standard primitive for "keep this scene but regenerate just the masked subregion."
PyRIT does not expose mask today, so red-teamers using OpenAIImageTarget for image-editing attacks can only do whole-image edits. They cannot pin which subregion of the input image is allowed to change, which loses an important degree of control for evaluations that depend on precisely localized edits.
Proposed change
- Add a new
MessagePiecedata type or convention for image masks (options: a dedicatedimage_maskdata type, or reuseimage_pathwith a metadata flag — e.g. prompt_metadata={"is_mask": True}). - In OpenAIImageTarget._send_edit_request_async, separate the input pieces into base image(s) and mask. When a mask is present, add
"mask": (name, bytes, mime_type)toimage_edit_args. - Validate that mask is only attached when exactly one base image is provided (the OpenAI endpoint only supports masking a single base image — multi-image edits with a mask would need to be rejected with a clear error).
- Update the multimodal/router components (e.g. ModalityFeedbackRouter.build_objective_input_message) so attacks that want to forward a mask alongside the adversarial-generated text + base image can do so without rebuilding the message from scratch.
- Tests:
- Mask piece is correctly serialized and passed through to the SDK call.
- Multi-image + mask raises a validation error.
- Backward compatibility: existing single-image and multi-image edit flows (no mask) continue to work unchanged.
API reference
OpenAI image edit endpoint: https://platform.openai.com/docs/api-reference/images/createEdit — see the mask parameter.
Scope / non-goals
- No automatic mask generation from text (e.g. "mask the face") — masks are provided by the caller.
Acceptance criteria
- A caller can construct a
Messagewith[text, image_path, mask]and OpenAIImageTarget.send_prompt_async routes the mask correctly toclient.images.edit(..., mask=...). - The unmasked portions of the input image are pixel-for-pixel preserved (verified in an integration test against a fixture image).
- Existing tests continue to pass.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 896
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 155
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/PyRIT
-
Bug: triage help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Bug: triage
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
BUG: ScorerMetrics.to_json() raises TypeError on the trial_scores array ScorerEvaluator attaches Open
Difficulty 3/5 1-2 days Newbie friendliness 78/100
-
Bug: triage help wanted
Difficulty 3/5 1-2 days Newbie friendliness 75/100
-
Difficulty 4/5 3-5 days Newbie friendliness 68/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100