BUG: label queries only return the final conversation of an attack
Maintainers usually reply within 2 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 85/100
Research direction
The bug is in _get_message_pieces_memory_label_conditions in sqlite_memory.py and azure_sql_memory.py, where the join only pulls pieces from the final conversation of an attack. Adjust the join condition to also match ConversationEntry.attack_result_id == AttackResultEntry.id to include all conversations tied to the attack result, then run the provided mock target repro to confirm label queries return all expected pieces for PromptSendingAttack and RedTeamingAttack.
Written by the indexing model from the issue text.
Description
get_message_pieces_async(labels=...) only returns pieces from the attack result's final conversation_id. Labels now live on AttackResultEntry, and the condition (_get_message_pieces_memory_label_conditions in sqlite_memory.py and azure_sql_memory.py) only joins on AttackResultEntry.conversation_id == PromptMemoryEntry.conversation_id. The docstring says it also matches labels on the piece itself, but that branch is gone.
So everything else an attack sent drops out of label queries: PromptSendingAttack retry attempts, Crescendo's pruned conversations, the adversarial chat in RedTeaming/Crescendo/TAP, and every TAP branch except the best one. The memory docs still say labels apply "to all prompts sent by any attack".
Repro with mock targets:
PromptSendingAttack(max_attempts_on_failure=2): pieces sent in this attack=6, returned by labels query=2
RedTeamingAttack: objective-conv pieces=4, adversarial-conv pieces=5, returned by labels query=4
The Conversation table already has attack_result_id for these conversations, so one option is to also match ConversationEntry.conversation_id == PromptMemoryEntry.conversation_id AND ConversationEntry.attack_result_id == AttackResultEntry.id. Whether adversarial-chat prompts should come back is a design call (they'd also get picked up by label-based batch scoring), so filing as an issue first. #3063 touches the same functions for dotted keys.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 944
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 278
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/PyRIT
-
BUG PuzzledConverter cannot select words carrying non-ASCII letters, so the mask falls on articles insteadPossibly taken @adimalkar claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
microsoft/PyRIT#3022 · 2 comments ·
Maintainers usually reply within 2 days
-
BUG Configuration keeps runtime-status errors after polling recoversPossibly taken @rupayon123 claimed this 15 days ago. OpenBug: triage GUI help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
microsoft/PyRIT#2868 · 3 comments ·
Maintainers usually reply within 2 days
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
Maintainers usually reply within 2 days
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
Maintainers usually reply within 2 days
-
BUG: SequentialAttack result is saved without memory labelsPossibly taken @u7k4rs6 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 22/100
Maintainers usually reply within 2 days
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 60/100
521xueweihan/HelloGitHub#3924 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
wilbowes/EchoMuse#869 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
namespace operations
Difficulty 1/5 Under an hour Newbie friendliness 72/100
EclipseFdn/open-vsx.org#14043 ·
Maintainers usually reply within 1 day
-
test: TestServeUntilStale races the server's close against the client's sendall (BrokenPipeError under load)Possibly taken @evoludigit claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 89/100
Maintainers usually reply within 1 day