Support batch insights in BatchEvaluationRunner
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- aws, python
- Domain
- ai-infra-agents, api, backend
Research direction
Examine the BatchEvaluationRunner class to understand its current dataset-based workflow. Review the StartBatchEvaluation API and the listed Builtin.Insight types (FailureAnalysis, UserIntent, ExecutionSummary) to understand the constraints. Determine if insights can be integrated into the existing runner or require a separate component. Check for any existing tests related to batch evaluation or insights.
Written by the indexing model from the issue text.
Description
BatchEvaluationRunner does not expose the batch insights workflow.
Requested behavior
Support requesting batch insights through BatchEvaluationRunner, with job polling and access to the resulting insights. The original report identifies these StartBatchEvaluation insights:
Builtin.Insight.FailureAnalysisBuiltin.Insight.UserIntentBuiltin.Insight.ExecutionSummary
The workflow should enforce the API constraint that insights and evaluators cannot be requested in the same job.
Why this matters
The reporter uses insights to cluster failures and identify root causes that can be addressed through prompt changes. Without SDK support, the evaluation improvement workflow requires custom orchestration outside BatchEvaluationRunner.
Open design question
Confirm whether the runner's dataset-based workflow is compatible with insights and which inputs are supported. Use that finding to determine how to expose insights through the runner, or whether a separate insights runner is needed. This compatibility question remains unresolved.
Original report environment: bedrock-agentcore 1.23.0, Python 3.13, us-west-2.
- Dominant language
- Python
- Stars
- 786
- Forks
- 163
- Avg merge
- 4h
- Merged PRs (30d)
- 22
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from aws/bedrock-agentcore-sdk-python
-
bug high-severity
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
aws/bedrock-agentcore-sdk-python#752 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
aws/bedrock-agentcore-sdk-python#751 · 1 reaction ·
Maintainers usually reply within 1 day
-
delete_all_long_term_memories_in_namespace deletes only the first 10 recordsPossibly taken @Fire162 claimed this 18 days ago. Openbug high-severity
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
aws/bedrock-agentcore-sdk-python#727 ·
Maintainers usually reply within 1 day
-
MPP: credential is always sent in Authorization, ignoring header="Payment-Authorization" in the challengePossibly taken @abhiramshaji-cyber claimed this 24 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
aws/bedrock-agentcore-sdk-python#702 ·
Maintainers usually reply within 1 day
-
[Bug] update_message fails with parameter validation error when SessionMessage.message_id is a Strands positional integer indexPossibly taken @citizen204 claimed this 144 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
aws/bedrock-agentcore-sdk-python#556 ·
Maintainers usually reply within 1 day
All issues in aws/bedrock-agentcore-sdk-python
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
snap-stanford/Biomni#380 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
microsoft/playwright-python#3361 ·
Maintainers usually reply within 1 day
-
documentation priority: low size: S status: needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
cubrid-lab/sqlalchemy-cubrid#1397 ·
Maintainers usually reply within 1 day
-
documentation priority: low size: S status: needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
cubrid-lab/pycubrid#1307 ·
Maintainers usually reply within 1 day
-
[Bug]: Installer hides SSL certificate errors as bare URLError, and the suggested fix can't workOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
ShaftHQ/chaosengine-harness#143 ·
Maintainers usually reply within 1 day