feat: Evaluation Client — Lifecycle, Orchestration & Online Pipeline
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- aws, python
- Domain
- backend-api-design, cloud, testing
Research direction
Start with operations/evaluation/control_plane_client.py and models.py, then read EvaluationProcessor in on_demand_processor.py, EvaluationDataPlaneClient, and delete_online_evaluation_config() in online_processor.py. Trace the linked entry points and existing integration-test setup first. Done means all listed evaluator, online-config, session-data, orchestration, error-result, cleanup, and CI integration-test criteria are covered.
Written by the indexing model from the issue text.
Description
Problem
The SDK's EvaluationClient only exposes run(). On the control plane side, customers cannot programmatically create custom evaluators (LLM-as-a-judge configs), list available evaluators, update or delete evaluators, or manage online evaluation configs for continuous evaluation on live traffic — evaluator provisioning requires the console. On the data plane side, the starter toolkit's EvaluationProcessor provides significantly richer orchestration than run(): it fetches session data from CloudWatch independently, groups evaluators by level (SESSION vs TRACE), determines which spans to send based on evaluator level, and runs multiple evaluators with per-evaluator error handling. The toolkit also provides input validation, IAM role cleanup on delete, and typed config/result models.
Acceptance Criteria
- Customers can create, get, list, update, and delete custom evaluators
- Customers can create, get, list, update, and delete online evaluation configs
- Online evaluation config supports enable/disable toggling and sampling rate adjustment
- Typed result models with error introspection (
has_error(),get_successful_results()) - Customers can fetch session trace data (spans + runtime logs) from CloudWatch for a given session and agent
- Customers can find the most recent session for an agent
- Multi-evaluator orchestration groups evaluators by level and selects appropriate spans per level
- Per-evaluator error handling — failures on one evaluator don't block others
- Online evaluation config deletion supports optional IAM execution role cleanup
- All functionality is verified via integration tests running in CI
Relevant Links
EvaluationControlPlaneClientcreate_evaluator()create_online_evaluation_config()update_online_evaluation_config()EvaluationResult/EvaluationResultsOnlineEvaluationConfigEvaluationProcessorevaluate_session()fetch_session_data()determine_spans_for_evaluator()execute_evaluators()EvaluationDataPlaneClientdelete_online_evaluation_config()
- Dominant language
- Python
- Stars
- 764
- Forks
- 149
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 7
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from aws/bedrock-agentcore-sdk-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
aws/bedrock-agentcore-sdk-python#496 · 1 comment ·
-
Pydantic deprecation warning: Support for class-based `config` is deprecated, use ConfigDict instead Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
aws/bedrock-agentcore-sdk-python#320 · 2 comments · 3 reactions ·
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
All issues in aws/bedrock-agentcore-sdk-python
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100