feat: Evaluation Client — Lifecycle, Orchestration & Online Pipeline

Open
#393 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Quiet
Tech stack
aws, python

Research direction

Start with operations/evaluation/control_plane_client.py and models.py, then read EvaluationProcessor in on_demand_processor.py, EvaluationDataPlaneClient, and delete_online_evaluation_config() in online_processor.py. Trace the linked entry points and existing integration-test setup first. Done means all listed evaluator, online-config, session-data, orchestration, error-result, cleanup, and CI integration-test criteria are covered.

Written by the indexing model from the issue text.

Description

enhancement

Problem

The SDK's EvaluationClient only exposes run(). On the control plane side, customers cannot programmatically create custom evaluators (LLM-as-a-judge configs), list available evaluators, update or delete evaluators, or manage online evaluation configs for continuous evaluation on live traffic — evaluator provisioning requires the console. On the data plane side, the starter toolkit's EvaluationProcessor provides significantly richer orchestration than run(): it fetches session data from CloudWatch independently, groups evaluators by level (SESSION vs TRACE), determines which spans to send based on evaluator level, and runs multiple evaluators with per-evaluator error handling. The toolkit also provides input validation, IAM role cleanup on delete, and typed config/result models.

Acceptance Criteria

  • Customers can create, get, list, update, and delete custom evaluators
  • Customers can create, get, list, update, and delete online evaluation configs
  • Online evaluation config supports enable/disable toggling and sampling rate adjustment
  • Typed result models with error introspection (has_error(), get_successful_results())
  • Customers can fetch session trace data (spans + runtime logs) from CloudWatch for a given session and agent
  • Customers can find the most recent session for an agent
  • Multi-evaluator orchestration groups evaluators by level and selects appropriate spans per level
  • Per-evaluator error handling — failures on one evaluator don't block others
  • Online evaluation config deletion supports optional IAM execution role cleanup
  • All functionality is verified via integration tests running in CI

Relevant Links

Dominant language
Python
Stars
764
Forks
149
Avg merge
1d 19h
Merged PRs (30d)
7

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from aws/bedrock-agentcore-sdk-python

All issues in aws/bedrock-agentcore-sdk-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.