elastic/elastic-evals-sdk-python

View on GitHub

Offline LLM evaluation framework

Stars: 2
Forks: 0
Open beginner issues: 1
Indexed issues: 21
Avg merge: 1d 5h
Merged PRs (30d): 17

Dominant language: Python
License: No license data
Last GitHub push: Sep 16, 2026
Latest indexed: Sep 15, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

18 open issues indexed

Run id baggage never reaches spans or Kibana, and uses a key Kibana does not read Open

kbn-evals Team:nightshift-context-and-research

Difficulty 4/5 3-5 days Newbie friendliness 64/100

elastic/elastic-evals-sdk-python#72 · Opened Sep 16, 2026
[kbn-evals] Non-dict expected output is silently dropped before Kibana upload Open

kbn-evals Team:nightshift-context-and-research

Difficulty 3/5 1-2 days Newbie friendliness 78/100

elastic/elastic-evals-sdk-python#70 · Opened Sep 14, 2026
Experiment results URL points to nonexistent Kibana /runs/ route Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

elastic/elastic-evals-sdk-python#64 · Opened Sep 9, 2026
[kbn-evals] Add IR evaluators with rank-based metrics (HitRate@K, MRR@K, NDCG@K, MAP@K) Open

kbn-evals Team:nightshift-context-and-research

Difficulty 4/5 3-5 days Newbie friendliness 50/100

elastic/elastic-evals-sdk-python#60 · Opened Sep 3, 2026
[kbn-evals] Code quality and dead-code cleanup Open

kbn-evals Team:nightshift-context-and-research

Difficulty 5/5 Over a week Newbie friendliness 32/100

elastic/elastic-evals-sdk-python#43 · Opened Aug 12, 2026
[kbn-evals] Small correctness (UX) fixes Open

kbn-evals Team:nightshift-context-and-research

Difficulty 4/5 3-5 days Newbie friendliness 48/100

elastic/elastic-evals-sdk-python#42 · Opened Aug 12, 2026
[kbn-evals] Repository and CI health Open

kbn-evals Team:nightshift-context-and-research

Difficulty 4/5 3-5 days Newbie friendliness 52/100

elastic/elastic-evals-sdk-python#41 · Opened Aug 12, 2026
[kbn-evals] Examples should not ship in the installed wheel Open

kbn-evals Team:nightshift-context-and-research

Difficulty 2/5 1-3 hours Newbie friendliness 76/100

elastic/elastic-evals-sdk-python#40 · Opened Aug 12, 2026
[kbn-evals] HTTP efficiency: connection pooling and score batching Open

kbn-evals Team:nightshift-context-and-research

Difficulty 4/5 3-5 days Newbie friendliness 55/100

elastic/elastic-evals-sdk-python#39 · Opened Aug 12, 2026
[kbn-evals] KibanaInferenceClient bypasses shared retry and error stack Open

kbn-evals Team:nightshift-context-and-research

Difficulty 4/5 3-5 days Newbie friendliness 68/100

elastic/elastic-evals-sdk-python#38 · Opened Aug 12, 2026
[kbn-evals] connector_id reported as model identity Open

kbn-evals Team:nightshift-context-and-research

Difficulty 4/5 3-5 days Newbie friendliness 52/100

elastic/elastic-evals-sdk-python#37 · Opened Aug 12, 2026
[kbn-evals] Replace evaluator factories with named classmethods Open

kbn-evals Team:nightshift-context-and-research

Difficulty 4/5 3-5 days Newbie friendliness 52/100

elastic/elastic-evals-sdk-python#36 · Opened Aug 12, 2026
[kbn-evals] Tutorial-grade onboarding example Open

kbn-evals Team:nightshift-context-and-research

elastic/elastic-evals-sdk-python#35 · 1 assignee · Opened Aug 12, 2026
[kbn-evals] Documentation: prerequisites, quick start, env vars, evaluator reference Open

kbn-evals Team:nightshift-context-and-research

Difficulty 4/5 3-5 days Newbie friendliness 48/100

elastic/elastic-evals-sdk-python#34 · Opened Aug 12, 2026
[kbn-evals] Evaluator discovery and preflight validation Open

kbn-evals Team:nightshift-context-and-research

Difficulty 4/5 3-5 days Newbie friendliness 58/100

elastic/elastic-evals-sdk-python#33 · Opened Aug 12, 2026
KibanaInferenceClient sends no Authorization header (401 on real Kibana) Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 76/100

elastic/elastic-evals-sdk-python#8 · Opened Jul 9, 2026
Shift phoenix tracing away from the framework Open

elastic/elastic-evals-sdk-python#3 · 1 reaction · 2 assignees · Opened Feb 5, 2026
Add unit tests for the basic functionality Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

elastic/elastic-evals-sdk-python#2 · Opened Feb 5, 2026

elastic/elastic-evals-sdk-python

Get new issues in your inbox