Optional EvalPort interop for Datalab label-quality results

Open
  • Is something wrong?
#1,334 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
38/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
data, machine-learning

Research direction

Start with CONTRIBUTING.md and the Datalab API calls shown in the issue: find_issues(), get_issues(), and get_issue_summary(). Compare their real output with EvalPort’s actual JSON Schema, then resolve whether the adapter belongs in a standalone package or this repository. Done means the placement is agreed and real-schema tests validate the ResultSet round trip.

Written by the indexing model from the issue text.

Description

Hi cleanlab team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a TestCase/Suite/ResultSet schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Filing this as an issue first per CONTRIBUTING.md before writing any code.

I installed cleanlab (2.9.0) and ran the real Datalab API rather than guessing:

lab = Datalab(data=data, label_name="labels", task="classification")
lab.find_issues(pred_probs=pred_probs)
lab.get_issues()

get_issues() returns one row per example with columns like is_label_issue (bool), label_score (float, 0–1), plus the same is_*_issue / *_score pair for every other issue type that ran (outlier, non_iid, class_imbalance, etc.). get_issue_summary() aggregates those into issue_type / score / num_issues rows.

That's a very direct match for EvalPort's other half: each example in the audited dataset is a TestCase, and the is_label_issue + label_score (or any other issue-type's bool+score pair) that Datalab computes per example is exactly the shape of an EvalPort Grader result — a per-TestCase pass/fail plus a numeric score — so Datalab.get_issues() output could round-trip into an EvalPort ResultSet with very little translation.

Two ways I could see this landing, and I don't have a strong preference:

  1. A standalone cleanlab-openeval-adapter package in the EvalPort repo, depending on cleanlab as a normal dependency. Zero footprint on this repo.
  2. A small optional module inside this repo if you'd rather it live here.

Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.

Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.

Dominant language
Python
Stars
11.7k
Forks
940
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from cleanlab/cleanlab

All issues in cleanlab/cleanlab

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.