cleanlab/cleanlab

Turn near-duplicate score test into a property-based test

Open

#944 创建于 2024年1月5日

在 GitHub 查看
 (1 评论) (0 反应) (0 负责人)Python (604 fork)batch import
good first issuehelp-wanted

仓库指标

Star
 (7,479 star)
PR 合并指标
 (30 天内没有已合并 PR)

描述

After updating the near-duplicate scores, a test was added to ensure that near-duplicate examples have worse scores than non-near-duplicates. RIght now, the test only works on a small, toy dataset. Turning it into a property-based test would be ideal.

So that the test doesn't take too long, it may be a good idea to make a different embedding strategy that:

  1. Generates a random array
  2. Selects a handful of random examples to have near-duplicates. Append a new example that takes the original example with variable noise.
    # append to existing feature array
    x[sample_ids] + small_random_noise
    
  3. The embedding strategy should always generate "proper" near-duplicates that will get flagged. We don't want to test cases where there are no near-duplicates.

Below is the existing test case for comparison. We want to make sure that the issue scores created by the issue manager don't violate this property.

https://github.com/cleanlab/cleanlab/blob/23af72a2bd260be3b39cf35f66a8c187ebc4d1d2/tests/datalab/issue_manager/test_duplicate.py#L89-L98

Originally posted by @elisno in https://github.com/cleanlab/cleanlab/pull/943#discussion_r1442401494

贡献者指南