CID22: full dataset annotations not included in download
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Quiet
- Domain
- data, machine-learning
Research direction
Start with the CID22 full dataset download and its CID22_validation_set.csv, then compare the annotation coverage with the CID22 labs page description. Done would mean the MCOS, RMOS, and Elo scores for all 250 references are published, or the documentation clearly states which annotations are withheld.
Written by the indexing model from the issue text.
Description
The CID22 full dataset download (7.2 GB, 250 references, ~22k compressed images) only includes CID22_validation_set.csv with annotations for the 49-reference validation subset (4,292 scored pairs).
The remaining ~17.7k pairs from the other 201 references have no published human scores. The CID22 labs page doesn't mention that the full set annotations are withheld — it describes the dataset as containing "22,000 subjective scores" but only ~4,300 are in the download.
Could the full set annotations (MCOS/RMOS/Elo scores for all 250 references) be published? This would make CID22 much more useful as a benchmark for independent metric development and comparison.
- Dominant language
- C++
- Stars
- 324
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from cloudinary/ssimulacra2
-
Difficulty 2/5 Under an hour Newbie friendliness 22/100
cloudinary/ssimulacra2#18 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
cloudinary/ssimulacra2#16 · 1 comment ·
All issues in cloudinary/ssimulacra2
Similar issues
-
std::fdim drops partial derivative w.r.t second argument in forward and reverse modePossibly taken @ADITYASINGH1206 claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 85/100
vgvassilev/clad#2534 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 61/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 3 days
-
HAP.cpp fixOpen
Difficulty 2/5 1-3 hours Newbie friendliness 68/100