WMT21 & WMT22 · huggingface/datasets#4709

(10 评论) (0 反应) (1 负责人)Python (18,313 star) (2,496 fork)batch import

dataset requestgood first issue

描述

Adding a Dataset

Name: WMT21 & WMT22
Description: We are going to have three tracks: two small tasks and a large task. The small tracks evaluate translation between fairly related languages and English (all pairs). The large track uses 101 languages.
Paper: /
Data: https://statmt.org/wmt21/large-scale-multilingual-translation-task.html https://statmt.org/wmt22/large-scale-multilingual-translation-task.html
Motivation: Many more languages than previous WMT versions - Could be very high impact

Instructions to add a new dataset can be found here.

I could also tackle this. I saw the existing logic for WMT models is a bit complex (datasets are stored on the wmt account & retrieved in separate wmt datasets afaict). How long do you think it would take me? @lhoestq

贡献者指南

技术栈: python
领域: datamachine learning
议题类型: feature
难度: 3
预计时间: 3-5 days
活动状态: blocked
清晰度: mostly clear
前置要求: Python programmingHugging Face Datasets libraryWMT translation tasksGit and GitHub basics
新手友好度: 40
研究方向: Review the existing WMT dataset implementation in the repository (likely in datasets/wmt*.py) to understand the pattern. Follow the ADD NEW DATASET.md guide. There is a current assignee, so verify if the work is still needed. Then implement the dataset loading script for WMT21 and WMT22 based on the provided data URLs.