Data loading in multi-lingual translation
#2.410 aberto em 2 de ago. de 2020
Métricas do repositório
- Stars
- (29.107 stars)
- Métricas de merge de PR
- (Nenhuma PRs mesclada em 30d)
Description
Hello, I use the multi-lingual translation task and find some issues:
-
When I use the round_robin_dataset and multi_corpus_sampled_dataset for more than 20 language pairs, the data loading will take much more time in the functions of filter_by_size and batch_by_size. I find that in multi_corpus_sampled_dataset, the computation cost in calling the function of num_tokens is O(N^2). Could you provide some optimization for these two datasets?
-
The data loading will fail (RuntimeError: received 0 items of ancdata) when I use more language pairs, such as 28 language pairs. I find there is a daemon thread, which supports the buffer mechanism. When I close the buffer mechanism, the data loading can succeed. I do not figure out why this happens. Could you fix this issue? Thank you very much.