Data loading in multi-lingual translation
#2.410 geöffnet am 2. Aug. 2020
Repository-Metriken
- Stars
- (29.107 Stars)
- PR-Merge-Metriken
- (Keine gemergten PRs in 30 T)
Beschreibung
Hello, I use the multi-lingual translation task and find some issues:
-
When I use the round_robin_dataset and multi_corpus_sampled_dataset for more than 20 language pairs, the data loading will take much more time in the functions of filter_by_size and batch_by_size. I find that in multi_corpus_sampled_dataset, the computation cost in calling the function of num_tokens is O(N^2). Could you provide some optimization for these two datasets?
-
The data loading will fail (RuntimeError: received 0 items of ancdata) when I use more language pairs, such as 28 language pairs. I find there is a daemon thread, which supports the buffer mechanism. When I close the buffer mechanism, the data loading can succeed. I do not figure out why this happens. Could you fix this issue? Thank you very much.