Add the possibility to pretrain on multiple tasks
#743 geöffnet am 15. Sept. 2021
Repository-Metriken
- Stars
- (1.516 Stars)
- PR-Merge-Metriken
- (Keine gemergten PRs in 30 T)
Beschreibung
For the moment, the nc_benchmark generator function allows for a nc_first_task option, which is good for pre-training in the class-incremental learning scenario. However, the same kind of option is not available if one wants to pretrain in the task-incremental scenario. It would be nice to have an option that could be used together with task_labels=True and allows for pretraining on multiple tasks at the same time, in a multitask training manner.
This kind of pre-training is used for instance in Lifelong Learning of Compositional Structures
A quick fix that I'm using for now, but that is breaking some things (maybe to be put in bugs?) is the following:
# Number of tasks to pretrain on
pretrain = 4
pretrain_datasets = [exp.dataset for exp in scenario.train_stream[:pretrain]]
# Modify the first experience so that it contains data of the 4 first ones
first_experience = scenario.train_stream[0]
first_experience.dataset = AvalancheConcatDataset(pretrain_datasets)
# Train on the modified first experience
cl_strategy.train(first_experience)
# Train on the rest of the experiences
for t, experience in enumerate(scenario.train_stream[pretrain:]):
cl_strategy.train(experience)
Doing this works as intended except that it multiplies the batch_size by the number of pretraining tasks for some reason:
- size of strategy.mb_x when pretrain=4: (256, 3, 32, 32)
- size of strategy.mb_x when pretrain=1: (64, 3, 32, 32)