Add the possibility to pretrain on multiple tasks

For the moment, the nc_benchmark generator function allows for a nc_first_task option, which is good for pre-training in the class-incremental learning scenario. However, the same kind of option is not available if one wants to pretrain in the task-incremental scenario. It would be nice to have an option that could be used together with task_labels=True and allows for pretraining on multiple tasks at the same time, in a multitask training manner.

This kind of pre-training is used for instance in Lifelong Learning of Compositional Structures

A quick fix that I'm using for now, but that is breaking some things (maybe to be put in bugs?) is the following:

# Number of tasks to pretrain on
pretrain = 4
pretrain_datasets = [exp.dataset for exp in scenario.train_stream[:pretrain]]

# Modify the first experience so that it contains data of the 4 first ones
first_experience = scenario.train_stream[0]
first_experience.dataset = AvalancheConcatDataset(pretrain_datasets)

# Train on the modified first experience
cl_strategy.train(first_experience)

# Train on the rest of the experiences
for t, experience in enumerate(scenario.train_stream[pretrain:]):
    cl_strategy.train(experience)

Doing this works as intended except that it multiplies the batch_size by the number of pretraining tasks for some reason:

• size of strategy.mb_x when pretrain=4: (256, 3, 32, 32)
• size of strategy.mb_x when pretrain=1: (64, 3, 32, 32)