Lable Mapping not aligned when prefixes are set to be an empty string

Reproduce the failure with all prefixes set to empty strings, then trace compute_duplicated_labels() and the activation_storage[n] assignment where the tensor-size error appears. Compare the sizes of idx_map and dummy_labels; the issue is done when empty prefixes produce matching label sizes without the runtime error.

Hi @ericwtodd

I am trying to play with different prefixes. The default one is: prefixes = {"input":"Q:", "output":"A:","instructions":""}. When I change it to empty string prefixes = {"input":"", "output":"","instructions":""}, I get a error saying that:

activation_storage[n] = stack_filtered
RuntimeError: The expanded size of the tensor (54) must match the existing size (53) at non-singleton dimension 2.  Target sizes: [40, 40, 54, 128].  Tensor sizes: [40, 40, 53, 128]

I have further checked it and found that this is caused by different label size (the size of idx_map is different from dummy_labels).
When the prefix is made of empty strings, the function compute_duplicated_labels() will an index_map that is of different size of dummy label. I try to debug where went wrong and found it too hard to dive in...

The error only pops up when I have empty strings for prefixes. All other prefixes do not have this problem....

It would be great if you could share some insights on this. I wonder have you experienced similar problems when using empty string prefixes...