ignore_modules in Quantization via Bitsandbytes

Description & Motivation

Currently, the strings provided in ignore_modules will be checked in as a prefix. This will cause problems when someone wants to quantise specific module and ignore some of the modules. Like in the following module list:

llama.visual_backbone.visual.transformer.resblocks.7.mlp.c_fc
llama.visual_backbone.visual.transformer.resblocks.7.mlp.gelu
llama.visual_backbone.visual.transformer.resblocks.7.mlp.c_proj
llama.visual_backbone.visual.transformer.resblocks.7.ln_2
llama.visual_backbone.visual.transformer.resblocks.7.adapter_attn
llama.visual_backbone.visual.transformer.resblocks.7.adapter_attn.conv_A
llama.visual_backbone.visual.transformer.resblocks.7.adapter_attn.conv_B
llama.visual_backbone.visual.transformer.resblocks.8
llama.visual_backbone.visual.transformer.resblocks.8.attn
llama.visual_backbone.visual.transformer.resblocks.8.attn.out_proj
llama.visual_backbone.visual.transformer.resblocks.8.ln_1
llama.visual_backbone.visual.transformer.resblocks.8.mlp
llama.visual_backbone.visual.transformer.resblocks.8.mlp.c_fc
llama.visual_backbone.visual.transformer.resblocks.8.mlp.gelu
llama.visual_backbone.visual.transformer.resblocks.8.mlp.c_proj
llama.visual_backbone.visual.transformer.resblocks.8.ln_2
llama.visual_backbone.visual.transformer.resblocks.8.adapter_attn
llama.visual_backbone.visual.transformer.resblocks.8.adapter_attn.conv_A
llama.visual_backbone.visual.transformer.resblocks.8.adapter_attn.conv_B

there are many layers of resblocks, when I want to quantize all layers other than conv_A and conv_B, I can't do that because in the ignore_modules I've to provide complete path to resblocks and since there are many layers, it's simply not possible.

Pitch

What I suggest is to instead of checking prefix, one should search for module while ignoring, like:

if isinstance(child, torch.nn.Linear) and not any(s in fullname for s in ignore_modules):

Alternatives

No response

Additional context

No response

cc @borda @carmocca @awaelchli