Local multi-headed self-attention · pyg-team/pytorch

(3 commentaires) (1 réaction) (0 personne assignée)Python (3 514 forks)batch import

featurehelp wanted

Métriques du dépôt

I am unable to find the clean implementation of local multi-headed self-attention in pytorch geometric. I found three types of multi-head attention, one TransformerConv (https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.TransformerConv.html#torch_geometric.nn.conv.TransformerConv). But this one calculates a linear combination of all features with different attention weights as opposed to dividing features into multiple heads and taking their linear combination: another RGATConv in the similar direction (https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.RGATConv.html). And finally GPSConv (https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.conv.GPSConv.html) that does multi-head attention but is global.

I think it is nice to have the implementation of local self-attention with multiple heads where each head looks into a part of the feature dimension.

No response

Direction de recherche: Implémenter un mécanisme d'auto attention locale multi tête qui divise les dimensions des caractéristiques entre les têtes, similaire à TransformerConv mais avec des sous espaces de caractéristiques par tête. Étudier les implémentations existantes comme TransformerConv, RGATConv et GPSConv dans PyTorch Geometric pour comprendre les motifs et les étendre.
Stack technique: pythonpytorch
Domaine: machine learning
Type d'issue: Fonctionnalité
Difficulté: 3
Temps estimé: 1-2 jours
Statut d'activité: Active
Clarté: Plutôt claire
Prérequis: PythonPyTorchGraph Neural Networks basics
Accessibilité débutant: 30