facebookresearch/fairseq

key_padding_mask is not used in transformer decoder layer?

Open

#537 opened on Feb 27, 2019

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Python (29,107 stars) (6,224 forks)batch import
bughelp wanted

Description

when reading the source code i found that key_padding_mask is not used when calculating self attention. There is no problem when the target is padded right by default because attn_mask could do the same thing. But how about left padding on the target?

Contributor guide

Tech stack
pythonpytorch
Domain
machine learning
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
PythonPyTorchTransformer attentionfairseq internals
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Investigate the fairseq transformer decoder layer implementation, likely in `fairseq/models/transformer.py`. Look for the self attention computation and how `key padding mask` is handled. Compare with the encoder layer or other transformer implementations to identify where the mask should be applied. Consider testing with left padded targets to confirm the issue. Propose a fix by modifying the attention function to incorporate `key padding mask`.
key_padding_mask is not used in transformer decoder layer? · facebookresearch/fairseq#537 | Good First Issue