facebookresearch/ParlAI

Support tied positional weights in transformers

Open

#2,184 opened on Nov 19, 2019

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Python (10,371 stars) (2,089 forks)batch import
EnhancementH1 2020Help WantedMediumdonotreap

Description

Add a flag to use tied positional embeddings in transformer/generator and transformer/retrieval, and implement the tied weights. Should be False by default for backwards compatibility, and upgrade_opt should be used to migrate old models.

Contributor guide

Tech stack
python
Domain
machine learning
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Understanding of transformer architecturesFamiliarity with ParlAI codebasePython programming
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Examine the transformer generator and retrieval modules in ParlAI (likely under parlai/agents/transformer/). Review how positional embeddings are currently implemented. Add a new boolean flag for tied positional weights, defaulting to False. Implement the tied weights logic and use upgrade opt to handle migration of old models. Check issue #2184 comments for any additional guidance.