FluxML/Flux.jl

'cell_clip' and 'proj_clip' for Flux: LSTM

Open

#653 opened on Mar 1, 2019

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Julia (4,725 stars) (619 forks)batch import
enhancementhelp wanted

Description

Looking through the docs. , I found that this feature is missing for LSTM layers. It comes as an optional parameter in tensorflow LSTMCell This will be useful for porting stuff to flux while achieving the same results as from tensorflow implementations for some popular algorithms (like ELMo). Looking in the code base of tensorflow if says that it clip the value of cell states between [-cell_clip, cell_clip] in it's clip_by_value method.

Contributor guide

Tech stack
None
Domain
machine learning
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Familiarity with LSTM internalsJulia basicsFlux.jl library structure
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
The issue requests adding cell clip and proj clip parameters to LSTM layers in Flux.jl. To implement, first locate the existing LSTM implementation in the Flux source, likely in `src/layers/stateless.jl` or similar. Study how TensorFlow's LSTMCell applies clipping using `clip by value`. Then modify the Flux LSTM forward pass to optionally clip cell and hidden states. Consider adding a new keyword argument to the LSTM constructor and integrating it into the recurrent cell computation. Finally, write tests to verify the clipping behavior matches TensorFlow's. Review any linked PRs or comments for additional context.
'cell_clip' and 'proj_clip' for Flux: LSTM · FluxML/Flux.jl#653 | Good First Issue