FluxML/model-zoo

60 minute blitz uses stacked Dense layers with no activation function

Open

#339 opened on Mar 2, 2022

View on GitHub
 (1 comment) (1 reaction) (0 assignees)Julia (934 stars) (327 forks)batch import
good first issue

Description

In the 60 minute blitz tutorial, we use a sequence of stacked Dense layers, each with no activation function. This doesn't make much sense, as multiple linear operators can always be combined down into a single linear operator:

julia> using Flux
       model = Chain(
           Dense(200, 120, bias=false),
           Dense(120, 84, bias=false),
           Dense(84, 10, bias=false),
       )

       model_condensed = Chain(
           Dense(model[3].W * model[2].W * model[1].W),
       )

       x = randn(200)
       sum(abs, model(x) .- model_condensed(x))
2.4189600187907168e-6

While yes, there are machine precision/rounding issues that cause it to not be exactly equivalent, you don't get any material benefit from multiple stacked Dense layers, and in fact you get a performance penalty due to the same values moving in and out of CPU cache.

It would be better to either add nonlinearities between these Dense layers to increase model flexibility, or replace them with a single Dense layer that directly drops from rank 200 to 10.

Contributor guide

Tech stack
None
Domain
machine learningdocumentation
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
under 1 hour
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
FluxActivation functionsLinear algebra
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70
Research direction
Locate the file 'tutorials/60 minute blitz/60 minute blitz.jl' at lines 333-335. Replace the stacked Dense layers without activation with either adding a nonlinear activation (e.g., relu) between each Dense layer or collapsing them into a single Dense layer. Ensure the change is consistent with the tutorial's intent. No linked PRs exist yet, so submit a PR updating the tutorial code.