Supporting non-vision models

Open
#18 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
python, pytorch
Domain
machine-learning

Research direction

Start by tracing get_previous_layer to the re.sub call that receives None, then compare its Conv2d/BatchNorm2d assumptions with the reported Conv1d, Linear, LayerNorm, GELU, transpose, dropout, and add chains. Done means documenting which predecessor should be selected for non-vision models and defining behavior when no supported layer is found.

Written by the indexing model from the issue text.

Description

Hi,
could you explain how to use this for non-vision models?
Trying to apply this to an audio model I am getting an error because get_previous_layer never finds a Conv2d or a BatchNorm2d and hence, after recursing through all the modules, just returns None, which then fails at re.sub.

Could you explain the logic behind "going back to the previous layer of exactly this type"?

the immediate earlier layers are: (add, causing the search) -> [transpose] -> gelu -> transpose -> layer_norm -> transpose -> conv1d -> ...

which layer would you expect to find here? Are you looking for the last layer that learns anything (which would be layer norm) or with actual learnable parameters (then it's conv1d)

there is a second case where this happens where the chain is: (add, causing the search) -> [dropout] -> linear -> layer_norm -> transpose -> gelu -> transpose -> layer_norm -> transpose -> conv1d -> ...

again, please help me out which layer should be found. I would guess the linear layer?

on a side note: instead of deep recursion, checking the whole list each time, building a tree or doubly-linked-list-like structure seems more appropriate (and easier to debug)

Dominant language
Python
Stars
36
Forks
3
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from EIDOSLAB/simplify

All issues in EIDOSLAB/simplify

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.