Collapse reference+learner hydra heads when using LoRa · CarperAI/trlx#320

(6 comments) (2 reactions) (0 assignees)Python (450 forks)batch import

contributions welcomefeature requestgood first issue

Repository metrics

Stars: (4,184 stars)
PR merge metrics: (30d に merged PR はありません)

説明

🚀 The feature, motivation, and pitch

With additive (delta-style) parameter-efficient tuning methods such as LoRa, we should be able to make a slightly more mem-efficient hydra architecture by using a single block that does ~frozen_head + tunable_weights for the learner/policy head's fwd-pass and simply frozen_head for the reference, instead of maintaining 2x heads.

CC @LouisCastricato and @cat-state for pointing this out

Alternatives

No response

Additional context