Parallelise _only_ the convolution+loss

Open
#2,434 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
machine-learning, python

Research direction

The issue asks to refactor the training loop so that convolution and loss computation can run in parallel per model. Start by locating where models are stacked and fitted, then identify the fktable and loss computation that happens after sampling. The goal is to separate these components so each model trains independently without weight freezing, enabling per-model parallelization.

Written by the indexing model from the issue text.

Description

enhancement

This is probably quite complicated to do without harming the performance of the "single-model" training but it could be quite interesting.

Right now we are stacking together all the models but keeping track of the separated component so that we can stop the fit for each of them independently.
This is very efficient until the fits don't stabilise. Then one by one the fit stop (by freezing the weights) until eventually the last one stops and the fit finishes.

However, the models we use are not very heavy. Most (almost all in the tests we did... 3 years ago at this point...) the advantage of GPU running happens after the model is sampled: fktable and loss. We can probably separate them completely such that each model trains completely by itself (meaning we have complete freedom on the model side, and are not restricted to the architecture).

Dominant language
Python
Stars
44
Forks
15
Avg merge
8d 12h
Merged PRs (30d)
7

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NNPDF/nnpdf

All issues in NNPDF/nnpdf

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.