Microsoft/CNTK

Implementation of a random sampling node -> minibatch layout troubles

Open

#1371 opened on Jan 27, 2017

View on GitHub
 (9 comments) (0 reactions) (1 assignee)C++ (16,085 stars) (4,260 forks)batch import
help wanted

Description

Hi CNTK Team!

I'm currently trying to implement a Variational Autoencoder using CNTK. I'd need functions for random sampling (like random_normal() in Tensorflow) for this. CNTKs backend already has random distributions in it's matrix lib so I started implementing a ComputationNode that would use these to fill the value matrix with random numbers during the forward pass. So far, not too difficult...

However, I'm struggling to get the minibatch layout consistent. Since my random sampling node has no inputs (parameters of the random distribution are static), I have found no proper way to infer the minibatch layout for the output automatically.

I tried specifying the minibatch layout via static parameters, but this is not very elegant and leads to other nodes in the computation graph complaining about an inconsistent layout with regard to sequences.

I could also use a node in the computation graph that has the correct minibatch layout as additional input to my random sampling node and then use this only to get the minibatch layout from. However, this would mean introducing an input variable to my node and then not use it's actual values. Also not very elegent...

Do you have any tips on how to do this correctly (or maybe I'm just missing something obvious)?

If this will result in a usable implementation, I'd be happy to contribute my code.

Enrico

Contributor guide

Tech stack
cpp
Domain
machine learning
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
C++CNTK ComputationNodeminibatch layout
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
Investigate the CNTK source code for existing nodes that generate random values, such as those in the Matrix library, to understand how they handle minibatch layout. Examine the ComputationNode base class and review how constant nodes propagate layout. The issue creator proposed using an additional input node to inherit the layout; evaluate whether this approach is acceptable or if there is a cleaner method.