clab/dynet

non-deterministic behavior on the GPU?

Open

#399 opened on Mar 23, 2017

View on GitHub
 (7 comments) (0 reactions) (0 assignees)C++ (701 forks)github user discovery
good first issueminor bug

Repository metrics

Stars
 (3,433 stars)
PR merge metrics
 (PR metrics pending)

Description

I am unsure why, but we observe some non-deterministic behavior when running on the GPU. The dynet-seed is set (as well as the python seed and the numpy seed), yet the losses in different runs start to diverge after a while (by the 100th update, is almost surely happens).

This does not happen with the same code on the CPU. Maybe there is a race condition somewhere?

See the following gist: https://gist.github.com/yoavg/510154a959e76370627a81da173cec9b

python not_deterministic.py > t1 python not_deterministic.py > t2 diff t1 t1

Contributor guide

Research direction
Inspect the gist code and trace the random number generation on GPU, check for race conditions in operations.
Tech stack
cpppython
Domain
machine learning
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
C++Python
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40