non-deterministic behavior on the GPU?

I am unsure why, but we observe some non-deterministic behavior when running on the GPU. The dynet-seed is set (as well as the python seed and the numpy seed), yet the losses in different runs start to diverge after a while (by the 100th update, is almost surely happens).

This does not happen with the same code on the CPU. Maybe there is a race condition somewhere?

See the following gist: https://gist.github.com/yoavg/510154a959e76370627a81da173cec9b

python not_deterministic.py > t1 python not_deterministic.py > t2 diff t1 t1