carpedm20/BEGAN-tensorflow

Training loss goes to NaN

Open

#11 opened on Apr 14, 2017

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Python (233 forks)github user discovery
bughelp wanted

Repository metrics

Stars
 (924 stars)
PR merge metrics
 (PR metrics pending)

Description

I started training with Celeb database, but around 50k iteration, the loss goes to NaN & k_t goes to 1. Have you seen this before. 10%|█████▎ | 49650/500000 [5:29:56<49:13:32, 2.54it/s][49650/500000] Loss_D: 0.112764 Loss_G: 0.053848 measure: 0.1190, k_t: 0.0458 10%|█████▎ | 49700/500000 [5:30:16<49:18:01, 2.54it/s][49700/500000] Loss_D: nan Loss_G: nan measure: nan, k_t: 1.0000

Having said that, the output at 49k is starting to look good. Below are my last good outputs (real, d_real, g, d_fake). x_fixed 49500_d_real 49500_g 49500_d_fake

My training stats looks like this began_training_screenshot

Contributor guide

Research direction
Investigate common causes of NaN loss in GAN training: check for vanishing/exploding gradients, learning rate too high, or numerical instability. Review the BEGAN implementation for potential issues in loss calculation or optimization.
Tech stack
pythontensorflow
Domain
machine learningai
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonTensorFlow
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65