pytorch/examples

Ambiguous code in reinforce

Open

#297 opened on Feb 2, 2018

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Python (21,634 stars) (9,429 forks)batch import
good first issue

Description

In /reinforcement_learning/reinforce.py, line 91:

running_reward = running_reward * 0.99 + t * 0.01

The variable running_reward seems to used for record average episodic rewards(not actually average, but I think the concept is similar), 0.01, is the scalar to update the average episodic rewards and t is done step. Add some comment or refactor naming may help beginners to understand this example.

Contributor guide

Tech stack
python
Domain
documentationmachine learning
Issue type
documentation
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
1
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
under 1 hour
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
basic Pythonreinforcement learning basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
90
Research direction
The issue points to line 91 in `/reinforcement learning/reinforce.py`. The variable `running reward` and the update formula `running reward * 0.99 + t * 0.01` are confusing. A comment explaining that this computes an exponential moving average of episode rewards, or renaming `running reward` to something like `avg reward`, would clarify the code. Check the file for existing comments and coding style. No linked pull requests or maintainer responses are present. Ensure the change is consistent with other examples in the repository.