pytorch/examples

Actor critic example not using discount rate properly

Open

Aperta il 27 mar 2020

Vedi su GitHub
 (3 commenti) (0 reazioni) (0 assegnatari)Python (21.634 star) (9429 fork)batch import
good first issuetriaged

Descrizione

The Actor Critic example (which is actually an implementation of REINFORCE-with-baseline as pointed out in https://github.com/pytorch/examples/issues/573), does not use the discount rate properly.

The loss should include \gamma ^ t, as shown in the box on page 330 of Sutton & Barto:

image

Guida contributor

Tech stack
pythonpytorch
Dominio
machine learningai
Tipo issue
bug
DifficoltàQuanto dovrebbe essere impegnativa per un nuovo contributor.
3
Tempo stimatoTempo stimato per completare e verificare uno scope piccolo.
1-3 hours
Stato attivitàQuanto è probabile che la issue sia ancora attiva e reviewable.
stale
ChiarezzaQuanto chiaramente la issue descrive problema, scope e risultato atteso.
clear
Prerequisiti
REINFORCE algorithmPyTorch basicsDiscount factor concept
Adatta ai principiantiQuanto la issue è adatta a contributor alla prima esperienza.
45
Direzione di ricerca
Examine the actor critic example file (likely in the reinforcement learning folder). The current code likely computes the loss without weighting by gamma^t. Refer to the referenced issue #573 and the Sutton & Barto equation. The fix involves multiplying each log probability term by gamma^t in the policy gradient loss calculation. Check the comments in the issue for any additional guidance.