rlworkgroup/garage

Add MAML Sampler for faster meta-task sampling

オープン

#1,115 opened on 2019/12/31

 (4 件のコメント) (0 件のリアクション) (0 人の担当者)Python (297 件のフォーク)batch import
good first issuetf

Repository metrics

Stars
 (1,728 個のスター)
PR merge metrics
 (30d に merged PR はありません)

説明

I found that ProMP does sampling much faster than garage. This is because ProMP has a specialized sampler, call MAML Sampler, that parallelizes sampling at task-level. I think this is also important for garage.

A MAML Sampler is a sampler that samples all tasks in one run (i.e. one call to sampler.obtain_samples(). This is contrary to the current design of sampler, which handles a single task once at a time. MAML sampler has control for task-level scheduling, so it allows parallelism at task-level.

Under MAML sampler, the training loop will be similar to something like

sampler = MAMLSampler(tasks)
for many batches
    policies = num_tasks copies of policy
    paths_batch = []
    for some gradient steps
        paths_all_tasks = sampler.obtain_samples(policies)
        update policies using paths_all_tasks
        add paths_all_tasks to paths_batch
    optimize policy using policies and paths_batch

while currently, a MAML training loop has to switch task outside of sampler. Although sampler does parallel sampling at rollout-level, this has a higher overhead than the above MAML sampler.

for many batches
    policies = num_tasks copies of policy
    paths_batch = []
    for tasks[i]
        for some gradient steps
            paths = sampler.obtain_samples(policies[i], tasks[i])
            update policies[i] using paths
            add paths to paths_batch
    optimize policy using policies and paths_batch

コントリビューターガイド