rlworkgroup/garage

Add MAML Sampler for faster meta-task sampling

開放

#1,115 建立於 2019年12月31日

 (4 則留言) (0 個反應) (0 位負責人)Python (297 個分叉)batch import
good first issuetf

倉庫指標

星標
 (1,728 顆星)
PR 合併指標
 (30 天內沒有已合併 PR)

描述

I found that ProMP does sampling much faster than garage. This is because ProMP has a specialized sampler, call MAML Sampler, that parallelizes sampling at task-level. I think this is also important for garage.

A MAML Sampler is a sampler that samples all tasks in one run (i.e. one call to sampler.obtain_samples(). This is contrary to the current design of sampler, which handles a single task once at a time. MAML sampler has control for task-level scheduling, so it allows parallelism at task-level.

Under MAML sampler, the training loop will be similar to something like

sampler = MAMLSampler(tasks)
for many batches
    policies = num_tasks copies of policy
    paths_batch = []
    for some gradient steps
        paths_all_tasks = sampler.obtain_samples(policies)
        update policies using paths_all_tasks
        add paths_all_tasks to paths_batch
    optimize policy using policies and paths_batch

while currently, a MAML training loop has to switch task outside of sampler. Although sampler does parallel sampling at rollout-level, this has a higher overhead than the above MAML sampler.

for many batches
    policies = num_tasks copies of policy
    paths_batch = []
    for tasks[i]
        for some gradient steps
            paths = sampler.obtain_samples(policies[i], tasks[i])
            update policies[i] using paths
            add paths to paths_batch
    optimize policy using policies and paths_batch

貢獻者指南