Online adaptation advances observation history multiple times per policy query

Abierto
#66 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@sylvesterkaczmarek ya está trabajando en esto.

Desde el 30/10/2026.

  • #67 de @sylvesterkaczmarek — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
52/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python, pytorch

Línea de trabajo

Start with _prepare, embed_for_buffer, propose_noise, act, and invert in rho/online/adaptation.py, then trace the call sequence in environments/metaworld/online_adapt.py’s run_episode. Reproduce the two-frame example and inspect how history and padding change across helper calls. Done when history advances once per query, inversion uses that query’s snapshot, episode resets clear history, and regression coverage exercises the listed rollout cases.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

A single policy query in environments/metaworld/online_adapt.py calls embed_for_buffer(chunk_obs), optionally propose_noise(chunk_obs), and then act(chunk_obs, noise). Each path calls PolicyInterface.process_observation(), which appends the observation to its temporal queues. Finalizing a corrected action chunk calls invert(chunk_obs, ...), which appends the same starting observation again.

Consequently, observation history represents helper calls rather than distinct incoming observations. The conditioning used to collect a noise-policy training example can differ from the conditioning used to generate or invert its action chunk. Padding masks can also claim that an incomplete history is fully populated.

Affected code at 148bb1ae75b92a1162172ea36d8e5ceb830ceebb:

  • rho/online/adaptation.py: _prepare, embed_for_buffer, propose_noise, act, and invert
  • environments/metaworld/online_adapt.py: run_episode

Reproduction

Use a policy with observation deltas [-2, -1, 0], identity input transforms, and successive single-frame observations with values 1 and 2. For each query, follow the rollout's sequence:

embedding = adapter.embed_for_buffer(obs)
noise = adapter.propose_noise(obs)
actions = adapter.act(obs, noise)

On the second query, the model should see history [1, 1, 2] and padding mask [True, False, False]. It instead sees [2, 2, 2] with [False, False, False]. The first query's padding is already wrong after the repeated insertions.

This was reproduced through the actual run_episode() and PolicyInterface using a deterministic local environment and a lightweight policy, without model downloads or physical hardware. The Gaussian-noise path is also affected: it still calls both embed_for_buffer() and act(). Inversion of corrected chunks adds another history update.

The rollout also resets the environment without resetting PolicyInterface, so multi-step history can cross episode boundaries. Single-frame policies are unaffected by this particular history-ordering failure. This report does not establish an error in any published benchmark result.

Proposed fix

Prepare one explicit observation snapshot per query, advancing the history once. Reuse its transformed inputs for embedding, noise proposal, and action generation. Retain its raw, remapped history separately so executed physical actions can be normalized against the original query state during inversion, without advancing the queues or consulting a later query's history.

Reset the interface at episode boundaries. Add regression coverage for real rollout call sequences, Gaussian and learned noise, corrected and uncorrected chunks, history padding, repeated helper calls, consecutive multi-frame observations, delayed inversion, and reused environment buffers.

Lenguaje dominante
Python
Estrellas
38
Forks
6
Merge medio
2 d 9 h
PR fusionados (30 d)
20

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/rhobotics

Todos los issues de microsoft/rhobotics

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.