Online adaptation advances observation history multiple times per policy query

Aperta
#66 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@sylvesterkaczmarek ci sta già lavorando.

Dal 30/10/2026.

  • #67 di @sylvesterkaczmarek — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
52/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, pytorch

Direzione di ricerca

Start with _prepare, embed_for_buffer, propose_noise, act, and invert in rho/online/adaptation.py, then trace the call sequence in environments/metaworld/online_adapt.py’s run_episode. Reproduce the two-frame example and inspect how history and padding change across helper calls. Done when history advances once per query, inversion uses that query’s snapshot, episode resets clear history, and regression coverage exercises the listed rollout cases.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem

A single policy query in environments/metaworld/online_adapt.py calls embed_for_buffer(chunk_obs), optionally propose_noise(chunk_obs), and then act(chunk_obs, noise). Each path calls PolicyInterface.process_observation(), which appends the observation to its temporal queues. Finalizing a corrected action chunk calls invert(chunk_obs, ...), which appends the same starting observation again.

Consequently, observation history represents helper calls rather than distinct incoming observations. The conditioning used to collect a noise-policy training example can differ from the conditioning used to generate or invert its action chunk. Padding masks can also claim that an incomplete history is fully populated.

Affected code at 148bb1ae75b92a1162172ea36d8e5ceb830ceebb:

  • rho/online/adaptation.py: _prepare, embed_for_buffer, propose_noise, act, and invert
  • environments/metaworld/online_adapt.py: run_episode

Reproduction

Use a policy with observation deltas [-2, -1, 0], identity input transforms, and successive single-frame observations with values 1 and 2. For each query, follow the rollout's sequence:

embedding = adapter.embed_for_buffer(obs)
noise = adapter.propose_noise(obs)
actions = adapter.act(obs, noise)

On the second query, the model should see history [1, 1, 2] and padding mask [True, False, False]. It instead sees [2, 2, 2] with [False, False, False]. The first query's padding is already wrong after the repeated insertions.

This was reproduced through the actual run_episode() and PolicyInterface using a deterministic local environment and a lightweight policy, without model downloads or physical hardware. The Gaussian-noise path is also affected: it still calls both embed_for_buffer() and act(). Inversion of corrected chunks adds another history update.

The rollout also resets the environment without resetting PolicyInterface, so multi-step history can cross episode boundaries. Single-frame policies are unaffected by this particular history-ordering failure. This report does not establish an error in any published benchmark result.

Proposed fix

Prepare one explicit observation snapshot per query, advancing the history once. Reuse its transformed inputs for embedding, noise proposal, and action generation. Retain its raw, remapped history separately so executed physical actions can be normalized against the original query state during inversion, without advancing the queues or consulting a later query's history.

Reset the interface at episode boundaries. Add regression coverage for real rollout call sequences, Gaussian and learned noise, corrected and uncorrected chunks, history padding, repeated helper calls, consecutive multi-frame observations, delayed inversion, and reused environment buffers.

Lingua principale
Python
Stelle
38
Fork
6
Merge medio
2g 9h
PR unite (30g)
20

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/rhobotics

Tutte le issue di microsoft/rhobotics

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.