Online adaptation advances observation history multiple times per policy query
I maintainer di solito rispondono entro 1 giorno
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 52/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Ambito
- machine-learning
Direzione di ricerca
Start with _prepare, embed_for_buffer, propose_noise, act, and invert in rho/online/adaptation.py, then trace the call sequence in environments/metaworld/online_adapt.py’s run_episode. Reproduce the two-frame example and inspect how history and padding change across helper calls. Done when history advances once per query, inversion uses that query’s snapshot, episode resets clear history, and regression coverage exercises the listed rollout cases.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Problem
A single policy query in environments/metaworld/online_adapt.py calls embed_for_buffer(chunk_obs), optionally propose_noise(chunk_obs), and then act(chunk_obs, noise). Each path calls PolicyInterface.process_observation(), which appends the observation to its temporal queues. Finalizing a corrected action chunk calls invert(chunk_obs, ...), which appends the same starting observation again.
Consequently, observation history represents helper calls rather than distinct incoming observations. The conditioning used to collect a noise-policy training example can differ from the conditioning used to generate or invert its action chunk. Padding masks can also claim that an incomplete history is fully populated.
Affected code at 148bb1ae75b92a1162172ea36d8e5ceb830ceebb:
rho/online/adaptation.py:_prepare,embed_for_buffer,propose_noise,act, andinvertenvironments/metaworld/online_adapt.py:run_episode
Reproduction
Use a policy with observation deltas [-2, -1, 0], identity input transforms, and successive single-frame observations with values 1 and 2. For each query, follow the rollout's sequence:
embedding = adapter.embed_for_buffer(obs)
noise = adapter.propose_noise(obs)
actions = adapter.act(obs, noise)
On the second query, the model should see history [1, 1, 2] and padding mask [True, False, False]. It instead sees [2, 2, 2] with [False, False, False]. The first query's padding is already wrong after the repeated insertions.
This was reproduced through the actual run_episode() and PolicyInterface using a deterministic local environment and a lightweight policy, without model downloads or physical hardware. The Gaussian-noise path is also affected: it still calls both embed_for_buffer() and act(). Inversion of corrected chunks adds another history update.
The rollout also resets the environment without resetting PolicyInterface, so multi-step history can cross episode boundaries. Single-frame policies are unaffected by this particular history-ordering failure. This report does not establish an error in any published benchmark result.
Proposed fix
Prepare one explicit observation snapshot per query, advancing the history once. Reuse its transformed inputs for embedding, noise proposal, and action generation. Retain its raw, remapped history separately so executed physical actions can be normalized against the original query state during inversion, without advancing the queues or consulting a later query's history.
Reset the interface at episode boundaries. Add regression coverage for real rollout call sequences, Gaussian and learned noise, corrected and uncorrected chunks, history padding, repeated helper calls, consecutive multi-frame observations, delayed inversion, and reused environment buffers.
- Lingua principale
- Python
- Stelle
- 38
- Fork
- 6
- Merge medio
- 2g 9h
- PR unite (30g)
- 20
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di microsoft/rhobotics
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 3/5 1-2 giorni Idoneità per principianti 45/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 25/100
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di microsoft/rhobotics
Issue simili
-
Difficoltà 1/5 1-3 ore Idoneità per principianti 76/100
samuelclay/NewsBlur#2214 ·
I maintainer di solito rispondono entro 1 giorno
-
MODEL_NOT_FOUND gives no suggestion for a model listed only as model:profileForse già presa @ShaiShmuel l’ha presa oggi. Apertaenhancement rust
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
superlinked/sie#985 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
readthedocs/readthedocs-builder#73 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
reflex-dev/reflex#7915 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno