Flaky: upgrade test v0.3.0-HEAD fails when the old version's LLM extraction drops a fact
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 68/100
Research direction
Start with upgrade_tests/test_upgrade.py::TestUpgrade::test_upgrade_preserves_memories and inspect how the test seeds memories and checks the result after upgrading from v0.3.0. Compare the expected facts with what the old version actually extracted, or make the seed content reliably extractable. Done when extraction variance no longer causes a false upgrade regression while the test still checks that stored memories survive the upgrade.
Written by the indexing model from the issue text.
Description
In CI run 42212236193 (job test-upgrade), upgrade_tests/test_upgrade.py::TestUpgrade::test_upgrade_preserves_memories[v0.3.0-HEAD] failed with Alice not found in results after upgrade. The old-version stats already showed only 2 nodes (Bob, deadline): the real LLM on v0.3.0 never extracted the Alice fact, so the test failed before the upgrade was involved.
The assertion should compare against what the old version actually stored (or the seed content should make the fact unmissable), so an extraction variance does not read as an upgrade regression. Passed on rerun.
- Dominant language
- Python
- Stars
- 45.2k
- Forks
- 5.9k
- Avg merge
- 3h 44m
- Merged PRs (30d)
- 491
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from vectorize-io/hindsight
-
core
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
vectorize-io/hindsight#6430 ·
Maintainers usually reply within 1 day
-
core
Difficulty 1/5 Under an hour Newbie friendliness 83/100
vectorize-io/hindsight#6427 ·
Maintainers usually reply within 1 day
-
core
Difficulty 1/5 Under an hour Newbie friendliness 75/100
vectorize-io/hindsight#6416 ·
Maintainers usually reply within 1 day
-
core
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
vectorize-io/hindsight#6415 ·
Maintainers usually reply within 1 day
-
core
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
vectorize-io/hindsight#6396 ·
Maintainers usually reply within 1 day
All issues in vectorize-io/hindsight
Similar issues
-
ChatBedrockConverse changes the keys of JSON tool results to camelCase (`order_id` becomes `orderId`)Possibly taken @geyazheng claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
langchain-ai/langchain-aws#1469 ·
Maintainers usually reply within 1 day
-
GCToggler leaves collection enabled if the body re-enables itPossibly taken @Gambit-Checkmate claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 7 days
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
doronz88/pymobiledevice3#2176 ·
Maintainers usually reply within 1 day
-
b4b usability
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day