Consolidation: strict schema requires `observation_text`, but the prompt says to leave it out for unrelated observations — small models run to the token cap
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 70/100
Research direction
Read hindsight-api-slim/hindsight_api/engine/structured_output.py around line 77 (OpenAIStrictSchemaGenerator, where every property is forced into required) alongside engine/consolidation/prompts.py line ~121, which tells the model to omit observation_text for unrelated observations. Reproduce with HINDSIGHT_API_LLM_STRICT_SCHEMA_CONSOLIDATION=true and a small model, and unit-test the generated schema's required list. Done when prompt and schema agree on the unrelated case and generation terminates. Note #6673 and the unconfirmed field-name-as-text symptom.
Written by the indexing model from the issue text.
Description
Seen on self-hosted Hindsight 1.2.1 (rolled back to 1.1.0 since), checked again against v1.2.3 and main (80bd005fc). Paths are under hindsight-api-slim/hindsight_api/.
Setup: OpenAI-compatible provider serving a small Japanese model (sarashina3-mini), HINDSIGHT_API_LLM_STRICT_SCHEMA_CONSOLIDATION=true.
What happens
Since #6164 (1.2.0, one answer per fact), consolidation batches sometimes never finish: the model emits whitespace until the 8000-token cap (~105 s) and the call fails with OutputTooLongError. 1.1.0 with the same model and settings does not do this.
Why (from the source)
engine/structured_output.py,OpenAIStrictSchemaGenerator(line ~77) setsjson_schema["required"] = list(properties)— every field becomes required under strict mode, includingobservation_text.engine/consolidation/prompts.py(line ~121) tells the model: "observation_text: for repeats, changes or ends, the observation's text copied exactly. Leave it out for unrelated."
For an unrelated observation the grammar will not let the model close the object without observation_text, while the prompt tells it not to write one. A strong model writes "" and moves on; a small model stalls and pads with whitespace until the cap.
When it does finish, we also saw field names (observation_text, same_as_fact, kind, deletes, …) stored as observation text (22 rows in one hour), which we did not see in the week before upgrading. That part we cannot tie to the source with certainty.
Possible fixes
- Make the prompt match the schema: "for unrelated, set
observation_textto an empty string", or - make
observation_textnullable / optional in the strict schema for the unrelated case.
Possibly related: #6673 (strict-schema replies that never end), #6019.
- Dominant language
- Python
- Stars
- 45.2k
- Forks
- 5.9k
- Avg merge
- 5h 51m
- Merged PRs (30d)
- 358
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from vectorize-io/hindsight
-
core
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
vectorize-io/hindsight#6730 ·
Maintainers usually reply within 1 day
-
Hermes plugin: switch to stop mirroring built-in memory writes into the bank (mirror_builtin_memory)Openintegration:hermes
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
vectorize-io/hindsight#6725 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
vectorize-io/hindsight#6690 · 1 comment ·
Maintainers usually reply within 1 day
-
core
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
vectorize-io/hindsight#6683 ·
Maintainers usually reply within 1 day
-
core
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
vectorize-io/hindsight#6662 ·
Maintainers usually reply within 1 day
All issues in vectorize-io/hindsight
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 86/100
py-econometrics/pyfixest#2072 ·
Maintainers usually reply within 1 day
-
~planning docs
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
Kromatic-Innovation/athenaeum#3084 ·
Maintainers usually reply within 1 day
-
Bosch Room Thermostat II number entities offer -500 to 3000 °C setpoint limits and a -32768 to 32767 °C outdoor temperaturePossibly taken @zigpy-review-bot claimed this today. Openbot: code audit
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
zigpy/zha-device-handlers#5769 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 Half a day Newbie friendliness 80/100
R0Y00/Xbox2Windows#254 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Kludex/python-multipart#375 ·
Maintainers usually reply within 1 day