Consolidation: strict schema requires `observation_text`, but the prompt says to leave it out for unrelated observations — small models run to the token cap

Open Beginner friendly
#6,752 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
70/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
ai

Research direction

Read hindsight-api-slim/hindsight_api/engine/structured_output.py around line 77 (OpenAIStrictSchemaGenerator, where every property is forced into required) alongside engine/consolidation/prompts.py line ~121, which tells the model to omit observation_text for unrelated observations. Reproduce with HINDSIGHT_API_LLM_STRICT_SCHEMA_CONSOLIDATION=true and a small model, and unit-test the generated schema's required list. Done when prompt and schema agree on the unrelated case and generation terminates. Note #6673 and the unconfirmed field-name-as-text symptom.

Written by the indexing model from the issue text.

Description

core

Seen on self-hosted Hindsight 1.2.1 (rolled back to 1.1.0 since), checked again against v1.2.3 and main (80bd005fc). Paths are under hindsight-api-slim/hindsight_api/.

Setup: OpenAI-compatible provider serving a small Japanese model (sarashina3-mini), HINDSIGHT_API_LLM_STRICT_SCHEMA_CONSOLIDATION=true.

What happens

Since #6164 (1.2.0, one answer per fact), consolidation batches sometimes never finish: the model emits whitespace until the 8000-token cap (~105 s) and the call fails with OutputTooLongError. 1.1.0 with the same model and settings does not do this.

Why (from the source)

  • engine/structured_output.py, OpenAIStrictSchemaGenerator (line ~77) sets json_schema["required"] = list(properties) — every field becomes required under strict mode, including observation_text.
  • engine/consolidation/prompts.py (line ~121) tells the model: "observation_text: for repeats, changes or ends, the observation's text copied exactly. Leave it out for unrelated."

For an unrelated observation the grammar will not let the model close the object without observation_text, while the prompt tells it not to write one. A strong model writes "" and moves on; a small model stalls and pads with whitespace until the cap.

When it does finish, we also saw field names (observation_text, same_as_fact, kind, deletes, …) stored as observation text (22 rows in one hour), which we did not see in the week before upgrading. That part we cannot tie to the source with certainty.

Possible fixes

  • Make the prompt match the schema: "for unrelated, set observation_text to an empty string", or
  • make observation_text nullable / optional in the strict schema for the unrelated case.

Possibly related: #6673 (strict-schema replies that never end), #6019.

Dominant language
Python
Stars
45.2k
Forks
5.9k
Avg merge
5h 51m
Merged PRs (30d)
358

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from vectorize-io/hindsight

All issues in vectorize-io/hindsight

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.