A chat answer is never checked against the data
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 65/100
Research direction
Start from truth/tpch.mappings.json and the existing #501 scoring script it shares ids with; you will need to generate truth/tpch.questions.json keyed by the same 24 ids. Then trace query_data and the tool_exchange table to see where the SQL the model ran is recorded, and re-run it against the database, comparing the single value with the gold query's using #501's numbers-not-names comparison. Done means two reported columns, sql_right and answer_right, plus a script that can join a failed answer to whether its definition has a confirmed mapping. Read the 5-comment thread first — it likely settles where the runner lives and how the corpus is emitted.
Written by the indexing model from the issue text.
Description
#501 measures the proposals the explorer makes. Nothing measures the thing those proposals exist for: whether a person asking a question in chat gets the right number back.
The two are not the same measurement and one does not imply the other. A confirmed definition can be perfect and the answer still wrong — the model can pick the wrong source, join badly, filter on the wrong date column, or ignore the semantic layer and guess from the schema document. Conversely a base with no confirmed mappings at all still answers questions, by reading the schema document and writing its own SQL, and sometimes it is right. Today there is no number for either case.
What is checkable
query_data records very little: its step is {kind, label: source, detail: purpose} — the SQL it ran and the rows it got go only into tool_exchange, truncated, as the message replayed to the model. So the material for scoring a turn is:
- the assistant's final text,
- the tool calls in
tool_exchange, which carry the SQL the model actually ran.
That second one is the useful one. Re-run the SQL the model ran and compare its result with the gold query's — the same numbers-not-names judgement #501 uses, and it survives the truncation because the SQL is short even when its output was cut.
Then two columns rather than one:
sql_right— the SQL it ran computes what the question askedanswer_right— the number it printed is that number
They come apart. A model that runs the right query and then rounds, mislabels a unit, or narrates a different figure is wrong in a way a SQL-only score would call right, and that is exactly the failure a person reading the answer would suffer.
Corpus
Reuse truth/tpch.mappings.json. Each of its 24 definitions is already a question: "what is the total discounted revenue", "how many orders are there", "what is the stock value at cost". The gold SQL is already written. That gives 24 single-value questions for free, in truth/tpch.questions.json keyed by the same id, so a failed answer points at the definition it needed.
A second tier — TPC-H's own 22 queries, which group and order and return many rows — comes after. Their business questions are in the specification; scoring compares result sets, sorted. Single-value first because it is the case where "the answer is a number" is unambiguous, and because it is the tier that closes the loop below.
The loop this is for
The point is not one score. It is:
- run the questions, see which are wrong;
- for each wrong one, ask whether the definition it needed has a confirmed mapping — the script can answer this, since questions and mappings share an id;
- if not, that is a coverage gap: explore again, or write the mapping by hand;
- if it does have one and the answer is still wrong, the fault is in the prompt, the tool, or the definition itself;
- repeat.
"The mappings are complete" then has a definite meaning — every question's definition has a confirmed mapping — and "chat is accurate" has a number that moves. #503 raised proposal coverage from 25% to 33% on TPC-H; this issue is where that number is supposed to show up as answers that are right.
Not in scope
An LLM judging the answers. The gold SQL makes the judgement objective; a judge model would add a second thing that can be wrong, and it would be wrong in ways correlated with the model being judged.
- Dominant language
- Rust
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 21h 2m
- Merged PRs (30d)
- 263
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from deeplethe/utopia
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
bug
Difficulty 3/5 1-2 days Newbie friendliness 56/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 35/100
deeplethe/utopia#1341 · 1 comment ·
Maintainers usually reply within 1 day
-
bug
Difficulty 5/5 Over a week Newbie friendliness 48/100
deeplethe/utopia#1338 · 2 comments ·
Maintainers usually reply within 1 day
All issues in deeplethe/utopia
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
Maintainers usually reply within 1 day
-
o8 next debug build aborts on Glass to Dark appearance switch (nil identifier messaged before check)Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
hurttlocker/o8#5112 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
Maintainers usually reply within 1 day
-
memory_patch_page drops the `#` of headings like `## Notes on C#`Possibly taken @viniciusdsandrade claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
akitaonrails/ai-memory#1859 ·
Maintainers usually reply within 1 day
-
bug prio:p3 triaged ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day