Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

JSON FTS query fails with "Invalid triple format" when the searched value contains ';' (or the key contains ',')

Open
  • Is something wrong?
#9,675 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
python, rust
Domain
databases, search

Research direction

Start with rust/lance-index/src/scalar/inverted/tokenizer/document_tokenizer.rs around flatten_triplet at lines 180-181, then run the repros in $S/hunt/datashapes/json1.py and json2.py. Review the public MatchQuery and create_scalar_index documentation before resolving the query-syntax choice. Done means JSON triplet queries can represent values containing semicolons and keys containing commas without errors, with the grammar documented.

Written by the indexing model from the issue text.

Description

Bug bash: Needs team input — design question

Bug bash priority: Medium · critical-fix: no
From the FTS for Code bug bash (2026-10-01), findings: datashapes-2.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Decision needed: the query syntax (public API).

Options:

  1. Add escaping to the triple syntax.
  2. Add a structured (path, type, value) query object alongside the string form.

Recommendation: 2.

  • Category: bug
  • Environment: pylance local, Python 3.12, Linux arm64
  • Artifacts: pylance 13.0.0b18 (lance 13.0.0-beta.18); pylance 12.0.0 for comparison
  • Target tracker: GitHub lance-format/lance

Repro

Scripts: $S/hunt/datashapes/json1.py, json2.py. JSON column indexed with analyzer="code", split_identifiers=True, with_position=True.

ds.to_table(full_text_query=MatchQuery("code,str,return x;", "j"))
ds.to_table(full_text_query=MatchQuery("code,str,x;y", "j"))
ds.to_table(full_text_query=MatchQuery("a,b,str,getUserName", "j"))   # doc {"a,b": "getUserName"}

pylance 13.0.0b18:

match  'code,str,return x;'   -> ERROR ArrowInvalid: External error: Invalid user input: Invalid triple format: , /rustc/2d8144b7880597b6e6d3dfd63a9a9efae3f533d3/library/core/src/ops/function.rs:250:5
match  'code,str,x;y'         -> ERROR ArrowInvalid: External error: Invalid user input: Invalid triple format: y, /rustc/2d8144b7880597b6e6d3dfd63a9a9efae3f533d3/library/core/src/ops/function.rs:250:5
match  'a,b,str,getUserName'  -> ERROR ArrowInvalid: External error: Invalid user input: Invalid triple type: b, /rustc/.../function.rs:250:5

pylance 12.0.0 panicked (document_tokenizer.rs:147:71 ... Result::unwrap() on an Err value, then Lance reader producer terminated before end of stream), so lance#8813 holds: it is now a clean error, not a panic. But there is still no way to search a code value containing ; (common in code: return x;, for (i=0; i<n; i++)) or a key containing ,. The error location points at a rustc std path, not lance code.

Expected behavior

A user can pass an arbitrary code snippet as the value of a JSON triplet. With the code analyzer ; is never indexed anyway, so code,str,return x; should behave like code,str,return x.

Evidence: the text column accepts MatchQuery("return x;") without error.

Suspected root cause

flatten_triplet splits the whole query on ; before parsing path,type,value (lance rust/lance-index/src/scalar/inverted/tokenizer/document_tokenizer.rs:180), and splits each triple on the first two commas (:181). There is no escaping, so ; in a value starts a new (invalid) triple and , in a key shifts the type field. Not documented in the pylance create_scalar_index / MatchQuery docstrings.

Fix direction

Triage: either add an escape / structured (non-string) JSON query form, or only split on ; that is followed by a valid path,type, prefix; at minimum document the triplet grammar and its reserved characters.

Dominant language
Rust
Stars
7.2k
Forks
900
Avg merge
4d 22h
Merged PRs (30d)
366

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.