Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

DynamicCollection: range filters and sorts on an indexed double field return wrong results (keys are little-endian BitConverter bytes)

Open
  • Is something wrong?
#254 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
csharp
Domain
databases

Research direction

Read src/BLite.Core/DynamicCollection.cs at BsonValueToIndexKey, CreateIndexKeyFromObject, and PersistIndexMetadata, then compare with the typed IndexKey(double) path and the migrations mentioned in the issue. Reproduce the indexed range and sort failures with the supplied snippet. Done means numeric ranges and ordering match the unindexed results, and existing dynamic B-tree indexes are rebuilt safely after the key encoding changes.

Written by the indexing model from the issue text.

Description

area/indexing bug package/core severity/high

Found while fixing #249 (see #252), and checked on its own. On a DynamicCollection with a B-tree index on a double field, BLQL range filters and sorts on that field return wrong results: documents are missing, or come back in the wrong order. The unindexed collection returns the right answer.

Repro

using var engine = new BLiteEngine(path);
var col = engine.GetOrCreateCollection("zz");
await col.CreateIndexAsync("ratio");
foreach (var (n, v) in new[] { ("a", 0.1), ("b", 2.2), ("c", 1.3), ("d", 5.7), ("e", -3.3), ("f", 1.0000001), ("nz", -0.0), ("pz", 0.0) })
    await col.InsertAsync(col.CreateDocument(["name", "ratio"], b => b.AddString("name", n).AddDouble("ratio", v)));

col.Query("{\"ratio\": {\"$lt\": 2.0}}").ToList();
col.Query("{\"ratio\": {\"$gte\": 1.0, \"$lt\": 3.0}}").ToList();
col.Query().OrderBy("ratio").ToList();
Query Without the index (correct) With the index
{ratio: {$lt: 2.0}} a, c, e, f, nz, pz pz
{ratio: {$gte: 1.0, $lt: 3.0}} b, c, f (empty)
{ratio: 0.0} nz, pz pz
OrderBy("ratio") e, pz, nz, a, f, c, b, d pz, nz, e, b, a, f, d, c
OrderByDescending("ratio").Take(3) d, b, c c, d, f
QueryIndexAsync(idx, 1.0, 3.0) — (empty)

{ratio: {$gte: 1.0}} happens to come out right with these values: its upper bound is MaxKey, and every document it should return has a key above the lower bound.

Mechanism (verified on 0450b97)

  • DynamicCollection keys a double as new IndexKey(BitConverter.GetBytes(d)). This happens for stored values in BsonValueToIndexKey (src/BLite.Core/DynamicCollection.cs:2725) and for query bounds in CreateIndexKeyFromObject (:2812). BitConverter.GetBytes is little-endian IEEE 754: the first byte compared is the lowest mantissa byte and the sign bit is compared last, so key order has no relation to numeric order, and -0.0 and +0.0 get different keys. The typed path uses the order-preserving IndexKey(double) (IndexKey.cs:94-105).
  • Equality on a non-zero value still works, because both sides use the same bytes. Ranges and ordering do not.
  • BLQL takes the index range as its only candidate source. BlqlQuery.GetScanSource (src/BLite.Core/Query/Blql/BlqlQuery.cs:302) returns QueryIndexAsync(index, min, max) whenever the filter field is indexed, and the residual filter can only drop documents, never add the ones the scan missed.
  • The single-key sort fast path (BlqlQuery.cs:233-260) iterates the same index in key order, so the result comes out in byte order.
  • Dynamic B-tree indexes do not persist a KeyFormat: PersistIndexMetadata sets it for full-text indexes only (DynamicCollection.cs:2209). Every existing dynamic B-tree index therefore reads back as IndexMetadata.LegacyKeyFormat.

Affected versions: the encoding dates from 39e6c89 (2026-02-21, BLiteEngine, first in v1.4.0). BLQL has used the index for range scans and sorts since 6364585 (2026-03-25, first in v3.8.1-preview.2), which is when this became visible through BLQL. Before that, only QueryIndexAsync with bounds was affected. Reproduced on v5.2.3 / 0450b97.

Fix considerations

  • Key doubles with new IndexKey(d) in both places. That is order-preserving, and with #252 it gives -0.0 and +0.0 one key.
  • Stored keys change, so existing dynamic indexes holding doubles must be rebuilt. An old key cannot be told apart by looking at it: it is a 9-byte 0x02… key, like an Int64. Two clean triggers:
    • Since dynamic B-tree indexes read back as LegacyKeyFormat, write CurrentKeyFormat for them from now on and rebuild every dynamic B-tree index that reads back as legacy on open. This is the same pattern as the typed decimal migration (MigrateLegacyDecimalIndexes) and the dynamic full-text rebuild. It is a one-time rebuild of every dynamic B-tree index, including those without doubles.
    • Alternatively, fold it into the v6 KeyFormat bump planned for #212.
  • DateTime is unaffected: it is keyed through IndexKey(long).

Happy to send the fix if you want it, with the first option unless you prefer to wait for the v6 bump.

Dominant language
C#
Stars
66
Forks
6
Avg merge
1d 2h
Merged PRs (30d)
58

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from EntglDb/BLite

All issues in EntglDb/BLite

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.