No way to query past Query.Prefix?

Open
#116 31 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
go
Domain
api, database

Research direction

Start with the Query API and the iterator path illustrated in go-ds-badger around line 270, then compare the datastore implementations that must honor Prefix. Review the existing comment history before deciding on the API behavior. Done means the seek and scanning semantics are agreed, implementations are updated consistently, and both bounded and unbounded iteration are covered.

Written by the indexing model from the issue text.

Description

Hi gents!

Currently, Query.Prefix is used to do the initial seek when a Query begins, and iteration only continues while that prefix holds. There needs to be a flag that basically says, hey, don't use Prefix once iteration has begun. Otherwise, there's no way to effectively do a seek and do an arbitrary amount of scanning. This seems like a critical feature to leave behind, yikes.

Hopefully, I'm missing something!?

For example, in go-ds-badger: (line 270+)

it := txn.NewIterator(opt)
it.Seek(prefix)
...
qrb.Process.Go(func(worker goprocess.Process) {
    ...
    for sent := 0; it.ValidForPrefix(prefix); sent++ {
        ...
    }
}
...

could be enhanced to:

it := txn.NewIterator(opt)
it.Seek(prefix)
if q.PrefixOnlySeeks {
    prefix = nil
}
...

Adding this would be easy, but all the impls would need to be updated of course. So maybe it's done as a selectively available feature?

This is a dealbreaker for us, but we love Datastore and want to use and support it! If some at PL agrees to accept our pull requests in a timely fashion, we'll help do the work. I'm sad we're now in this position, and would take less than a few mins for dbs like leveldb, badger, and bolt. What do y'all think?

We kindly need resolution on this relatively soon one way or the other so that we know we need to pass on Datastore.

Thanks,
Drew

Dominant language
Go
Stars
248
Forks
68
PR merge metrics
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ipfs/go-datastore

All issues in ipfs/go-datastore

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.