New Datastore Interfaces

Open
#64 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
go
Domain
database

Research direction

No files or tests are named. Start by reviewing the existing go-datastore interfaces, then compare the proposed StreamingDatastore, ReadClosurer, and OutParamer designs and the earlier discussion in #28; done requires a decided interface design and agreed scope.

Written by the indexing model from the issue text.

Description

The interfaces in go-datastore have been needing updating for a while. This came up in #28 so i decided to start the conversation up. Lets collect all these ideas together here so we can move forward with this.

Interface additions i've been looking for involve streaming interfaces:

type StreamingDatastore interface {
    // GetR returns a reader to read the value directly from the source (sans buffering)
    GetR(k ds.Key) (io.Reader, error)

    // PutW returns a new put writer that can be used to stream a value 
    // into the datastore
    PutW() (WritePutter, error)
}

// WritePutter is used to stream values into the datastore. Since the caller may not 
// know the key they want to store this value as until the write is complete (content 
// addressing) the key is set after the write is complete
type WritePutter interface{
    io.Writer
    FinishWithKey(k ds.Key) error
}

Also, for queries and maybe 'gets' in general, it would be nice to have a readonly view interface. Many storage backends (bolt in particular) allow you to read and operate on the data within the datastore without making a copy (i.e. using the datastores memory) within a closure. This would be really useful for things like 'load this protobuf object from disk' where we don't need our own copy of the bytes.

// A datastore implementing ReadClosurer allows passing a closure to operate
// on a readonly view of the data in question
type ReadClosurer interface {
    GetFunc(k ds.Key, func(i interface{}) error) error
}

Alternatively, this could be done with an output parameter and interfaces:

type OutParamer interface{
    GetOut(k ds.Key, out DSUnmarshaler) error
}

type DSUnmarshaler interface {
  DSUnmarshal(val []byte) error
}

This would behave similarly to the json.Unmarshaler interfaces.

Dominant language
Go
Stars
248
Forks
68
PR merge metrics
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ipfs/go-datastore

All issues in ipfs/go-datastore

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.