akka/akka-core

Allow DData entries to be lazily loaded

Open

#25 560 ouverte le 1 sept. 2018

Voir sur GitHub
 (3 commentaires) (0 réactions) (0 assignés)Scala (3 547 forks)batch import
1 - triagedhelp wantedt:distributed-data

Métriques du dépôt

Stars
 (13 277 stars)
Métriques de merge PR
 (Merge moyen 8j 19h) (10 PRs mergées en 30 j)

Description

Motivation

At the current moment, akka.distributed-data can be used to provide an eventually consistent, replicated in-memory store. Moreover with ability to set up durable keys, we can make its state persistent. This however comes with a limitation - all of the entries (no matter if they are only in-memory or durable) must fit into memory. This reduces potential area of possibile fields, on which they could be used.

Proposal

The idea here is to take advantage of durable keys feature in order to allow keys to be lazily loaded only on demand.

When a replicator detects that some keys is tried to be accessed, we first check in-memory index in order to serve response: just like it works right now. With lazy load, we however accept the possibility, that not all entries may be present in memory, as some of them may reside in durable store. In that case when key was not found, we try to fallback and load it from a durable store: if it was found there, we can bring it into active, in-memory set.

For each key we could also track the last time it was accessed, and schedule periodic job, which could either:

  1. Reap entries that have not been accessed past the given threshold.
  2. Prune N least used entries (which the oldest last used timestamp).

There are also several different options for checking if a given key was present (if this is needed), i.e.:

  1. First presented option was to make naive check for in-memory, active set first and make fallback to durable store.
  2. Another option is to load the entire index (keys only) at the LoadAllCompleted event, and use lazy load only for values. This of course means that the index itself must still fit into memory.
  3. Use probabilistic data structures (i.e. bloom filter) to quickly determine if entry doesn't exists. In this case I'm not sure if it's worth an effort and increasing the complexity.

Guide contributeur