Zannick/logic-graph

Replace rocksdb with relational database (eg sql) for queue and state db

Open

#222 opened on Feb 18, 2025

View on GitHub
 (14 comments) (0 reactions) (0 assignees)Rust (0 forks)auto 404
enhancementhelp wanted

Repository metrics

Stars
 (4 stars)
PR merge metrics
 (PR metrics pending)

Description

Having a relational database with indexes might be better for #141. The new structure I'm envisioning would keep the statedb and use that as the queue by way of indexes:

Mode state progress score elapsed remaining hist prev next in-memory-queue won
lookup = x
insert ... ... ... ... ... ... ... ... ... ...
update = x set min set set set set
process = x set set false
evict = x set true
retrieve = x sort sort sort = false, set true
recover set false if true
solutions sort = true
  • Our statedb lookups are for data at the primary key (state).
  • As we process states, we insert the new states (or update the ones we've found faster routes to), and update the processed states with the next steps.
  • Evicting from the in-memory-queue simply flips the bit.
  • Retrieving states looks for states with the in-memory-queue bit set to false (and then sets them to true)
  • On recovery, we can either retrieve all the elements marked as being in the memory queue, or set them all back to false and perform a retrieve.
  • And we can retrieve solutions directly on recovery as well.

Contributor guide

Research direction
Study the current RocksDB usage in the codebase, understand the queue and state DB operations described, and explore how to replace them with a relational database like SQLite, focusing on the indexing and query patterns in the table.
Tech stack
rust
Domain
backenddatabase
Issue type
Refactor
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
RustSQL
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30