Engine abstraction tracking issue

This issue is to organize and track progress on abstracting TiKV over generic storage engines (previous issue: https://github.com/tikv/tikv/issues/4184). There are many tasks to be done and contributions are welcome.

In scope: the process of abstracting the entire RocksDB engine API into the engine_traits and engine_rocks crates, decoupling TiKV and its test suite from RocksDB., and prototyping new engine bindings. Out of scope: refactoring engine_traits to better support with non-RocksDB engines.

The engine_traits crate docs describe the design and migration process.

The issue is updated to contain a checklist of tasks that definitely need to happen, along with name of who is working on it. When you decide to take a task, say so on the issue before attempting it so that someone else doesn't duplicate your work.

It's hard to know in what order these need to happen, nor exactly how to complete each of them, as each tends to present new challenges. Instead I have documented here, in the engine_traits crate docs, and in a status report, my experiences so far and suggestions for how to proceed.

The most important thing to keep in mind is:

Make small changes: small commits, small PRs. When you find that something else needs to change first, stop what you are doing, and make that change, and submit that PR. Don't let your patches grow out of control. Don't be afraid to throw away your work and try something else.

Task list (updated 2021/01/10)

• launch tikv-server with PanicEngine
  • https://github.com/tikv/tikv/issues/10528
• eliminate direct use of engine_rocks in raftstore (brson)
  • blocked on https://github.com/tikv/tikv/pull/9243
• Extract storage::kv to a crate
  • https://github.com/tikv/tikv/issues/9508
• Remove table properties abstractions
  • https://github.com/tikv/tikv/issues/10068
• Remove engine_rocks from tikv_kv
  • https://github.com/tikv/tikv/issues/9609
• Refactor storage::kv
  • https://github.com/tikv/tikv/issues/9480
• eliminate direct use of engine_rocks in tikv crate
  • lots of small abstraction leaks
  • (blocked) https://github.com/tikv/tikv/pull/8980
  • (related) https://github.com/tikv/tikv/issues/9480
• figure out how to construct all engines and configure tikv to use them
  • https://github.com/tikv/tikv/issues/9242
  • need to solve previous problems first
  • this will probably involve an engines_ctor crate that knows about every engine, based on code already in the engine_test crate that constructs multiple engines
  • to start with, non-rocks engines will probably need to use the current rocks-oriented configuration options on a best-effort basis, just to get tikv running
• modify test cases to use engine_test instead of engine_rocks
• Write a engine_traits_tests test suite that is parameterized over implementations
• Create engine_traits bindings for agatedb
  • https://github.com/tikv/tikv/issues/9844
  • brson started
• Write tests for advanced features in engine_traits_tests
• Remove unnecessary dependencies from engine_traits
  • there are several here at the wrong abstraction level, e.g. protobufs
  • this is just cleanup, not necessary functionality
• Clean up WriteBatch / Mutable traits
  • https://github.com/tikv/tikv/issues/9481