Closed DHT Based Routing
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Domain
- distributed-systems
Research direction
No files, tests, or entry points are named. Start by reviewing the proposal and related discussion to determine the membership, routing-table, consensus, and record-lifetime requirements. Done would require an agreed design and an identified implementation plan.
Written by the indexing model from the issue text.
Description
Libp2p currently relies on a fully p2p Kademlia DHT. Unfortunately, even if we get it to the point where it behaves optimally, it still won't scale enough to support our content routing needs.
Proposal: Implement a "closed" DHT with known members. Note: This is often called a "Distributed KV Store".
Unlike our Kademlia DHT:
- This system would be permissioned instead of permissionless. Joining as a DHT "server" would require either human or blockchain consensus.
- Have a fixed, well-known routing table, fetched when initially connecting to this DHT.
Motivations:
- 1-RTT lookup
- batch put
- long-lived routing records
Performing a 1-RTT lookup and/or a batch put requires a known routing table. The round-trips in traditional p2p DHTs all come from discovering this routing table along the way.
A known routing table requires some form of consensus on the members of this routing table. That's where the trust comes in.
The last part of this is long-lived routing records. The Internet Archive has ~400e9 files which equates to at least 35TiB of "provider records". However, the IA isn't adding 400e9 files per day. Given stable nodes that can be trusted to keep long-lived records alive, the IA wouldn't have to keep re-broadcasting old records to keep them alive.
Notes:
- Really, we may want to treat large services like the IA as "trackers". However, the current system won't even scale for smaller services.
- We may also want to integrate payment for large users to prevent abuse but we can probably punt on that for now.
CC @Kubuxu, IIRC you already proposed something kind of like this but I couldn't find the proposal.
CC @obo20 as this is really important for pinning services.
- Dominant language
- No language data
- Stars
- 38
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from libp2p/notes
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
DHT
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
Similar issues
-
kind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
ScalingModifiers formula fails with "formula returned non-float result" when expression evaluates to an integerPossibly taken @Sarthak-Pandey claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
kedacore/keda#8270 · 1 comment ·
Maintainers usually reply within 1 day
-
area:manager bug triage:confirmed
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Cotal-AI/Cotal#3404 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
Maintainers usually reply within 1 day