Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Multiaddr Based Content Routing

Open
  • Is something wrong?
#11 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
aws
Domain
cloud, distributed-systems, networking

Research direction

No files or tests are named. Start by reviewing the Exchange (Bitswap) and Content Routing flow, including how PeerInfo and DHT provide records are handled. Done would require an agreed design and implementation for retrieving data from non-libp2p multiaddrs, with its verification and multi-block implications resolved.

Written by the indexing model from the issue text.

Description

Context

A standard IPFS data request causes the Exchange (i.e. Bitswap) to search the Content Routing system for a set of PeerInfo objects (which are just PeerIDs + their multiaddrs). The Exchange then takes these PeerInfo objects and requests data from the peers.

This causes us to need libp2p peers to proxy all data available to the Content Routing system, but if the data is available elsewhere shouldn't we be able to access it?

Proposal

I would like to be able to request data from multiaddrs that do not correspond to libp2p peers. For example, if we want to store data with some cloud storage provider like AWS S3 we could put a provide record in the DHT that Hash(Data) lives at /http/mybucket.s3.amazonaws.com/Data.

Motivation

While we could also run an a compute node, like EC2, with a set of IPFS cluster daemons on them with an S3-backed datastore it's certainly more costly. This is even more interesting if we can "draft" data that's publicly available over HTTP into IPFS.

Implications for future work

While the first iteration of this idea is conceptually fairly simple, it has implications for some of our ongoing endeavors. For instance, if we have 1000 small blocks hosted on /http/mysite.com/Data1-1000 that are all part of a single IPLD object we wouldn't be able to just provide the root IPLD node since there's no peer that will be able to tell us where the other 999 blocks are. There are various ways we could extend the protocol to allow us to tell retrievers where the other 999 blocks are, but it's not as simple as with the existing peer based retrieval.

Additionally, we would likely face increasing demand to support large files that are available over HTTP. Since we don't want users to download a lot of data before it's verified we'd probably want to extend the protocol with some ability for the peers advertising the content in the DHT to add (references to) hashes of chunks of the large file that could be verified. Similarly, we'd want to add the ability to download ranges of bytes when presented with a multiaddr that supports that functionality.

I think implementing this functionality could make running "pinning" services much easier and less expensive as well as greatly increasing the amount of content accessible via IPFS. But what about you @stebalien @raulk @bigs ?

Dominant language
No language data
Stars
38
Forks
2
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from libp2p/notes

All issues in libp2p/notes

Similar issues

More Cloud issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.