Naming blocks in the datastore

未关闭
#242 16 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
20/100
Issue 类型
功能
描述清晰度
需要澄清
活跃度
停滞

调研方向

先阅读链接的 go-ipfs 和 js-ipfs issue,然后比较它们当前针对 CIDv0 和 CIDv1 的 datastore 查找行为。审阅列出的 multihash、CIDv1、双重间接寻址和元数据提案。要视为完成,需要达成一致的 datastore 命名规范和迁移路径;这里未定义实现范围。

由索引模型根据 Issue 内容生成。

描述

P1

Context: https://github.com/ipfs/ipfs/issues/337

Currently, both js-ipfs and go-ipfs index blocks in the datastore by CID. Unfortunately, it's possible for a single block to have multiple CIDs due to (a) different CID versions and (b) due to different multicodecs (e.g., dag-cbor v. cbor v. raw).

The primary concern is (a). As we switch to returning CIDv1 (for multibase support), we still want to be able to lookup blocks that were fetched/added as CIDv0.

Currently, when looking up a CID, both go-ipfs and js-ipfs will first attempt to lookup the CID under the original CID version, then under the other CID version. However, this costs us two look-ups.


Primary use-case:

Ensure that CIDv1 and CIDv0 can be used interchangeably, especially in the gateway.

Proposals:

  1. Index blocks by multihash only.
  2. Index blocks by CIDv1.
  3. Double indirection. Map CIDs to a locally chosen hash function to blocks.
  4. Store extra metadata along with the block (known codecs, refcounts, etc.).
    • Unclear how to do this efficiently with a datastore (would likely require a bunch of separate writes/keys).

Desired properties:

  1. No duplicate blocks when...
    a. CID Versions differ: 1, 2 & 3
    b. CID Codecs differ: 1 & 3.
    c. Hash functions differ: 3.
  2. No discarded structural information (keep codecs): 2 & 3.
  3. Viable migration path: 1, 2 & 3.
  4. Fast migration path: 1 & 3.
  5. Zero Overhead:
    a. Time: 1, 2, & 4.
    b. Space: 1 & 2.

The current consensus is option 1 (multihash) (https://github.com/ipfs/go-ipfs/issues/6815, https://github.com/ipfs/js-ipfs/issues/2415). The proposal here is to consider option 2 (and at least look at the others).

However, option 1 doesn't give us property 2 as it discards the codec on write. @jbenet has objected strongly to this as we will be discarding structural information about the data. This information will still be stored in the pin set but we'll be losing this information for unpinned data.

We have also run into this same issue when trying to write a reverse datastore migration, migrating back from multihashes to CIDs: we need to somehow recover the codecs. The reverse migration in Option 2 would simply be: do nothing.

We need to consider the pros/cons of switching to option 1 before proceeding.


Side note: why should we even care about (b), multiple codecs for the same block?

IPLD objects as "files"

  • I might want to add one or more CBOR-IPLD objects to an IPFS directory for legacy applications, and address them in IPLD.
  • I might want to address a yaml/json object in an IPFS directory as an IPLD object.

Dumb block transport

I might want to take an unbalanced DAG (blockchain), a DAG with weird codecs (git, eth, etc.), etc. and sync it with another node. It would be nice if I could take this DAG, treat all the nodes as raw nodes, then build a well-balanced "overlay-dag" using simple, well-supported codecs.

This might, for example, be useful for storing DAGs with custom codecs in pinning services.

主要语言
HTML
星标
1.2k
派生
250
PR 合并指标
30 天内没有已合并 PR

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南,环境需要你自己搭建:先看它的 README,通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

ipfs/specs 的其他 Issue

查看 ipfs/specs 的全部 Issue

相似的 Issue

更多 Distributed Systems Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。