Support import of partial CARs without prefetching the full DAG

オープン
#2,380 コメント 6 件 リアクション 1 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
停滞
技術スタック
javascript, react
領域
frontend

調査の方向性

まず、Web UI の CAR ファイルのインポートフローと、issue に記載されている ipfs dag import の動作を確認し、次に、ガベージコレクションの前にインポートされたブロックがどのようにピン留めされるかを追跡します。完了の条件は、ユーザーが部分的な CAR をインポートし、完全な DAG を事前取得せずに、それに含まれるリンクを GC 後も保持でき、さらに広範なコンテンツを保持するかどうかを選択できることです。

索引モデルが issue の本文から書いたものです。

説明

effort/days exp/intermediate help wanted P2 status/blocked

Is your feature request related to a problem? Please describe.

H: As a user, I would expect that when I import a CAR file that it is retained, including all the links it contains, and that parts are not lost through GC.Many links in my NFTs are of this type and as the owner of these I would also like to save them and make them available.
I would be interested to know whether all links are retained when I import the CAR file into Storacha or Filbase.

Describe the solution you'd like

We should allow users the option of pinning all content automatically when importing a car, so NFT owners do not lose content when GC is ran, but they can also only import the content for the car they import.

Describe alternatives you've considered

  • Auto pinning all blocks/CIDs in a car recursively - bad because sometimes folks don't want to do this.. but then they could also remove it easily manually.. which is more likely: importing a car and deleting it later, or importing a car and always keeping all of it?
  • ???

Additional context

From user convo on slack

C: I have downloaded a partial car file of collection_cid/image123.pngWhen I import that car file into kubo with ipfs dag import collection_cid_image123.png.car
Then done ipfs files cp /ipfs/collection_cid/image123.png
How to ensure the parent cid (collection cid) is not garbage collected?

C: I have tested
ipfs pin add <collection_cid>/image.png
ipfs pin add -r=false <collection_cid>
Now I can fetch the ipfs link <collection_cid>/image.png
But when I run garbage collection I cant fetch the link <collection_cid>/image.png anymore. I can only fetch the cid of the image.png
How to prevent, that the parent CID <collection_cid> of the link is not garbage collected?

C: I am offline and I dont want to do ipfs pin add <collection_cid> online, because it then pins the whole collection directory with all files inside which is super big in size.

C: When offline. I import the car and pin the root and parent cid. And the I can fetch the link /image123.png But after running garbage collection the link is gone.

H: ipfs pin add --recursive=false <collection_cid> that will pin only the root cid without pinning the full collection, so that it can resolve <collection_cid>/image.png just fine later. This is all assuming the collection is not huge and the root cid is not a "sharded" repository spanning several blocks.

C: Many thanks Hector. That helps me a lot. 🙏 Yes, that's right, when I direct pin the collection_cid with
ipfs pin add --recursive=false collection_cid
my link <collection_cid>/image123.png is resolved without any problems, even when I run the garbage collection.
But for some very large collections_cids, that's not enough, because of the issue you alluded to in your answer.How should I proceed then? :face_with_monocle:
I don't want the links, which are actually in the car file, to be lost due to garbage collection.
Example:
https://ipfs.io/ipfs/bafybeih4aaint73mwasanappkamkimwjaoe2m7pm6cagfoamjutxzbt76a/7345.png?format=car

C: Now I understand why the links that are present in the CAR files no longer work after I ran the garbage collection. I would not have expected that. 😢 Why is there no option for ipfs dag import to pin the root cid (recursive pin) and also all parent cids (direct pin).
Even the import of car files in ipfs-desktop via the webgui does not do this. Then the car import is useless for me if all links that are present in the car file are lost after running garbage collection. 😢

H: so... if the root directory is sharded, you would need to pin at least one level of subblocks... you can ipfs refs -r --max-depth 1 and then pin each cid individually. That should be enough for moderately big directories already.

C: The complicated process should take place automatically when importing the car file with ipfs dag import carfile.car or in the webui Import CAR file.

cc @lidel @agmap

主要言語
JavaScript
スター
1.6k
フォーク
539
PR マージ指標
30日以内にマージされた PR はありません

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

ipfs/ipfs-webui のほかの issue

ipfs/ipfs-webui の issue をすべて見る

似ている issue

JavaScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。