Need for IPIP: Data Onboarding via HTTP POST (and future ipfs:// POST|PUT)
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 35/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 冷清
调研方向
从 Kubo Gateway.Writable configuration documentation、WIP IPIP draft 以及 specs pull request 224 中的想法开始。定义用于 delegated files/directories 和 native blocks/CARs 的与供应商无关的 HTTP POST onboarding 协议,同时解决 TAR、form-data 和中断上传相关的问题;完成的标准是形成一份可用的 IPIP 规范和参考实现范围。
由索引模型根据 Issue 内容生成。
描述
Problem statement
HTTP Gateways are the most successful way for retrieving content-addressed data. Successful use of HTTP for retrieval use cases proves that IPFS does not replace HTTP, but augment it by providing variability and resiliency. IPFS over HTTP brings more value than the sum of its parts.
Removing the need for implementation specific RPC APIs (like one in Kubo) allowed not only faster adoption of CIDs on the web, but enabled alternative implementations of IPFS (like Iroh in Rust) to test compliance and benchmark thenselves against each other.
While we have HTTP Gateways as a standard HTTP-based answer to the retrieval of data stored with IPFS (including verifiable application/vnd.ipld.raw and application/vnd.ipld.car responses), the data onboarding over HTTP is currently done with vendor-specific APIs.
The status quo at 2023 Q1 is pretty bad from the end user/developer’s perspective: every IPFS implementation, including online services providing storage and pinning services, exposes custom opinionated HTTP API for onboarding data to IPFS.
Why we need IPIP for HTTP Data Onboarding
To illustrate, some prominent examples (2022 Q4):
Click to expand :see_no_evil:
- Implementations
- Kubo RPC (AKA legacy /api/v0/..)
- Is often used as a “standard HTTP API upload template” because it has commands for all onboarding needs:
- https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-add – files and directories
- FLAG: it uses custom form-data handling that requires special library for directory upload, which is an awful papercut for someone expecting simple upload with “curl” (http://web.archive.org/web/20221201011916/https://docs.ipfs.tech/reference/kubo/rpc/#request-body)
- FLAG: Kubo RPC was never designed to be used in browser context, and there are known bugs around the way it handles uploads (example: https://github.com/ipfs/kubo/issues/5168)
- https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-block-put – raw block
- https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-dag-put – JSON-like documents and custom DAGs (DAG-JSON and DAG-CBOR)
- https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-dag-import – arbitrary bags of blocks in CAR format
- https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-add – files and directories
- Is often used as a “standard HTTP API upload template” because it has commands for all onboarding needs:
- JS-IPFS
- Reimplements most of the Kubo RPC and exposes it over HTTP,, but diverged long time ago and is not 1:1
- FLAG: In addition to HTTP, JS-IPFS exposes selected commands over gRPC-over-WebSockets, to work-around browser issues caused by Kubo RPC (https://web.archive.org/web/20220528152743/https://github.com/ipfs/js-ipfs/tree/master/packages/ipfs-grpc-server#why)
- IPFS Cluster
- Acts as a reverse proxy for Kubo RPC, but has own commands too and provides special behavior on top of what Kubo RPC does:
- https://web.archive.org/web/20220911053755/https://ipfscluster.io/documentation/reference/api/ –
/addendpoint uses unixfs by default, but also accepts CARs when HTTP POST request is made with?format=carand it only accepts CARs with single root.
- https://web.archive.org/web/20220911053755/https://ipfscluster.io/documentation/reference/api/ –
- Acts as a reverse proxy for Kubo RPC, but has own commands too and provides special behavior on top of what Kubo RPC does:
- Kubo RPC (AKA legacy /api/v0/..)
- Online services
- Pinata
- https://web.archive.org/web/20220930091452/https://docs.pinata.cloud/pinata-api/pinning/pin-file-or-directory – onboarding file or directory
- https://web.archive.org/web/20220817122725/https://docs.pinata.cloud/pinata-api/pinning/pin-json – onboarding JSON document
- web3storage
- http://web.archive.org/web/20220914153854/https://web3.storage/docs/reference/http-api/ – file and CAR uploads
- note: no block API (impossible to import DAG-CBOR without the overhead of single-block-CAR for every CID)
- Infura
- http://web.archive.org/web/20220429202905/https://docs.infura.io/infura/networks/ipfs/http-api-methods/add – file and directory import API that is carbon-copy of Kubo’s internal RPC API
- http://web.archive.org/web/20220429203039/https://docs.infura.io/infura/networks/ipfs/http-api-methods/block_put – raw block import that is carbon-copy of Kubo’s internal RPC API
- note: no CAR import
- TODO: source more examples
- Pinata
And the CAR upload API insanity corca 2024 Q1:
This state of things introduces an artificial barrier to adoption: the user needs to learn what APIs are available, and then “pick winners” – decide which implementations and services are the most future-proof. And even then, many choices are burdened by legacy of Kubo RPC and it’s degraded performance and DX/UX in web browsers.
Goal: create data onboarding protocol for both HTTP and native IPFS
The intention here is to create IPIP with a vendor-agnostic protocol for onboarding data that:
- is easy to use and implement in HTTP (
POST https://)- does not require any libraries or documentation,
- and is as easy to work with from JS with
fetchAPI as it is in the command-line withcurl
- follow the retrieval story, where
ipfs://behavior is analogous to subdomain gateways- 👉 what we want, is to have a protocol that can be represented as both
POST https://ANDPOST ipfs://APIs
- 👉 what we want, is to have a protocol that can be represented as both
IPIP scope
We want two IPIPs: one for onboarding data with HTTP POST, and one for authoring (modifying/pathing) it with HTTP PUT.
This allows us to ship most useful onboarding first, and then do authoring as an optional add-on, which services may support, but dont have to (if they are only onboarding to filecoin etc).
For now, focusing on the POST
POST Requests (Onboarding)
👉 This is the minimal scope we need to cover from the day one, ensuring every use case has a vendor-agnostic spec.
- Delegated
- Single File (UnixFS) or single (DAG-)CBOR/JSON document
- Arbitrary Directory tree (UnixFS)
- Option A: TAR stream
- open question: how does this handle interrupted upload? can server tell some data is missing?
- Option B: custom form-data? (think twice, we have lessons learned around RPC at
/api/v0/addin Kubo)
- Option A: TAR stream
- Native
- Raw block
- CAR stream
The working code for this will be reference implementation that replaces/updates the legacy Gateway.Writable feature in Kubo with the above feature set.
PUT/PATCH/DELETE Requests (Authoring)
This will be a separate IPIP, but flagging this as long term plans that should feel idiomatic too.
- TBD: Delegated vs Native
- Critical: ensure no surprises, UX/DX is paramount. Needs research and analysis.
References
- Revisit the concept of Writable Gateways
- https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#gatewaywritable
- https://discuss.ipfs.io/t/writeable-http-gateways/210
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Location#pointing_to_a_new_document_http_201_created
- WIP private IPIP draft: https://www.notion.so/protocollabs/wip-IPIP-Data-Onboarding-with-HTTP-POST-4c394b8ebb774f2d87d34466019257fc
- Alex prototyped some REST APIs in https://github.com/ipfs/specs/pull/224/files (while this was intending to be update to Kubo RPC, the document includes some ideas around patching files and directories)
- https://docs.api.video/vod/delegated-upload-tokens as prior art where opaque token can be used with standard tools like curl
- 主要语言
- HTML
- 星标
- 1.2k
- 派生
- 250
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
这个项目没有提供开发容器、Dockerfile 或贡献指南,环境需要你自己搭建:先看它的 README,通用步骤见我们的新手贡献指南。
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
ipfs/specs 的其他 Issue
-
need/triage
难度 2/5 1-3 小时 新手友好度 68/100
-
need/triage
难度 5/5 一周以上 新手友好度 30/100
-
dnslink:// URIs未关闭need/triage
难度 5/5 一周以上 新手友好度 30/100
-
难度 5/5 一周以上 新手友好度 35/100
-
need/triage
难度 3/5 1-2 天 新手友好度 55/100
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 68/100
ElectronicWave/Elemental#265 ·
维护者通常 1 天内回复
-
feature needs-triage
难度 2/5 1-3 小时 新手友好度 78/100
BoardGamesEmpire/board-games-empire-backend#1306 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 80/100
-
难度 1/5 1 小时以内 新手友好度 90/100
MattiaPARRINELLO/backstage#80 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
harvester/harvester#12311 · 1 条评论 ·
维护者通常 1 天内回复