Need for IPIP: Data Onboarding via HTTP POST (and future ipfs:// POST|PUT)

Aperta
#375 12 commenti 17 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla

Direzione di ricerca

Inizia con la Kubo Gateway.Writable configuration documentation, il WIP IPIP draft e le idee nel specs pull request 224. Definisci il protocollo di onboarding HTTP POST indipendente dal fornitore per delegated files/directories e native blocks/CARs, risolvendo anche le questioni relative a TAR, form-data e agli upload interrotti; il lavoro è completato quando sono disponibili una specifica IPIP utilizzabile e l’ambito di una implementazione di riferimento.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

IPIP need/community-input P2

Problem statement

HTTP Gateways are the most successful way for retrieving content-addressed data. Successful use of HTTP for retrieval use cases proves that IPFS does not replace HTTP, but augment it by providing variability and resiliency. IPFS over HTTP brings more value than the sum of its parts.

Removing the need for implementation specific RPC APIs (like one in Kubo) allowed not only faster adoption of CIDs on the web, but enabled alternative implementations of IPFS (like Iroh in Rust) to test compliance and benchmark thenselves against each other.

While we have HTTP Gateways as a standard HTTP-based answer to the retrieval of data stored with IPFS (including verifiable application/vnd.ipld.raw and application/vnd.ipld.car responses), the data onboarding over HTTP is currently done with vendor-specific APIs.

The status quo at 2023 Q1 is pretty bad from the end user/developer’s perspective: every IPFS implementation, including online services providing storage and pinning services, exposes custom opinionated HTTP API for onboarding data to IPFS.

Why we need IPIP for HTTP Data Onboarding

To illustrate, some prominent examples (2022 Q4):

Click to expand :see_no_evil:

And the CAR upload API insanity corca 2024 Q1:

This state of things introduces an artificial barrier to adoption: the user needs to learn what APIs are available, and then “pick winners” – decide which implementations and services are the most future-proof. And even then, many choices are burdened by legacy of Kubo RPC and it’s degraded performance and DX/UX in web browsers.

Goal: create data onboarding protocol for both HTTP and native IPFS

The intention here is to create IPIP with a vendor-agnostic protocol for onboarding data that:

  • is easy to use and implement in HTTP (POST https://)
    • does not require any libraries or documentation,
    • and is as easy to work with from JS with fetch API as it is in the command-line with curl
  • follow the retrieval story, where ipfs:// behavior is analogous to subdomain gateways
    • 👉 what we want, is to have a protocol that can be represented as both POST https:// AND POST ipfs:// APIs

IPIP scope

We want two IPIPs: one for onboarding data with HTTP POST, and one for authoring (modifying/pathing) it with HTTP PUT.
This allows us to ship most useful onboarding first, and then do authoring as an optional add-on, which services may support, but dont have to (if they are only onboarding to filecoin etc).

For now, focusing on the POST

POST Requests (Onboarding)

👉 This is the minimal scope we need to cover from the day one, ensuring every use case has a vendor-agnostic spec.

  • Delegated
    • Single File (UnixFS) or single (DAG-)CBOR/JSON document
    • Arbitrary Directory tree (UnixFS)
      • Option A: TAR stream
        • open question: how does this handle interrupted upload? can server tell some data is missing?
      • Option B: custom form-data? (think twice, we have lessons learned around RPC at /api/v0/add in Kubo)
  • Native
    • Raw block
    • CAR stream

The working code for this will be reference implementation that replaces/updates the legacy Gateway.Writable feature in Kubo with the above feature set.

PUT/PATCH/DELETE Requests (Authoring)

This will be a separate IPIP, but flagging this as long term plans that should feel idiomatic too.

  • TBD: Delegated vs Native
  • Critical: ensure no surprises, UX/DX is paramount. Needs research and analysis.
    • One idea is to keep it limited to patching UnixFS paths and DAG-JSON/CBOR documents.
    • Other idea is to have syntax parity with JSON-based IPLD Path and have the same JSON syntax as dag diff and dag patch commands.

References

Lingua principale
HTML
Stelle
1.2k
Fork
250
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ipfs/specs

Tutte le issue di ipfs/specs

Issue simili

Altre issue su Backend & API Design

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.