[Task] Research public IPFS interoperability and hybrid peering
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 35/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- nodejs, typescript
调研方向
从 issue 中链接的当前拓扑文档和所引用的 IPFS 概念开始。比较四种部署模型,在隔离环境中复现场景 A 和 B,并衡量对可靠性、资源使用和隐私的影响。当英文研究报告或 ADR 回答清单并记录 go/no-go 建议时,即视为完成。
由索引模型根据 Issue 内容生成。
描述
Summary
Research whether ADAMANT IPFS nodes should interoperate with IPFS Mainnet and third-party providers while preserving priority for ADAMANT peers, user privacy, bounded resource usage, and operational reliability.
This issue is for investigation and a go/no-go recommendation. It does not authorize production networking changes.
Details
Current state
The current implementation uses Helia, libp2p, UnixFS, and Bitswap, but deliberately forms a controlled mesh from configured ADAMANT peers. It does not enable the public Amino DHT, public bootstrap discovery, mDNS, relay, NAT traversal, or HTTP gateway routing.
Consequently, protocol compatibility does not currently make the node a discoverable provider on IPFS Mainnet:
- A public gateway such as
dweb.linknormally cannot discover an ADAMANT-hosted CID from public routing records - An ADAMANT node normally cannot discover an arbitrary third-party provider when its configured peers do not hold the requested CID
- A direct Bitswap exchange may still be technically possible when peers already know and can dial each other's multiaddresses, but that is not public content discovery
Public IPFS is not a database replicated to every node. Nodes store selected content, providers advertise availability through content-routing systems, and persistence still depends on at least one reachable provider retaining the content.
Research objective
Determine whether public interoperability provides enough practical reliability, availability, decentralization, or ecosystem value to justify the added privacy exposure, attack surface, resource cost, and operational complexity.
Evaluate at least these deployment models:
- Keep the current ADAMANT-only mesh
- Allow outbound retrieval from public providers without advertising ADAMANT content publicly
- Keep storage nodes controlled and introduce separately operated public bridge nodes
- Let every ADAMANT node participate directly in public content routing and block exchange
The research must not assume that the most open model is automatically the most decentralized or reliable. Public gateways and routing services may themselves become dependencies or choke points.
Expected interoperability scenarios
Scenario A: retrieve an ADAMANT upload through a public gateway
Example lifecycle:
- An ADAMANT client uploads an encrypted file and receives CID
A - An ADAMANT node pins the file according to its lifecycle policy
- An eligible public-facing node advertises provider records for CID
Athrough the selected public routing mechanism - A browser requests
https://A.ipfs.dweb.link/orhttps://dweb.link/ipfs/A - The gateway discovers a reachable provider and retrieves the blocks over a supported protocol
- The gateway returns the verified content over HTTP
Research and demonstrate what happens when the advertising node is unreachable, the provider record expires, the content is not pinned, the gateway cache is cold, the gateway applies rate limits or denylisting, or the CID is not publicly announced.
dweb.link is a best-effort public gateway, not an upload service, pinning provider, or availability guarantee. It must not become a critical production dependency.
Scenario B: retrieve third-party IPFS content through an ADAMANT node
Example lifecycle:
- The application requests CID
B, which was pinned by a third-party IPFS provider - The ADAMANT node asks preferred ADAMANT peers and starts public provider discovery in parallel or after a measured hedge delay
- Preferred peers receive the larger reliability timeout; a bounded number of untrusted secondary providers receive shorter per-peer timeouts
- The first complete valid DAG wins and all unnecessary work is cancelled
- Every received block is verified against its CID
- The response is returned to the application
- Third-party content remains an unpinned bounded cache entry unless an explicit policy says otherwise
Research whether Bitswap sessions, delegated routing, the Amino DHT, IPNI, a trustless HTTP gateway fallback, or a combination provides the safest and simplest implementation.
Network and routing questions
- Which public discovery mechanism is appropriate for provider lookup, provider advertisement, or both: Amino DHT, delegated routing, IPNI, explicit peers, or trustless gateways
- Whether public retrieval and public advertisement should be independently configurable
- Whether a small set of bridge nodes can isolate public-network risk from core storage nodes
- How provider records are published, refreshed, withdrawn, and monitored
- Which addresses and transports must be publicly reachable for gateways and third-party nodes
- Whether the current TCP, Noise, Yamux, and Bitswap configuration is sufficient for the selected public utilities
- How primary ADAMANT peers are preferred without preventing a faster verified secondary response
- How many secondary providers may be queried and when hedged requests begin
- How in-flight secondary work is cancelled after a primary or another secondary succeeds
- Whether unknown inbound Bitswap peers should be accepted, rejected, or restricted to public bridge nodes
- How current fixed-peer reconnection differs from public peer discovery and what "auto-peering" should mean in this project
Privacy assessment
Document the metadata exposed by every deployment model, including:
- Provider records linking CIDs to PeerIDs, public addresses, and availability
- CID requests or Bitswap wantlists revealing what content a node is seeking
- Timing and traffic-volume correlation between ADAMANT clients, storage nodes, bridge nodes, and public providers
- Public evidence that an ADAMANT node stores or requested a particular encrypted object
- Persistent peer identity and address exposure
- Information visible to DHT participants, delegated routers, gateways, relays, and direct peers
Payload encryption does not remove metadata exposure. Confirm whether attachment CIDs or related identifiers can be correlated with public ADAMANT data, and define what information must never appear in logs, metrics, routing records, or public APIs.
Compare the privacy impact of direct participation against a bridge design in which public peers see bridge identities rather than core storage-node identities.
Security and abuse assessment
- Unsolicited inbound block requests and bandwidth exhaustion
- Connection, stream, CPU, memory, file-descriptor, and disk-cache exhaustion
- Sybil peers, peer churn, slow providers, withheld blocks, malformed protocol messages, and timeout amplification
- Cache pollution with unique or oversized CIDs
- Abuse reports, denylisting, illegal content, and operator responsibilities when acting as a public provider or cache
- Expanded dependency and protocol attack surface from enabling DHT, NAT traversal, relay, additional transports, or gateway routing
- Separation between the authenticated ADAMANT replication protocol and untrusted public Bitswap traffic
- Safe failure behavior when public routing or gateway services are unavailable
Public interoperability must not let an unknown peer request durable pinning, participate in ADAMANT replication placement, call administrative APIs, or consume capacity reserved for primary traffic.
Resource policy
The proposed approximately 15% secondary allocation is not meaningful until its denominator and enforcement points are defined. Evaluate separate hard limits for:
- Connection and stream slots
- Concurrent inbound and outbound secondary transfers
- Aggregate and per-peer bandwidth
- In-flight bytes and memory
- CPU time and request duration
- File descriptors
- Disk used by unpinned third-party cache
- Cache lifetime and eviction priority
- Requests per PeerID, address, and aggregate public pool
Primary ADAMANT traffic must retain reserved capacity and must not be starved by public traffic. Determine whether 15% is appropriate for every resource, only selected resources, or should be replaced by explicit absolute and percentage limits.
Reliability and product value
Measure rather than assume:
- Retrieval success rate and latency with only ADAMANT peers
- Incremental success rate from public secondary providers
- Availability of an ADAMANT CID through at least two independent public gateway backends
- Provider discovery and first-byte latency for cold CIDs
- Public-routing failure and throttling behavior
- Bandwidth, CPU, memory, connection, and cache cost under normal and adversarial workloads
- Whether public retrieval materially improves attachment delivery in the actual ADAMANT workload
- Whether public advertisement provides meaningful redundancy when public nodes are not required to pin the content
Clarify whether the intended benefit is emergency retrieval, public sharing, additional cache sources, censorship resistance, independent verification, or marketing compatibility. Each goal may require a different topology.
Decision criteria
Produce a decision matrix for the four deployment models. Recommend implementation only if the selected model:
- Demonstrates measurable user or operational value
- Has an accepted privacy model and documented metadata exposure
- Preserves primary ADAMANT capacity and latency under the resource policy
- Has bounded failure behavior and can be disabled without affecting the controlled mesh
- Does not depend on one best-effort public gateway
- Has a staged rollout, monitoring, rollback, and abuse-response plan
- Keeps arbitrary third-party content unpinned by default
If the expected benefit is small or the privacy and operational costs cannot be bounded, explicitly recommend keeping the current controlled topology.
Checklist
- Document the current ADAMANT topology and the exact reasons public discovery does not occur
- Compare the four deployment models and relevant routing technologies
- Produce a privacy and metadata-flow assessment for each model
- Produce a security, abuse, and operator-liability assessment
- Define measurable primary and secondary resource budgets
- Design primary-first or hedged retrieval semantics with bounded cancellation and timeouts
- Determine the provider advertisement and re-providing requirements for public retrieval
- Reproduce Scenario A in an isolated non-production environment
- Reproduce Scenario B in an isolated non-production environment
- Benchmark success rate, latency, and resource use against the current controlled mesh
- Evaluate at least two independent gateway or retrieval backends rather than sibling hostnames on one backend
- Record a go/no-go recommendation and the preferred deployment model
- Create focused implementation issues only after the research decision is accepted
Notes
Related storage lifecycle and replication work: #22
Current topology documentation: https://github.com/Adamant-im/ipfs-node#network-topology
References:
- https://docs.ipfs.tech/concepts/how-ipfs-works/
- https://docs.ipfs.tech/concepts/lifecycle/
- https://docs.ipfs.tech/concepts/public-utilities/
- https://docs.ipfs.tech/concepts/privacy-and-encryption/
- https://docs.ipfs.tech/concepts/dht/
- https://docs.ipfs.tech/concepts/bitswap/
No production node should join public routing, advertise CIDs, accept unknown peers, or change its resource allocation as part of this research issue.
Verification
The issue is complete when an English research report or ADR:
- Answers every network, privacy, security, resource, reliability, and product-value question above
- Includes reproducible evidence for both end-to-end scenarios without using production content or user data
- States the limitations of public gateways, provider advertisement, caching, and pinning
- Quantifies the effect on primary ADAMANT traffic and verifies that secondary limits hold under load
- Recommends one deployment model or recommends no change, with explicit rationale
- Defines any required follow-up issues without implementing them in this task
Coordination with controlled-cluster health
Public interoperability must remain separate from the controlled-cluster health quorum defined in #23. Public providers, arbitrary Bitswap peers, delegated routers, gateways, and public bridge nodes must not satisfy configured-peer coverage or counted durability merely because they are reachable.
If a future accepted design promotes selected bridge nodes into counted storage membership, that must happen through the versioned membership policy from #28 and must remain distinguishable from general public reachability. A public DHT, delegated-routing, or gateway outage must not make an otherwise healthy controlled ADAMANT cluster stale unless the accepted product policy explicitly makes that service required.
Additional checklist
- Define whether each public or bridge role is uncounted, cache-only, or an accepted versioned member
- Keep public-routing availability independent from the controlled-cluster checkpoint unless a separately approved policy makes it required
Additional verification
- Disable every public routing and gateway dependency and confirm controlled-mesh health and replication remain truthful and available
- Connect arbitrary public peers and confirm they do not increase the checkpoint peer-coverage count
- 主要语言
- TypeScript
- 星标
- 309
- 派生
- 5
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 没有贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
Adamant-im/ipfs-node 的其他 Issue
-
APIs bug NodeJS
难度 3/5 1-2 天 新手友好度 78/100
Adamant-im/ipfs-node#49 · 1 条评论 ·
-
enhancement Infrastructure IPFS
难度 5/5 一周以上 新手友好度 38/100
Adamant-im/ipfs-node#30 ·
-
enhancement Infrastructure IPFS
难度 5/5 一周以上 新手友好度 30/100
Adamant-im/ipfs-node#29 ·
-
Composite task Decentralization Infrastructure IPFS
难度 5/5 一周以上 新手友好度 25/100
Adamant-im/ipfs-node#28 ·
-
Composite task Decentralization Encryption Governance IPFS
难度 5/5 一周以上 新手友好度 25/100
Adamant-im/ipfs-node#27 · 1 条评论 ·
查看 Adamant-im/ipfs-node 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 77/100
code-yeongyu/senpi#7139 ·
维护者通常 1 天内回复
-
community first-timers-only good first issue hacktoberfest help wanted low hanging fruit up-for-grabs
难度 1/5 1-3 小时 新手友好度 72/100
lingdojo/kana-dojo#34395 · 1 条评论 · 5 个 reaction ·
维护者通常 1 天内回复
-
bug status: needs triage
难度 2/5 1-3 小时 新手友好度 75/100
mastra-ai/mastra#31093 · 1 条评论 ·
维护者通常 1 天内回复
-
area:web p2 product:business triage:active type:bug
难度 1/5 1 小时以内 新手友好度 89/100
cuadrao-ai/argus#2328 ·
维护者通常 1 天内回复
-
bug frontend good first issue help wanted
难度 2/5 1-3 小时 新手友好度 88/100
nikhilb2/simple_invoicing#907 ·