[Feat] Account for node traffic and stop background transfers at a monthly limit

未关闭
#29 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
30/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
活跃
技术栈
nodejs, typescript

调研方向

从 #22 和 #26 的放置与修复工作开始,包括 replication.repairSchedule,然后跟踪 #23 中的存储指标和有界健康状态。检查 /api/node/info 以了解兼容性约束,并验证警告、月度滚动、retry backoff、已停止的后台传输、持续读取以及健康状态降级时的行为。所有检查清单和验证用例均通过,且不改变 legacy 响应字段,即表示完成。

由索引模型根据 Issue 内容生成。

描述

enhancement Infrastructure IPFS

Summary

Account for the network traffic a node generates and receives, warn at one monthly limit, and stop generating traffic at another, so a node cannot quietly run up a hosting bill.

Details

A node moves bytes on its own initiative, without a user asking for anything. Copies are pushed to peers on upload, the repair job pushes copies again for anything it considers under-replicated, and reads pull blocks from other nodes. All of that is billable on most hosting, and none of it is currently measured or capped.

The repair job is the clearest risk. It runs on replication.repairSchedule, and for every confirmed file it believes is under-replicated it calls the placement path again, which transfers the whole file to any designated peer that does not have it. There is no backoff and no memory of previous failures, so a peer that is full, unreachable, or refusing will be re-sent every file on every pass, indefinitely. With a few thousand files that is a large, repeating transfer that nobody asked for.

Retrieval has a smaller version of the same shape: a node that does not hold a popular file fetches it once and caches it, but the cache is reclaimed when space is short, after which the next read fetches it again. Under sustained pressure that becomes a loop — fetch, cache, evict, fetch — and the same bytes cross the network repeatedly for one popular file. Collection currently has no notion of which blocks were expensive to obtain.

The other cases worth measuring before deciding what to bound:

  • A transfer that fails partway is retried from the start on the next pass. Blocks the peer already received stay in its blockstore unpinned, so a retry usually resumes rather than restarting, but a collection in between erases that progress.
  • A peer that answers that it has room and then fails the transfer anyway is asked again on every pass. The capacity question makes the common case cheap; it does not make the failing case bounded, which is what backoff is for.
  • An upload burst can name the same node as a holder for many files at once, with nothing spreading that load over time.
What to measure

Bytes are the billable unit, and they should be attributed to the reason they moved, because the answers differ:

  • copies placed on peers, and copies accepted from peers
  • blocks fetched to serve a read, and blocks served to other nodes
  • repair, separately from ordinary placement, because it is the one that repeats

Counters have to survive a restart, and they have to roll over on a calendar month to match how hosting is billed.

What to do at the limits

Two thresholds, both from the configuration:

  • a warning limit, which logs and shows up in the storage report
  • a stop limit, at which the node stops generating traffic of its own accord

Stopping must degrade in the right order. Serving a user who is asking for a file is the node's purpose and should be the last thing to go. Repair and copy placement are background work and should stop first. A node that has stopped replicating is still useful; a node that has stopped serving is not.

Backoff regardless of limits

Even under the limits, repeated failure should cost less each time. A peer that refuses or times out for a file should not be retried on the very next pass, and a file that has failed repeatedly should be retried rarely. This is worth doing whether or not the byte limits are configured, because it removes the unbounded case rather than capping it.

Checklist

  • Count bytes sent and received, attributed to placement, repair, retrieval, and serving
  • Persist the counters and roll them over monthly
  • Report them through the storage metrics
  • Add monthly warning and stop limits to the configuration, off by default
  • Warn once per period rather than on every event
  • At the stop limit, halt repair and copy placement while continuing to serve reads
  • Back off retries per peer and per file, independently of the limits
  • Avoid re-fetching the same popular file after every collection, or measure how often it happens before deciding it does not matter
  • Document what a node does when it reaches each limit, and how an operator resets or raises it

Notes

Comes out of the placement and repair work in #22 and #26. Related to #28: a larger and less curated node set makes background traffic harder to predict.

Verification

  • Run a node with a peer that always refuses copies and confirm repair backs off instead of re-sending every file on every pass
  • Confirm the counters survive a restart and reset at a month boundary
  • Cross the warning limit and confirm it is reported once, not repeatedly
  • Cross the stop limit and confirm placement and repair stop while reads are still served
  • Confirm a node under the stop limit still answers reads for content it does not hold, or document why it cannot

Coordination with health and compatibility

Traffic limiting changes the node's ability to fulfil its storage role and must feed the bounded health state from #23. Reaching the stop limit may leave existing reads available, but the node must become degraded and its checkpoint height must stop advancing once repair or placement freshness is no longer valid.

The public health response should expose only a bounded policy state such as normal, warning, or stopped. Exact byte counters and billing details belong in storage metrics or authenticated operations output. This addition must not change the legacy /api/node/info fields required by current PWA and iOS clients.

Additional checklist
  • Record traffic-policy state for the health checkpoint without exposing detailed billing data publicly
  • Freeze checkpoint advancement when a traffic stop makes replication freshness invalid
Additional verification
  • Cross the background-transfer stop limit and confirm reads may continue while health becomes degraded or stale and does not advance a false freshness height
主要语言
TypeScript
星标
309
派生
5
PR 合并指标
30 天内没有已合并 PR

环境准备

  • 提供 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 没有贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

Adamant-im/ipfs-node 的其他 Issue

查看 Adamant-im/ipfs-node 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。