ipfs/kubo

Adding large blocks/directories (4MiB+) should cause error

Open

#5282 opened on Jul 25, 2018

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Go (13,906 stars) (2,725 forks)batch import
help wantedkind/bugstatus/deferred

Description

As mentioned in https://github.com/ipfs/notes/issues/296#issuecomment-407605712

I guess there is some hard limit on "do not accepts blocks over this size"

The generated directories are large and don't work, example: QmY6jEuqY3U5N9yxeBs6uoM63QnV2gjewXop9U4cQKauX8 has a distfiles directory with hash QmWvMmUjrH6neokv6RTqdgaYZSgxzr6hrjqaHosnDh6o7C which is 4821289 bytes in one block, I have republished that raw block as QmNzePsWN7kCF9fvbFBAnoVERWbCiVFjVEmr5EFqXFLtxL, Another example is QmfSAWwPTx5Kmre4bDF3zbkDuYXeNo9X7eTLvRUxyfr52B which can be found in QmPctnSoGCY8j19Lhxez3wqH2Ld55DzSqjnrbvNUuxoYgw. Using Sharding as described in #212 might solve this, but still the client should give error (or at least warning) when this happens. I guess that is the size here that is the issue? But should go-ipfs allow to create so large blocks? (It would be better for it to throw an error until there is a fix) I get a feeling there already must be an issue created about this?, but can't find a good match (that is still open)

What I recommend is that the client errors out if a block larger then that max limit is added. For those cases where this needs to be supported an extra flag can be added --allow-oversized-blocks

If an error can't be added then a Warning could be enough.

This message could also point to an issue disscussing this, or directions recommendingt to enable sharding.

Contributor guide

Tech stack
go
Domain
backend
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Go programmingIPFS block size limits
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
45
Research direction
Investigate the codebase to find where blocks are added (likely in core/commands or unixfs). Implement a check for block size greater than 4MiB, returning an error unless a new flag ` allow oversized blocks` is set. Consider adding a warning if an error cannot be added. The issue references #212 about sharding. Look for relevant tests and ensure the change aligns with existing behavior.