unixfs/mod: appending to a file whose single leaf holds inline data makes the original bytes unreadable
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 55/100
Research direction
Start in ipld/unixfs/mod/dagmodifier.go, in appendData and its *mdag.RawNode branch, which already wraps the old node as a first leaf. Then read ipld/unixfs/io/dagreader.go, which skips data on internal nodes, and the trickle append path in ipld/unixfs/importer/trickle. Run the reproduction test in the issue, which should read "abcdef" when done, along with the existing tests in ./ipld/unixfs/mod/, ./mfs/ and the trickle package.
Written by the indexing model from the issue text.
Description
AI disclosure: this bug was found while analyzing a package that is not yet published on pub.dev, and the analysis, the reproduction and the proposed fix below were made entirely with AI (Claude). The author wants everyone who reads this issue (and any PR that follows) to know that, and acknowledges that, despite an extensive battery of tests, AI can still make mistakes.
Problem
Appending to the end of an MFS file whose root node is a UnixFS leaf that holds its bytes inline (the single dag-pb node of a small add, no links) makes the original bytes unreadable. Size() still counts them, but reading the file returns only the appended part.
Seen with boxo v0.43.0 (through kubo v0.43.1); the same code is on main (3d6ac39a0f76). Go 1.26.5, linux/amd64 and windows/amd64.
Reproduction
package repro
import (
"bytes"
"context"
"io"
"testing"
chunker "github.com/ipfs/boxo/chunker"
mdtest "github.com/ipfs/boxo/ipld/merkledag/test"
ft "github.com/ipfs/boxo/ipld/unixfs"
"github.com/ipfs/boxo/ipld/unixfs/importer"
"github.com/ipfs/boxo/mfs"
)
func TestAppendInlineData(t *testing.T) {
ctx := context.Background()
ds := mdtest.Mock()
root, err := mfs.NewRoot(ctx, ds, ft.EmptyDirNode(), nil, nil)
if err != nil {
t.Fatal(err)
}
// "abc" fits in one chunk: a single leaf with the data inline
nd, err := importer.BuildDagFromReader(ds, chunker.DefaultSplitter(bytes.NewReader([]byte("abc"))))
if err != nil {
t.Fatal(err)
}
if err := mfs.PutNode(root, "/a", nd); err != nil {
t.Fatal(err)
}
n, _ := mfs.Lookup(root, "/a")
f := n.(*mfs.File)
fd, _ := f.Open(ctx, mfs.Flags{Write: true, Sync: true})
fd.Seek(0, io.SeekEnd)
fd.Write([]byte("def"))
fd.Close()
fd, _ = f.Open(ctx, mfs.Flags{Read: true})
defer fd.Close()
b, _ := io.ReadAll(fd)
if string(b) != "abcdef" {
t.Fatalf("got %q, want %q", b, "abcdef") // got "def"
}
}
Cause
DagModifier.appendData (ipld/unixfs/mod/dagmodifier.go) passes a *mdag.ProtoNode straight to trickle.Append. When that node is a leaf (no links, file bytes in its UnixFS Data), trickle.Append adds the new data as children and keeps the existing Data, so the node becomes an internal node that still carries file bytes. DagReader skips the data of internal nodes (ipld/unixfs/io/dagreader.go: "Skip internal nodes, they shouldn't have any file data"), so those bytes are never read, while Filesize still counts them.
The *mdag.RawNode branch of the same function already handles the equivalent case: it wraps the raw node as the first leaf of a new file node before calling trickle.Append.
Proposed fix
Do the same for a ProtoNode leaf with inline data: move the data into a first leaf (a raw leaf if RawLeaves is set, a dag-pb leaf otherwise), keep mode and mtime on the new root, then append as before. A sketch, which we have been running as a local patch:
case *mdag.ProtoNode:
nd, err := dm.inlineDataToLeaf(nd, dagserv)
if err != nil {
return nil, err
}
// ProtoNode can be directly passed to trickle.Append
...
func (dm *DagModifier) inlineDataToLeaf(nd *mdag.ProtoNode, dagserv ipld.DAGService) (*mdag.ProtoNode, error) {
if len(nd.Links()) > 0 {
return nd, nil
}
fsn, err := ft.FSNodeFromBytes(nd.Data())
if err != nil || len(fsn.Data()) == 0 {
return nd, nil
}
data := fsn.Data()
var leaf ipld.Node
if dm.RawLeaves {
prefix := dm.Prefix
prefix.Version, prefix.Codec = 1, cid.Raw
if leaf, err = mdag.NewRawNodeWPrefix(data, prefix); err != nil {
return nil, err
}
} else {
pn := mdag.NodeWithData(ft.FilePBData(data, uint64(len(data))))
pn.SetCidBuilder(dm.Prefix)
leaf = pn
}
if err := dagserv.Add(dm.ctx, leaf); err != nil {
return nil, err
}
root := ft.NewFSNode(ft.TFile)
root.SetMode(fsn.Mode())
root.SetModTime(fsn.ModTime())
root.AddBlockSize(uint64(len(data)))
b, err := root.GetBytes()
if err != nil {
return nil, err
}
out := mdag.NodeWithData(b)
out.SetCidBuilder(dm.Prefix)
if err := out.AddNodeLink("", leaf); err != nil {
return nil, err
}
return out, nil
}
With it, the reproduction reads "abcdef", and go test ./ipld/unixfs/mod/ ./mfs/ ./ipld/unixfs/importer/trickle/ still passes.
About CIDs: this changes only the DAG produced when appending to such a leaf, which today is a DAG whose first bytes cannot be read; the new layout is the one the RawNode branch already produces. Default add output is untouched.
If this approach is acceptable, we can open a PR with a regression test, a CHANGELOG.md entry and the companion kubo PR.
- Dominant language
- Go
- Stars
- 326
- Forks
- 165
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ipfs/boxo
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 4/5 3-5 days Newbie friendliness 46/100
-
need/triage
Difficulty 5/5 Over a week Newbie friendliness 38/100
-
dif/medium effort/hours kind/enhancement P3 topic/devexp
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
dif/expert effort/weeks kind/enhancement P2
Difficulty 5/5 Over a week Newbie friendliness 25/100
Similar issues
-
affects-8.5 component/session severity/moderate sig/transaction type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
enhancement good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
harvester/harvester#12311 · 1 comment ·
Maintainers usually reply within 1 day
-
verifyrun --bracket-digest --credit: relative --witness is could-not-check, absolute path creditsOpenbug effort:s impact:low raised-by:desk risk:low
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
medici-finance/assay#6580 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
blinklabs-io/gouroboros#3026 ·
Maintainers usually reply within 1 day