unixfs/mod: appending to a file whose single leaf holds inline data makes the original bytes unreadable

Open
#1,221 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go
Domain
data

Research direction

Start in ipld/unixfs/mod/dagmodifier.go, in appendData and its *mdag.RawNode branch, which already wraps the old node as a first leaf. Then read ipld/unixfs/io/dagreader.go, which skips data on internal nodes, and the trickle append path in ipld/unixfs/importer/trickle. Run the reproduction test in the issue, which should read "abcdef" when done, along with the existing tests in ./ipld/unixfs/mod/, ./mfs/ and the trickle package.

Written by the indexing model from the issue text.

Description

AI disclosure: this bug was found while analyzing a package that is not yet published on pub.dev, and the analysis, the reproduction and the proposed fix below were made entirely with AI (Claude). The author wants everyone who reads this issue (and any PR that follows) to know that, and acknowledges that, despite an extensive battery of tests, AI can still make mistakes.

Problem

Appending to the end of an MFS file whose root node is a UnixFS leaf that holds its bytes inline (the single dag-pb node of a small add, no links) makes the original bytes unreadable. Size() still counts them, but reading the file returns only the appended part.

Seen with boxo v0.43.0 (through kubo v0.43.1); the same code is on main (3d6ac39a0f76). Go 1.26.5, linux/amd64 and windows/amd64.

Reproduction

package repro

import (
	"bytes"
	"context"
	"io"
	"testing"

	chunker "github.com/ipfs/boxo/chunker"
	mdtest "github.com/ipfs/boxo/ipld/merkledag/test"
	ft "github.com/ipfs/boxo/ipld/unixfs"
	"github.com/ipfs/boxo/ipld/unixfs/importer"
	"github.com/ipfs/boxo/mfs"
)

func TestAppendInlineData(t *testing.T) {
	ctx := context.Background()
	ds := mdtest.Mock()
	root, err := mfs.NewRoot(ctx, ds, ft.EmptyDirNode(), nil, nil)
	if err != nil {
		t.Fatal(err)
	}
	// "abc" fits in one chunk: a single leaf with the data inline
	nd, err := importer.BuildDagFromReader(ds, chunker.DefaultSplitter(bytes.NewReader([]byte("abc"))))
	if err != nil {
		t.Fatal(err)
	}
	if err := mfs.PutNode(root, "/a", nd); err != nil {
		t.Fatal(err)
	}
	n, _ := mfs.Lookup(root, "/a")
	f := n.(*mfs.File)

	fd, _ := f.Open(ctx, mfs.Flags{Write: true, Sync: true})
	fd.Seek(0, io.SeekEnd)
	fd.Write([]byte("def"))
	fd.Close()

	fd, _ = f.Open(ctx, mfs.Flags{Read: true})
	defer fd.Close()
	b, _ := io.ReadAll(fd)
	if string(b) != "abcdef" {
		t.Fatalf("got %q, want %q", b, "abcdef") // got "def"
	}
}

Cause

DagModifier.appendData (ipld/unixfs/mod/dagmodifier.go) passes a *mdag.ProtoNode straight to trickle.Append. When that node is a leaf (no links, file bytes in its UnixFS Data), trickle.Append adds the new data as children and keeps the existing Data, so the node becomes an internal node that still carries file bytes. DagReader skips the data of internal nodes (ipld/unixfs/io/dagreader.go: "Skip internal nodes, they shouldn't have any file data"), so those bytes are never read, while Filesize still counts them.

The *mdag.RawNode branch of the same function already handles the equivalent case: it wraps the raw node as the first leaf of a new file node before calling trickle.Append.

Proposed fix

Do the same for a ProtoNode leaf with inline data: move the data into a first leaf (a raw leaf if RawLeaves is set, a dag-pb leaf otherwise), keep mode and mtime on the new root, then append as before. A sketch, which we have been running as a local patch:

	case *mdag.ProtoNode:
		nd, err := dm.inlineDataToLeaf(nd, dagserv)
		if err != nil {
			return nil, err
		}
		// ProtoNode can be directly passed to trickle.Append
		...

func (dm *DagModifier) inlineDataToLeaf(nd *mdag.ProtoNode, dagserv ipld.DAGService) (*mdag.ProtoNode, error) {
	if len(nd.Links()) > 0 {
		return nd, nil
	}
	fsn, err := ft.FSNodeFromBytes(nd.Data())
	if err != nil || len(fsn.Data()) == 0 {
		return nd, nil
	}
	data := fsn.Data()
	var leaf ipld.Node
	if dm.RawLeaves {
		prefix := dm.Prefix
		prefix.Version, prefix.Codec = 1, cid.Raw
		if leaf, err = mdag.NewRawNodeWPrefix(data, prefix); err != nil {
			return nil, err
		}
	} else {
		pn := mdag.NodeWithData(ft.FilePBData(data, uint64(len(data))))
		pn.SetCidBuilder(dm.Prefix)
		leaf = pn
	}
	if err := dagserv.Add(dm.ctx, leaf); err != nil {
		return nil, err
	}
	root := ft.NewFSNode(ft.TFile)
	root.SetMode(fsn.Mode())
	root.SetModTime(fsn.ModTime())
	root.AddBlockSize(uint64(len(data)))
	b, err := root.GetBytes()
	if err != nil {
		return nil, err
	}
	out := mdag.NodeWithData(b)
	out.SetCidBuilder(dm.Prefix)
	if err := out.AddNodeLink("", leaf); err != nil {
		return nil, err
	}
	return out, nil
}

With it, the reproduction reads "abcdef", and go test ./ipld/unixfs/mod/ ./mfs/ ./ipld/unixfs/importer/trickle/ still passes.

About CIDs: this changes only the DAG produced when appending to such a leaf, which today is a DAG whose first bytes cannot be read; the new layout is the one the RawNode branch already produces. Default add output is untouched.

If this approach is acceptable, we can open a PR with a regression test, a CHANGELOG.md entry and the companion kubo PR.

Dominant language
Go
Stars
326
Forks
165
PR merge metrics
No merged PRs in 30d

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ipfs/boxo

All issues in ipfs/boxo

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.