Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

tarfile: extractfile(member).read() of a sparse member is slow due to repeated bytes concatenation

Open Beginner friendly
  • Is something wrong?
#160,774 2 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@devdavidejesus is already working on this.

Since Oct 29, 2026.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
75/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend

Research direction

Read _FileInFile.read() in Lib/tarfile.py and trace how it appends data segments and sparse holes. Check the existing tarfile tests for sparse members, then add coverage for extractfile().read() and chunked reads against the expected content. Done means both read patterns return the correct content and the full read avoids repeated bytes concatenation.

Written by the indexing model from the issue text.

Description

performance stdlib triaged type-bug

Bug report

_FileInFile.read() in Lib/tarfile.py builds its result with buf += b for each data segment and buf += NUL * length for each hole of a sparse member. Each bytes concatenation copies the whole accumulated buffer, so reading a sparse member with TarFile.extractfile(member).read() takes time proportional to the member size times the number of sparse segments.

Holes take no space in the archive, so a small archive is enough to make this noticeable. extract(), extractall() and reading the member in small chunks (for example with shutil.copyfileobj()) are not affected.

Reproducer
import io, tarfile, time

def build(n, blk=4096):
    # GNU sparse 0.1 (pax) member: n data blocks separated by holes.
    t = tarfile.TarInfo("GNUSparseFile.0/a.img")
    t.size = n * blk
    t.pax_headers = {
        "GNU.sparse.map": ",".join(f"{2 * i * blk},{blk}" for i in range(n)),
        "GNU.sparse.name": "a.img",
        "GNU.sparse.realsize": str(2 * n * blk),
    }
    bio = io.BytesIO()
    with tarfile.open(fileobj=bio, mode="w", format=tarfile.PAX_FORMAT) as tf:
        tf.addfile(t, io.BytesIO(b"D" * t.size))
    return bio.getvalue()

def read_time(n):
    with tarfile.open(fileobj=io.BytesIO(build(n))) as tf:
        f = tf.extractfile(tf.next())
        t0 = time.perf_counter()
        f.read()
        return time.perf_counter() - t0

for n in (1000, 2000, 4000):
    print(f"{n} segments: {read_time(n):.2f}s")

On a release build (macOS, Apple M3) the read of a sparse member with 4000 segments takes several seconds; the time roughly quadruples when the segment count doubles. With the change below each read is a fraction of a second.

Suggested fix

Accumulate the pieces in a list and join once, instead of repeated concatenation:

-        buf = b""
+        chunks = []
         while size > 0:
             ...
-                buf += b
+                chunks.append(b)
             else:
-                buf += NUL * length
+                chunks.append(NUL * length)
             size -= length
             self.position += length
-        return buf
+        return b"".join(chunks)

I have a branch with this change and a regression test that reads a multi-segment sparse member through extractfile().read() and in chunks, and compares both to the expected content. I can open a PR if this looks right.

Dominant language
Python
Stars
77.8k
Forks
37.7k
Avg merge
11h 25m
Merged PRs (30d)
723

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from python/cpython

All issues in python/cpython

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.