Dash pattern operator `d` appends to dash_array instead of replacing it: memory grows quadratically on CAD drawings

Open Beginner friendly
  • Is something wrong?
#448 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 5 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
85/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp
Domain
backend

Research direction

Read src/parse/pdf_states/grph.h around line 626, where the d operator fills dash_array. Confirm the PDF dash pattern semantics described in the issue, including how an empty array resets the pattern, then reproduce with the provided script. Done means each d operator replaces the stored pattern and the reproduction no longer shows quadratic memory growth.

Written by the indexing model from the issue text.

Description

Summary

pdf_state<GRPH>::d appends the operands of every d operator to the existing dash_array instead of replacing it. Since every shape keeps a copy of the current dash array, memory grows with shapes × number of d operators since the last Q. CAD exports (seen from Creo Elements/Direct Drafting, SOLIDWORKS PDF Publisher, Ghostscript and 7-PDF) set a dash pattern before nearly every path and never use q/Q. A single A3 drawing with ~167k paths and ~4.4k d operators takes the process past 40 GB within about 12 s, even with OCR and table structure disabled. With pdf_backend: pypdfium2 the same file needs 2.7 GB.

Where

src/parse/pdf_states/grph.h (main at 1b7a5fe5e4, line 626):

if(arr.isArray())
  {
    for(int l=0; l<arr.getArrayNItems(); l++)
      {
        ...
        dash_array.push_back(val);   // never cleared before the loop
      }
  }

Per ISO 32000-1 §8.4.3.6, d sets the dash pattern. As a consequence of the append, [] 0 d doesn't reset to a solid line either: the loop runs zero times and the old pattern stays. Only Q restores it.

Reproduction

Self-contained script that writes a one-page PDF with n short lines and parses it:

import resource, sys, time
from docling_parse.pdf_parser import DoclingPdfParser


def make_pdf(path, n, wrap_in_q=False, dash_once=False):
    lines = []
    for i in range(n):
        dash = b"" if (dash_once and i) else b"[18 2 3 2] 0 d\n"
        line = dash + b"%d %d m %d %d l S\n" % (i % 1000, i // 10 % 800, i % 1000 + 1, i // 10 % 800)
        lines.append(b"q\n" + line + b"Q\n" if wrap_in_q else line)
    content = b"".join(lines)
    objs = [b"<< /Type /Catalog /Pages 2 0 R >>", b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
            b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 1190 842] /Contents 4 0 R >>",
            b"<< /Length %d >>\nstream\n" % len(content) + content + b"\nendstream"]
    out, offsets = bytearray(b"%PDF-1.4\n"), []
    for i, o in enumerate(objs, 1):
        offsets.append(len(out))
        out += b"%d 0 obj\n" % i + o + b"\nendobj\n"
    xref = len(out)
    out += b"xref\n0 %d\n0000000000 65535 f \n" % (len(objs) + 1)
    for o in offsets:
        out += b"%010d 00000 n \n" % o
    out += b"trailer\n<< /Size %d /Root 1 0 R >>\nstartxref\n%d\n%%%%EOF\n" % (len(objs) + 1, xref)
    open(path, "wb").write(out)


variant, n = sys.argv[1], int(sys.argv[2])  # variant: each | q | once
make_pdf("/tmp/t.pdf", n, wrap_in_q=variant == "q", dash_once=variant == "once")
t0 = time.time()
page = DoclingPdfParser().load("/tmp/t.pdf").get_page(1)
print(f"{variant:6} n={n:5}  shapes={len(page.shapes)}  {time.time() - t0:.1f}s  "
      f"peak RSS {resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 2**20:.2f} GiB")

Output with docling-parse 7.24.2 (Python 3.12, Linux x86_64):

each   n= 4000  shapes=4000  1.0s  peak RSS 2.34 GiB
each   n= 8000  shapes=8000  3.7s  peak RSS 8.88 GiB
q      n= 8000  shapes=8000  0.1s  peak RSS 0.20 GiB

With 7.22.3 the numbers are the same, plus once n=8000, which sets the pattern once before all lines: 0.19 GiB. Doubling the lines quadruples the memory (each). With the same lines wrapped in q … Q, or with a single d, it stays at 0.2 GiB.

Suggested fix

Clear the array before filling it, so that d replaces the pattern and [] 0 d resets it:

if(arr.isArray())
  {
    dash_array.clear();
    for(int l=0; l<arr.getArrayNItems(); l++)
Impact in practice

We run docling-serve 1.37.0 as a shared service. Eight real one-page drawings (A4 to A0, under 17 MB each) each took the worker into the container memory limit. Every such crash also killed the conversion running in parallel. As a workaround, we now compute the accumulated dash length per page from the pdf.js operator list on the client side and skip Docling above a threshold.

Dominant language
C++
Stars
346
Forks
92
Avg merge
4d 15h
Merged PRs (30d)
27

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from docling-project/docling-parse

All issues in docling-project/docling-parse

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.