Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Bug]: PDF URI links are silently discarded by PDFContentScrapingStrategy due to Links schema mismatch

Đang mở
  • Có gì không ổn?
#2,374 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@Hugo-DDT đang làm issue này rồi.

Từ ngày 15/10/2026.

  • #2375 của @Hugo-DDT — đang mở

Đánh giá

Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức phù hợp với người mới
35/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python
Lĩnh vực
data

Hướng nghiên cứu

Start in crawl4ai/processors/pdf/__init__.py at PDFContentScrapingStrategy.scrap(), where the {"urls": []} dictionary is built, then compare it with Links, Link and ScrapingResult in crawl4ai/models.py. Reproduce with the snippet in the issue, which should show URI links present in processor.py output but missing from scraped.links. The maintainer has not yet confirmed the link representation, so agree on the contract before changing it. Done means a generated PDF's URI link appears in ScrapingResult.links with its page preserved.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

crawl4ai version

0.9.5 (also observed in 0.9.4).

Expected Behavior

URI link annotations extracted from a PDF should survive PDFContentScrapingStrategy.scrap() and be accessible in the returned ScrapingResult.links, with their source page numbers preserved or represented by an explicitly documented contract.

This report concerns preservation of the extracted link inventory, not reconstructing clickable text positions in HTML/Markdown or PDF-internal page destinations.

Current Behavior

The public processor extracts the URI correctly, but the public PDF scraping strategy returns success=True and empty internal / external link lists for the same file.

The apparent cause is a schema mismatch:

  • PDFContentScrapingStrategy.scrap() builds links = {"urls": []} and appends {"url": link, "page": page.page_number}.
  • It passes that dictionary to ScrapingResult.
  • ScrapingResult.links is typed as Links, which defines only internal and external. Its Link items use href rather than url and have no explicit PDF page field.
  • The unrecognized urls key is silently ignored during model construction, losing the collected links.

Sources: PDF scraper, models, and PDF processor.

The PDF parsing documentation also describes links["urls"], suggesting that preserving PDF links is intended, but the current result model cannot represent that structure.

Is this reproducible?

Yes.

Inputs Causing the Bug

A local, generated one-page PDF containing one URI annotation. No browser, network requests, external files, credentials, or application-specific adapter is needed. The page is intentionally blank apart from its annotation to isolate link preservation.

Steps to Reproduce
  1. Use Python 3.12 and install pip install "crawl4ai[pdf]==0.9.5".
  2. Save the following snippet as repro_pdf_links.py.
  3. Run python repro_pdf_links.py.
  4. Observe that the processor contains the URI but the scraping result has empty link lists.
Code snippets
from pathlib import Path
from tempfile import TemporaryDirectory

from pypdf import PdfWriter
from pypdf.annotations import Link

from crawl4ai.processors.pdf import PDFContentScrapingStrategy
from crawl4ai.processors.pdf.processor import NaivePDFProcessorStrategy

with TemporaryDirectory() as folder:
    path = Path(folder) / "linked.pdf"
    writer = PdfWriter()
    writer.add_blank_page(width=612, height=792)
    writer.add_annotation(
        0,
        Link(rect=(20, 20, 200, 40), url="https://example.org/reference"),
    )
    writer.write(path)

    processed = NaivePDFProcessorStrategy(extract_images=False).process(path)
    scraped = PDFContentScrapingStrategy(extract_images=False).scrap(str(path), "")

    print("processor links:", processed.pages[0].links)
    print("scraping success:", scraped.success)
    print("scraping links:", scraped.links.model_dump())
OS

macOS 27.0.1, Apple Silicon (arm64).

Python version

3.12.13; pypdf 6.20.1.

Browser

Not applicable; the reproduction uses a local PDF and public SDK methods directly.

Browser version

Not applicable.

Error logs & Screenshots (if applicable)

No exception is raised. Actual output on 0.9.5:

processor links: ['https://example.org/reference']
scraping success: True
scraping links: {'internal': [], 'external': []}
Proposed direction / contribution

I would like to contribute a focused fix targeting develop, with regression coverage using a generated PDF and updated documentation. Before changing the public link contract, could you confirm the intended representation?

One possible approach is to map PDF URI links into the existing internal / external structure using href, and add an optional 1-based page field to Link so PDF page attribution is not lost. For remote PDFs, classification could reuse the existing URL classification rules; for local PDFs without a source URL, treating URI links as external is a possible fallback, subject to maintainer preference.

This is a proposal rather than an assumed API decision. Reader reuse / parallel parsing optimization and reconstruction of inline clickable text would remain outside this fix.

Ngôn ngữ chính
Python
Star
85.9k
Fork
8.9k
Merge trung bình
5 ngày 7 giờ
Pull request đã merge (30 ngày)
23

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của unclecode/crawl4ai

Tất cả issue của unclecode/crawl4ai

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.