[Bug]: FreshnessScorer defaults to current_year=2024, so this year's URLs score like undated ones

Ouverte
#2,426 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@Siddharth-DWT y travaille déjà.

Depuis le 21/10/2026.

  • #2427 par @Siddharth-DWT — ouverte
  • #2431 par @SohamKukreti — ouverte

Évaluation

Difficulté
2/5
Temps estimé
1-3 heures
Accessibilité débutants
35/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
python
Domaine
backend

Piste de recherche

Start with crawl4ai/deep_crawling/scorers.py and inspect FreshnessScorer's current_year default and _extract_year; reproduce the listed URL scores. Check the documentation example that uses FreshnessScorer(weight=0.8). Done when the default ranks a URL from the current year above an undated URL, while an explicitly supplied current_year keeps working as before.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

⚙️ In-progress 📌 Root caused
crawl4ai version

0.9.5 (also on main and develop)

Expected Behavior

As the docstring says, "more recent years score higher": with the default settings a URL dated this year scores highest.

Current Behavior

current_year defaults to a hard-coded 2024, and _extract_year discards any year later than current_year as invalid. So today, with the default:

  • a /2026/ or /2025/ URL scores 0.5, the same as a URL with no date
  • a /2024/ URL scores 1.0

BestFirstCrawlingStrategy with a default FreshnessScorer therefore ranks 2024 content above current content. The docs use it with the default (FreshnessScorer(weight=0.8)).

Is this reproducible?

Yes

Inputs Causing the Bug

Any FreshnessScorer() without current_year.

Steps to Reproduce

Run the snippet below.

Code snippets
from crawl4ai.deep_crawling.scorers import FreshnessScorer

scorer = FreshnessScorer()
for url in [
    "https://example.com/blog/2026/10/01/post",
    "https://example.com/blog/2024/01/01/post",
    "https://example.com/about",
]:
    print(url, scorer.score(url))
# 2026 -> 0.5, 2024 -> 1.0, undated -> 0.5
OS

Windows 11 (platform-independent)

Python version

3.13

Browser

Not involved

Error logs & Screenshots (if applicable)

None; no error is raised.

Fix: default current_year to None and use date.today().year; an explicit current_year keeps working as before. I'll open a PR against develop referencing this issue.

Investigated with AI assistance; I ran the reproduction and checked the output myself.

Langage dominant
Python
Étoiles
87.2k
Forks
9.1k
Merge moyen
1 j 6 h
PR mergées (30 j)
25

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de unclecode/crawl4ai

Toutes les issues de unclecode/crawl4ai

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.