[Bug]: FreshnessScorer defaults to current_year=2024, so this year's URLs score like undated ones

Abierto
#2,426 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@Siddharth-DWT ya está trabajando en esto.

Desde el 21/10/2026.

  • #2427 de @Siddharth-DWT — abierto
  • #2431 de @SohamKukreti — abierto

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
backend

Línea de trabajo

Start with crawl4ai/deep_crawling/scorers.py and inspect FreshnessScorer's current_year default and _extract_year; reproduce the listed URL scores. Check the documentation example that uses FreshnessScorer(weight=0.8). Done when the default ranks a URL from the current year above an undated URL, while an explicitly supplied current_year keeps working as before.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

⚙️ In-progress 📌 Root caused
crawl4ai version

0.9.5 (also on main and develop)

Expected Behavior

As the docstring says, "more recent years score higher": with the default settings a URL dated this year scores highest.

Current Behavior

current_year defaults to a hard-coded 2024, and _extract_year discards any year later than current_year as invalid. So today, with the default:

  • a /2026/ or /2025/ URL scores 0.5, the same as a URL with no date
  • a /2024/ URL scores 1.0

BestFirstCrawlingStrategy with a default FreshnessScorer therefore ranks 2024 content above current content. The docs use it with the default (FreshnessScorer(weight=0.8)).

Is this reproducible?

Yes

Inputs Causing the Bug

Any FreshnessScorer() without current_year.

Steps to Reproduce

Run the snippet below.

Code snippets
from crawl4ai.deep_crawling.scorers import FreshnessScorer

scorer = FreshnessScorer()
for url in [
    "https://example.com/blog/2026/10/01/post",
    "https://example.com/blog/2024/01/01/post",
    "https://example.com/about",
]:
    print(url, scorer.score(url))
# 2026 -> 0.5, 2024 -> 1.0, undated -> 0.5
OS

Windows 11 (platform-independent)

Python version

3.13

Browser

Not involved

Error logs & Screenshots (if applicable)

None; no error is raised.

Fix: default current_year to None and use date.today().year; an explicit current_year keeps working as before. I'll open a PR against develop referencing this issue.

Investigated with AI assistance; I ran the reproduction and checked the output myself.

Lenguaje dominante
Python
Estrellas
87.2k
Forks
9.1k
Merge medio
1 d 6 h
PR fusionados (30 d)
25

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de unclecode/crawl4ai

Todos los issues de unclecode/crawl4ai

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.