FetchItemQueues removes empty queues, which skips the politeness delay

Abierto
  • ¿Algo no está bien?
#2,303 2 comentarios 0 reacciones 2 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@GGraziadei ya está trabajando en esto.

Desde el 6/11/2026.

  • #2306 de @dpol1 — abierto

Evaluación

Este issue todavía no se ha evaluado.

Descripción

bug

FetchItemQueues.finishFetchItem removes a queue as soon as its last URL is done. The next URL for that host gets a new queue, which is ready straight away. If that URL arrives after the queue has emptied, for instance an outlink of the host's last page, it can be fetched at once, ignoring fetcher.server.delay and the Crawl-delay from the robots.txt.

This test fails in FetchItemQueuesTest, as a.net/2 is handed out at once:

FetchItemQueues q = queues("fetcher.server.delay", 5.0f);
add(q, "http://a.net/1");
q.finishFetchItem(q.getFetchItem(), false); // the queue of a.net is removed
add(q, "http://a.net/2");
Assertions.assertNull(q.getFetchItem());

In a local crawl with fetcher.server.delay: 3, I got two requests to the same host 11 ms apart. In 12-minute crawls of GQ32K hosts with the default delay of 1 s, a fetch started less than 1 s after the previous one to the same host had ended in 107 of 18,315 cases with OpenSearch, and in 5,422 of 14,151 with URLFrontier, across 799 hosts. The usual gap is just over 1 s. The higher figure is probably because URLFrontier hands out a host's next URLs sooner than the OpenSearch spout, so they more often arrive while the delay is still running. These figures come from the worker logs. I haven't traced them to this code path, and a shorter Crawl-delay could explain some of them.

3.7.0 has the same gap, though not in every case. Its empty queues stay until a getFetchItem scan reaches them, and the threads scan after each fetch and every 100 ms when idle, so a URL arriving before the scan still waits. Since #2132 finishFetchItem removes the queue itself, so the new queue is immediately eligible. I ran the 3.7.0 case only on its queue code.

NUTCH-2947 fixed the same problem in Nutch 1.19 by keeping the state of empty queues. I suggest keeping the time of a host's last fetch until its delay has passed. I can open a PR with the test and a fix, for 4.0 or later.

cc @GGraziadei, since #2132 expected the per-host rate to stay bounded by the crawl delay.

Lenguaje dominante
Java
Estrellas
1k
Forks
297
Merge medio
1 d 11 h
PR fusionados (30 d)
61

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/stormcrawler

Todos los issues de apache/stormcrawler

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.