FetchItemQueues removes empty queues, which skips the politeness delay

オープン
  • 何か問題がありますか？
#2,303 コメント 2 件 リアクション 0 件 担当者 2 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@GGraziadei がすでに取り組んでいます。

2026年11月6日 から。

  • #2306 @dpol1 による — オープン

評価

この issue はまだ評価されていません。

説明

bug

FetchItemQueues.finishFetchItem removes a queue as soon as its last URL is done. The next URL for that host gets a new queue, which is ready straight away. If that URL arrives after the queue has emptied, for instance an outlink of the host's last page, it can be fetched at once, ignoring fetcher.server.delay and the Crawl-delay from the robots.txt.

This test fails in FetchItemQueuesTest, as a.net/2 is handed out at once:

FetchItemQueues q = queues("fetcher.server.delay", 5.0f);
add(q, "http://a.net/1");
q.finishFetchItem(q.getFetchItem(), false); // the queue of a.net is removed
add(q, "http://a.net/2");
Assertions.assertNull(q.getFetchItem());

In a local crawl with fetcher.server.delay: 3, I got two requests to the same host 11 ms apart. In 12-minute crawls of GQ32K hosts with the default delay of 1 s, a fetch started less than 1 s after the previous one to the same host had ended in 107 of 18,315 cases with OpenSearch, and in 5,422 of 14,151 with URLFrontier, across 799 hosts. The usual gap is just over 1 s. The higher figure is probably because URLFrontier hands out a host's next URLs sooner than the OpenSearch spout, so they more often arrive while the delay is still running. These figures come from the worker logs. I haven't traced them to this code path, and a shorter Crawl-delay could explain some of them.

3.7.0 has the same gap, though not in every case. Its empty queues stay until a getFetchItem scan reaches them, and the threads scan after each fetch and every 100 ms when idle, so a URL arriving before the scan still waits. Since #2132 finishFetchItem removes the queue itself, so the new queue is immediately eligible. I ran the 3.7.0 case only on its queue code.

NUTCH-2947 fixed the same problem in Nutch 1.19 by keeping the state of empty queues. I suggest keeping the time of a host's last fetch until its delay has passed. I can open a PR with the test and a fix, for 4.0 or later.

cc @GGraziadei, since #2132 expected the per-host rate to stay bounded by the crawl delay.

主要言語
Java
スター
1k
フォーク
297
平均マージ
1日 11時間
マージ済み PR（30日）
61

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/stormcrawler のほかの issue

apache/stormcrawler の issue をすべて見る

似ている issue

Java の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。