Solr StatusUpdaterBolt overwrites the status of a URL when it is discovered again

Abierto
  • ¿Algo no está bien?
#2,304 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
55/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
java
Área
backend, databases

Línea de trabajo

Start with the Solr StatusUpdaterBolt and its status update path, then compare how the OpenSearch and SQL bolts handle DISCOVERED tuples for existing URLs. Reproduce the reported case with two bolt instances, including a topology restart, and check that an existing status record and its fetch metadata remain intact after the second DISCOVERED tuple.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug

The Solr StatusUpdaterBolt sends every status as a plain add, so a DISCOVERED tuple for a URL already in the status collection replaces its record. For DISCOVERED, the OpenSearch bolt uses create and the SQL one INSERT IGNORE, which keep what is there.

A test with two bolt instances, the second standing for a restarted topology, gives:

before: {"url":"https://www.url.net/missing.html","status":"FETCH_ERROR","metadata.fetch.error.count":["1"],"nextFetchDate":"2026-11-05T11:50:02Z",...}
after:  {"url":"https://www.url.net/missing.html","status":"DISCOVERED","nextFetchDate":"2026-11-05T09:50:03Z",...}

before: {"url":"https://www.url.net/denied.html","status":"ERROR","nextFetchDate":"2026-12-06T09:55:57Z",...}
after:  {"url":"https://www.url.net/denied.html","status":"DISCOVERED","nextFetchDate":"2026-11-05T09:55:58Z",...}

The test conf has no fetchInterval.error, hence the date in the ERROR record. The FETCH_ERROR test against the OpenSearch module keeps the record.

The status updater cache hides this as long as the URL is in it. After a topology restart, or once the entry has expired or been evicted, the DISCOVERED tuple reaches Solr. I first noticed this in a crawl with the Solr archetype: in a second crawl topology, the 8 outlinks of index.html were fetched again within a minute although none was due, and URLs that failed twice ended with fetch.error.count: 1.

From the code, an ERROR URL would also become due at once with fetchInterval.error: -1. For a URL denied by robots.txt the fetcher checks again, so the URL itself would get no request. The code at 3.7.0 sends the same add. I only ran this on main.

Solr's optimistic concurrency could give the same behaviour as OpenSearch: a negative _version_ means the document must not exist, and failOnVersionConflicts=false (SOLR-13320) lets the rest of a batch through. @mvolikas, would that fit the async batches of SolrConnection? I can open a PR with the test.

Separately, the bolt acks right after addAsync, before the update reaches Solr, which is why the test has to wait for the record to appear.

Lenguaje dominante
Java
Estrellas
1k
Forks
297
Merge medio
1 d 11 h
PR fusionados (30 d)
61

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/stormcrawler

Todos los issues de apache/stormcrawler

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.