Push publish: sender marks bundle FAILED_TO_PUBLISH while receiver is still publishing, never records receiver's SUCCESS

Abierto
#38,607 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
java

Línea de trabajo

Start with dotCMS/src/main/java/com/dotcms/publisher/business/PublisherQueueJob.java, especially updateBundleStatus() and the retry accounting around lines 504–505 and 580–584; then read PublishAuditAPIImpl.java and PushPublisher.java to trace polling and send counts. Check the related issue #38109 and add the integration test described in the acceptance criteria. Done means a later receiver SUCCESS is recorded by the sender without losing push history, while long-running publishing remains bounded.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

dotCMS : Push Publishing OKR : Customer Support Team : Maintenance Type : Defect
Problem Statement

The sending environment marks a push-publish bundle as FAILED_TO_PUBLISH while the receiver is still processing it, and never corrects it after the receiver succeeds. The Publishing Queue on the sender shows Failed to Publish with the endpoint frozen at Publishing Bundle, while the receiver's history shows Success.

Mechanism (dotCMS/src/main/java/com/dotcms/publisher/business/PublisherQueueJob.java):

  1. updateBundleStatus() finalizes a bundle when the retry budget is spent and the receiver reports any group still publishing, treating "still publishing" the same as "failed" (PublisherQueueJob.java:504-505):
    if ( localHistory.getNumTries() >= MAX_NUM_TRIES && (groupPushStats.getCountGroupFailed() > 0
            || groupPushStats.getCountGroupPublishing() > 0) ) {
    
    That branch sets FAILED_TO_PUBLISH, calls deletePushedAssets() for the bundle's environments and deletes the bundle from publishing_queue.
  2. FAILED_TO_PUBLISH is not in SELECT_PENDING_BUNDLES (PublishAuditAPIImpl.java:40-42), so the sender never polls the receiver for that bundle again. The receiver's later SUCCESS is never picked up.
  3. numTries is incremented on every send (PushPublisher.java:183) and on every poll that returns a failure status (PublisherQueueJob.java:580-584). With the default PUBLISHER_QUEUE_MAX_TRIES=3, a single receiver-side failure exhausts the budget in two ticks: send (1) → failure poll (2) → re-send (3). The very next poll, one minute after the re-send, finalizes the bundle if the receiver is still working on it. In practice the default of 3 tries allows exactly one re-send, judged about 60 seconds later.

Impact

  • Wrong terminal status on the sender for a bundle that actually published, so users retry or investigate pushes that succeeded.
  • deletePushedAssets() removes the sender's push history for every asset in the bundle, so the next "Everything and Dependencies" push treats them as never sent and re-includes them.
  • The likelihood scales with bundle size: any bundle whose retry takes longer than one poll interval on the receiver is affected. Large "Force Push Everything" bundles are almost guaranteed to hit it.

Observed in production (26.10.26-01, sanitized): a ~2,000-asset bundle failed on the receiver on its first attempt (transient ContentletIdentifier lock timeout). The sender's log shows:

-> Status : FAILED_TO_SEND_TO_ALL_GROUPS [4]   Re-publish attempts: 1 out of 3   (+ "is stalled" NOTE)
-> Status : FAILED_TO_SEND_TO_ALL_GROUPS [4]   Re-publish attempts: 2 out of 3
Processing bundle: ID: <bundle>. Status: 4                         <- re-send
-> Status : FAILED_TO_PUBLISH [8]              Re-publish attempts: 3 out of 3   <- 1 minute after re-send

The receiver logged Started bundle publish process 3 seconds after the re-send, was still running handlers when the sender finalized, and logged Finished bundle publish process with SUCCESS 21 minutes later. The sender never polled the bundle again.

Related, but a different trigger: dotCMS/core#38109 (bundles stuck forever in 12/13 because numTries is never incremented there). A fix for #38109 that starts counting tries while the receiver reports PUBLISHING_BUNDLE would make this bug fire more often, so the two should be fixed together.

Steps to Reproduce

Minimal reproduction (no receiver failure needed):

  1. Set up two dotCMS instances with push publishing between them (sender → receiver).
  2. On the sender, set DOT_PUBLISHER_QUEUE_MAX_TRIES=1 and restart. The value is a static final read at class load.
  3. Push a bundle that takes longer than one PublisherQueueJob interval (1 minute) to process on the receiver, e.g. Force Push Everything on a site with several hundred content items.
  4. Wait for the next sender poll.

Actual: the sender logs, on its first poll after the send:

16:40:47.103  PushPublisher : Status Update: Bundle sent
16:41:00.199  PublisherQueueJob : Status Update: Failed to publish
16:41:00.204  For bundle '<bundle>':
16:41:00.204  -> Status             : FAILED_TO_PUBLISH [8]
16:41:00.204  -> Re-publish attempts: 1 out of 1

The sender's Publishing Queue shows Failed to Publish, the endpoint row shows Publishing Bundle, and Publish end is blank. The receiver finishes about a minute later with Success / Everything ok. The sender never updates.

Expected: a bundle the receiver reports as still publishing is not finalized as failed. The sender keeps polling until the receiver reports a terminal status, and records SUCCESS when the receiver succeeds.

Default-settings variant (matches the production sequence): with MAX_NUM_TRIES=3, make the receiver's first attempt fail (e.g. hold a FOR UPDATE row lock on a contentlet_version_info row the bundle updates, then pg_terminate_backend the blocked receiver connection), keep the lock held so the re-send blocks, and observe FAILED_TO_PUBLISH 3 out of 3 one poll after the re-send. Release the lock and the receiver reports SUCCESS.

Acceptance Criteria
  • updateBundleStatus() does not finalize a bundle as FAILED_TO_PUBLISH while any receiver group reports PUBLISHING_BUNDLE, regardless of numTries.
  • A bundle whose receiver later reports SUCCESS / SUCCESS_WITH_WARNINGS ends in that status on the sender, and its push history (publishing_pushed_assets) is kept.
  • Polls that see a receiver status no longer consume the re-send budget, so PUBLISHER_QUEUE_MAX_TRIES counts sends (default 3 allows more than one re-send after a receiver failure).
  • Long-running receiver processing is still bounded by some safety net (coordinate with dotCMS/core#38109), so a bundle cannot sit in a non-terminal status forever.
  • Integration test: with MAX_NUM_TRIES=1 and a receiver that reports PUBLISHING_BUNDLE on the first poll and SUCCESS on a later one, the sender ends in SUCCESS.
dotCMS Version
  • 26.10.26-01 (customer production, observed)
  • 26.10.29-01 (local reproduction)
  • Current main: same code (PublisherQueueJob.java:504-505, :580-584; PushPublisher.java:183)

The getCountGroupPublishing() > 0 clause dates back to at least 2019 (dotCMS/core#17626 era).

Severity

Medium - Some functionality impacted

Links
Lenguaje dominante
Java
Estrellas
972
Forks
493
Merge medio
2 d 12 h
PR fusionados (30 d)
228

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de dotCMS/core

Todos los issues de dotCMS/core

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.