Push publish: sender marks bundle FAILED_TO_PUBLISH while receiver is still publishing, never records receiver's SUCCESS
Los mantenedores suelen responder en 2 días
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 48/100
- Tipo de issue
- Error
- Claridad
- Bien especificado
- Estado de actividad
- Activo
- Stack tecnológico
- java
- Área
- backend, distributed-systems, testing
Línea de trabajo
Start with dotCMS/src/main/java/com/dotcms/publisher/business/PublisherQueueJob.java, especially updateBundleStatus() and the retry accounting around lines 504–505 and 580–584; then read PublishAuditAPIImpl.java and PushPublisher.java to trace polling and send counts. Check the related issue #38109 and add the integration test described in the acceptance criteria. Done means a later receiver SUCCESS is recorded by the sender without losing push history, while long-running publishing remains bounded.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Problem Statement
The sending environment marks a push-publish bundle as FAILED_TO_PUBLISH while the receiver is still processing it, and never corrects it after the receiver succeeds. The Publishing Queue on the sender shows Failed to Publish with the endpoint frozen at Publishing Bundle, while the receiver's history shows Success.
Mechanism (dotCMS/src/main/java/com/dotcms/publisher/business/PublisherQueueJob.java):
updateBundleStatus()finalizes a bundle when the retry budget is spent and the receiver reports any group still publishing, treating "still publishing" the same as "failed" (PublisherQueueJob.java:504-505):
That branch setsif ( localHistory.getNumTries() >= MAX_NUM_TRIES && (groupPushStats.getCountGroupFailed() > 0 || groupPushStats.getCountGroupPublishing() > 0) ) {FAILED_TO_PUBLISH, callsdeletePushedAssets()for the bundle's environments and deletes the bundle frompublishing_queue.FAILED_TO_PUBLISHis not inSELECT_PENDING_BUNDLES(PublishAuditAPIImpl.java:40-42), so the sender never polls the receiver for that bundle again. The receiver's laterSUCCESSis never picked up.numTriesis incremented on every send (PushPublisher.java:183) and on every poll that returns a failure status (PublisherQueueJob.java:580-584). With the defaultPUBLISHER_QUEUE_MAX_TRIES=3, a single receiver-side failure exhausts the budget in two ticks: send (1) → failure poll (2) → re-send (3). The very next poll, one minute after the re-send, finalizes the bundle if the receiver is still working on it. In practice the default of 3 tries allows exactly one re-send, judged about 60 seconds later.
Impact
- Wrong terminal status on the sender for a bundle that actually published, so users retry or investigate pushes that succeeded.
deletePushedAssets()removes the sender's push history for every asset in the bundle, so the next "Everything and Dependencies" push treats them as never sent and re-includes them.- The likelihood scales with bundle size: any bundle whose retry takes longer than one poll interval on the receiver is affected. Large "Force Push Everything" bundles are almost guaranteed to hit it.
Observed in production (26.10.26-01, sanitized): a ~2,000-asset bundle failed on the receiver on its first attempt (transient ContentletIdentifier lock timeout). The sender's log shows:
-> Status : FAILED_TO_SEND_TO_ALL_GROUPS [4] Re-publish attempts: 1 out of 3 (+ "is stalled" NOTE)
-> Status : FAILED_TO_SEND_TO_ALL_GROUPS [4] Re-publish attempts: 2 out of 3
Processing bundle: ID: <bundle>. Status: 4 <- re-send
-> Status : FAILED_TO_PUBLISH [8] Re-publish attempts: 3 out of 3 <- 1 minute after re-send
The receiver logged Started bundle publish process 3 seconds after the re-send, was still running handlers when the sender finalized, and logged Finished bundle publish process with SUCCESS 21 minutes later. The sender never polled the bundle again.
Related, but a different trigger: dotCMS/core#38109 (bundles stuck forever in 12/13 because numTries is never incremented there). A fix for #38109 that starts counting tries while the receiver reports PUBLISHING_BUNDLE would make this bug fire more often, so the two should be fixed together.
Steps to Reproduce
Minimal reproduction (no receiver failure needed):
- Set up two dotCMS instances with push publishing between them (sender → receiver).
- On the sender, set
DOT_PUBLISHER_QUEUE_MAX_TRIES=1and restart. The value is astatic finalread at class load. - Push a bundle that takes longer than one
PublisherQueueJobinterval (1 minute) to process on the receiver, e.g. Force Push Everything on a site with several hundred content items. - Wait for the next sender poll.
Actual: the sender logs, on its first poll after the send:
16:40:47.103 PushPublisher : Status Update: Bundle sent
16:41:00.199 PublisherQueueJob : Status Update: Failed to publish
16:41:00.204 For bundle '<bundle>':
16:41:00.204 -> Status : FAILED_TO_PUBLISH [8]
16:41:00.204 -> Re-publish attempts: 1 out of 1
The sender's Publishing Queue shows Failed to Publish, the endpoint row shows Publishing Bundle, and Publish end is blank. The receiver finishes about a minute later with Success / Everything ok. The sender never updates.
Expected: a bundle the receiver reports as still publishing is not finalized as failed. The sender keeps polling until the receiver reports a terminal status, and records SUCCESS when the receiver succeeds.
Default-settings variant (matches the production sequence): with MAX_NUM_TRIES=3, make the receiver's first attempt fail (e.g. hold a FOR UPDATE row lock on a contentlet_version_info row the bundle updates, then pg_terminate_backend the blocked receiver connection), keep the lock held so the re-send blocks, and observe FAILED_TO_PUBLISH 3 out of 3 one poll after the re-send. Release the lock and the receiver reports SUCCESS.
Acceptance Criteria
-
updateBundleStatus()does not finalize a bundle asFAILED_TO_PUBLISHwhile any receiver group reportsPUBLISHING_BUNDLE, regardless ofnumTries. - A bundle whose receiver later reports
SUCCESS/SUCCESS_WITH_WARNINGSends in that status on the sender, and its push history (publishing_pushed_assets) is kept. - Polls that see a receiver status no longer consume the re-send budget, so
PUBLISHER_QUEUE_MAX_TRIEScounts sends (default 3 allows more than one re-send after a receiver failure). - Long-running receiver processing is still bounded by some safety net (coordinate with dotCMS/core#38109), so a bundle cannot sit in a non-terminal status forever.
- Integration test: with
MAX_NUM_TRIES=1and a receiver that reportsPUBLISHING_BUNDLEon the first poll andSUCCESSon a later one, the sender ends inSUCCESS.
dotCMS Version
- 26.10.26-01 (customer production, observed)
- 26.10.29-01 (local reproduction)
- Current
main: same code (PublisherQueueJob.java:504-505,:580-584;PushPublisher.java:183)
The getCountGroupPublishing() > 0 clause dates back to at least 2019 (dotCMS/core#17626 era).
Severity
Medium - Some functionality impacted
Links
- Lenguaje dominante
- Java
- Estrellas
- 972
- Forks
- 493
- Merge medio
- 2 d 12 h
- PR fusionados (30 d)
- 228
Preparar el entorno
Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.
- Sin Dockerfile ni archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de dotCMS/core
-
dotCMS : UVE OKR : Customer Support Team : Maintenance Type : Defect
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
Los mantenedores suelen responder en 2 días
-
OKR : Customer Support Team : Maintenance Type : Defect
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 2 días
-
Opening content from Content Search replaces the "Search" breadcrumb in the new Edit Content screenAbiertodotCMS: New Edit Contentlet OKR : Customer Support Team : Falcon Type : Defect
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 2 días
-
Escape the Binary field hint in the legacy content editor so an apostrophe doesn't break the fieldAbiertoOKR : Customer Support Team : Maintenance Type : Defect
Dificultad 2/5 1-3 horas Aptitud para principiantes 79/100
Los mantenedores suelen responder en 2 días
-
dotCMS : Permissions OKR : Customer Support Team : Maintenance Type : Defect
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
Los mantenedores suelen responder en 2 días
Todos los issues de dotCMS/core
Issues similares
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
it-at-m/Wahllokalsystem#3846 ·
Los mantenedores suelen responder en 2 días
-
Area/Connector module/solace Priority/Low Type/Bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100
ballerina-platform/ballerina-library#9849 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 65/100
Vakore/ZappierGames#327 ·
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
Los mantenedores suelen responder en 1 día
-
Radix number block generates NaN after switching a negative or decimal number to binary, octal or hexPosiblemente ocupada @matthewalberto la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 64/100
mit-cml/appinventor-sources#4246 · 1 comentario ·
Los mantenedores suelen responder en 1 día