Push publish: sender marks bundle FAILED_TO_PUBLISH while receiver is still publishing, never records receiver's SUCCESS

未关闭
#38,607 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 2 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
48/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
java

调研方向

Start with dotCMS/src/main/java/com/dotcms/publisher/business/PublisherQueueJob.java, especially updateBundleStatus() and the retry accounting around lines 504–505 and 580–584; then read PublishAuditAPIImpl.java and PushPublisher.java to trace polling and send counts. Check the related issue #38109 and add the integration test described in the acceptance criteria. Done means a later receiver SUCCESS is recorded by the sender without losing push history, while long-running publishing remains bounded.

由索引模型根据 Issue 内容生成。

描述

dotCMS : Push Publishing OKR : Customer Support Team : Maintenance Type : Defect
Problem Statement

The sending environment marks a push-publish bundle as FAILED_TO_PUBLISH while the receiver is still processing it, and never corrects it after the receiver succeeds. The Publishing Queue on the sender shows Failed to Publish with the endpoint frozen at Publishing Bundle, while the receiver's history shows Success.

Mechanism (dotCMS/src/main/java/com/dotcms/publisher/business/PublisherQueueJob.java):

  1. updateBundleStatus() finalizes a bundle when the retry budget is spent and the receiver reports any group still publishing, treating "still publishing" the same as "failed" (PublisherQueueJob.java:504-505):
    if ( localHistory.getNumTries() >= MAX_NUM_TRIES && (groupPushStats.getCountGroupFailed() > 0
            || groupPushStats.getCountGroupPublishing() > 0) ) {
    
    That branch sets FAILED_TO_PUBLISH, calls deletePushedAssets() for the bundle's environments and deletes the bundle from publishing_queue.
  2. FAILED_TO_PUBLISH is not in SELECT_PENDING_BUNDLES (PublishAuditAPIImpl.java:40-42), so the sender never polls the receiver for that bundle again. The receiver's later SUCCESS is never picked up.
  3. numTries is incremented on every send (PushPublisher.java:183) and on every poll that returns a failure status (PublisherQueueJob.java:580-584). With the default PUBLISHER_QUEUE_MAX_TRIES=3, a single receiver-side failure exhausts the budget in two ticks: send (1) → failure poll (2) → re-send (3). The very next poll, one minute after the re-send, finalizes the bundle if the receiver is still working on it. In practice the default of 3 tries allows exactly one re-send, judged about 60 seconds later.

Impact

  • Wrong terminal status on the sender for a bundle that actually published, so users retry or investigate pushes that succeeded.
  • deletePushedAssets() removes the sender's push history for every asset in the bundle, so the next "Everything and Dependencies" push treats them as never sent and re-includes them.
  • The likelihood scales with bundle size: any bundle whose retry takes longer than one poll interval on the receiver is affected. Large "Force Push Everything" bundles are almost guaranteed to hit it.

Observed in production (26.10.26-01, sanitized): a ~2,000-asset bundle failed on the receiver on its first attempt (transient ContentletIdentifier lock timeout). The sender's log shows:

-> Status : FAILED_TO_SEND_TO_ALL_GROUPS [4]   Re-publish attempts: 1 out of 3   (+ "is stalled" NOTE)
-> Status : FAILED_TO_SEND_TO_ALL_GROUPS [4]   Re-publish attempts: 2 out of 3
Processing bundle: ID: <bundle>. Status: 4                         <- re-send
-> Status : FAILED_TO_PUBLISH [8]              Re-publish attempts: 3 out of 3   <- 1 minute after re-send

The receiver logged Started bundle publish process 3 seconds after the re-send, was still running handlers when the sender finalized, and logged Finished bundle publish process with SUCCESS 21 minutes later. The sender never polled the bundle again.

Related, but a different trigger: dotCMS/core#38109 (bundles stuck forever in 12/13 because numTries is never incremented there). A fix for #38109 that starts counting tries while the receiver reports PUBLISHING_BUNDLE would make this bug fire more often, so the two should be fixed together.

Steps to Reproduce

Minimal reproduction (no receiver failure needed):

  1. Set up two dotCMS instances with push publishing between them (sender → receiver).
  2. On the sender, set DOT_PUBLISHER_QUEUE_MAX_TRIES=1 and restart. The value is a static final read at class load.
  3. Push a bundle that takes longer than one PublisherQueueJob interval (1 minute) to process on the receiver, e.g. Force Push Everything on a site with several hundred content items.
  4. Wait for the next sender poll.

Actual: the sender logs, on its first poll after the send:

16:40:47.103  PushPublisher : Status Update: Bundle sent
16:41:00.199  PublisherQueueJob : Status Update: Failed to publish
16:41:00.204  For bundle '<bundle>':
16:41:00.204  -> Status             : FAILED_TO_PUBLISH [8]
16:41:00.204  -> Re-publish attempts: 1 out of 1

The sender's Publishing Queue shows Failed to Publish, the endpoint row shows Publishing Bundle, and Publish end is blank. The receiver finishes about a minute later with Success / Everything ok. The sender never updates.

Expected: a bundle the receiver reports as still publishing is not finalized as failed. The sender keeps polling until the receiver reports a terminal status, and records SUCCESS when the receiver succeeds.

Default-settings variant (matches the production sequence): with MAX_NUM_TRIES=3, make the receiver's first attempt fail (e.g. hold a FOR UPDATE row lock on a contentlet_version_info row the bundle updates, then pg_terminate_backend the blocked receiver connection), keep the lock held so the re-send blocks, and observe FAILED_TO_PUBLISH 3 out of 3 one poll after the re-send. Release the lock and the receiver reports SUCCESS.

Acceptance Criteria
  • updateBundleStatus() does not finalize a bundle as FAILED_TO_PUBLISH while any receiver group reports PUBLISHING_BUNDLE, regardless of numTries.
  • A bundle whose receiver later reports SUCCESS / SUCCESS_WITH_WARNINGS ends in that status on the sender, and its push history (publishing_pushed_assets) is kept.
  • Polls that see a receiver status no longer consume the re-send budget, so PUBLISHER_QUEUE_MAX_TRIES counts sends (default 3 allows more than one re-send after a receiver failure).
  • Long-running receiver processing is still bounded by some safety net (coordinate with dotCMS/core#38109), so a bundle cannot sit in a non-terminal status forever.
  • Integration test: with MAX_NUM_TRIES=1 and a receiver that reports PUBLISHING_BUNDLE on the first poll and SUCCESS on a later one, the sender ends in SUCCESS.
dotCMS Version
  • 26.10.26-01 (customer production, observed)
  • 26.10.29-01 (local reproduction)
  • Current main: same code (PublisherQueueJob.java:504-505, :580-584; PushPublisher.java:183)

The getCountGroupPublishing() > 0 clause dates back to at least 2019 (dotCMS/core#17626 era).

Severity

Medium - Some functionality impacted

Links
主要语言
Java
星标
972
派生
493
平均合并
2 天 12 小时
30 天内合并 PR
228

环境准备

在 Codespaces 中打开

在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

dotCMS/core 的其他 Issue

查看 dotCMS/core 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。