Push publish: sender marks bundle FAILED_TO_PUBLISH while receiver is still publishing, never records receiver's SUCCESS
维护者通常 2 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 48/100
- Issue 类型
- 缺陷
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
- 技术栈
- java
调研方向
Start with dotCMS/src/main/java/com/dotcms/publisher/business/PublisherQueueJob.java, especially updateBundleStatus() and the retry accounting around lines 504–505 and 580–584; then read PublishAuditAPIImpl.java and PushPublisher.java to trace polling and send counts. Check the related issue #38109 and add the integration test described in the acceptance criteria. Done means a later receiver SUCCESS is recorded by the sender without losing push history, while long-running publishing remains bounded.
由索引模型根据 Issue 内容生成。
描述
Problem Statement
The sending environment marks a push-publish bundle as FAILED_TO_PUBLISH while the receiver is still processing it, and never corrects it after the receiver succeeds. The Publishing Queue on the sender shows Failed to Publish with the endpoint frozen at Publishing Bundle, while the receiver's history shows Success.
Mechanism (dotCMS/src/main/java/com/dotcms/publisher/business/PublisherQueueJob.java):
updateBundleStatus()finalizes a bundle when the retry budget is spent and the receiver reports any group still publishing, treating "still publishing" the same as "failed" (PublisherQueueJob.java:504-505):
That branch setsif ( localHistory.getNumTries() >= MAX_NUM_TRIES && (groupPushStats.getCountGroupFailed() > 0 || groupPushStats.getCountGroupPublishing() > 0) ) {FAILED_TO_PUBLISH, callsdeletePushedAssets()for the bundle's environments and deletes the bundle frompublishing_queue.FAILED_TO_PUBLISHis not inSELECT_PENDING_BUNDLES(PublishAuditAPIImpl.java:40-42), so the sender never polls the receiver for that bundle again. The receiver's laterSUCCESSis never picked up.numTriesis incremented on every send (PushPublisher.java:183) and on every poll that returns a failure status (PublisherQueueJob.java:580-584). With the defaultPUBLISHER_QUEUE_MAX_TRIES=3, a single receiver-side failure exhausts the budget in two ticks: send (1) → failure poll (2) → re-send (3). The very next poll, one minute after the re-send, finalizes the bundle if the receiver is still working on it. In practice the default of 3 tries allows exactly one re-send, judged about 60 seconds later.
Impact
- Wrong terminal status on the sender for a bundle that actually published, so users retry or investigate pushes that succeeded.
deletePushedAssets()removes the sender's push history for every asset in the bundle, so the next "Everything and Dependencies" push treats them as never sent and re-includes them.- The likelihood scales with bundle size: any bundle whose retry takes longer than one poll interval on the receiver is affected. Large "Force Push Everything" bundles are almost guaranteed to hit it.
Observed in production (26.10.26-01, sanitized): a ~2,000-asset bundle failed on the receiver on its first attempt (transient ContentletIdentifier lock timeout). The sender's log shows:
-> Status : FAILED_TO_SEND_TO_ALL_GROUPS [4] Re-publish attempts: 1 out of 3 (+ "is stalled" NOTE)
-> Status : FAILED_TO_SEND_TO_ALL_GROUPS [4] Re-publish attempts: 2 out of 3
Processing bundle: ID: <bundle>. Status: 4 <- re-send
-> Status : FAILED_TO_PUBLISH [8] Re-publish attempts: 3 out of 3 <- 1 minute after re-send
The receiver logged Started bundle publish process 3 seconds after the re-send, was still running handlers when the sender finalized, and logged Finished bundle publish process with SUCCESS 21 minutes later. The sender never polled the bundle again.
Related, but a different trigger: dotCMS/core#38109 (bundles stuck forever in 12/13 because numTries is never incremented there). A fix for #38109 that starts counting tries while the receiver reports PUBLISHING_BUNDLE would make this bug fire more often, so the two should be fixed together.
Steps to Reproduce
Minimal reproduction (no receiver failure needed):
- Set up two dotCMS instances with push publishing between them (sender → receiver).
- On the sender, set
DOT_PUBLISHER_QUEUE_MAX_TRIES=1and restart. The value is astatic finalread at class load. - Push a bundle that takes longer than one
PublisherQueueJobinterval (1 minute) to process on the receiver, e.g. Force Push Everything on a site with several hundred content items. - Wait for the next sender poll.
Actual: the sender logs, on its first poll after the send:
16:40:47.103 PushPublisher : Status Update: Bundle sent
16:41:00.199 PublisherQueueJob : Status Update: Failed to publish
16:41:00.204 For bundle '<bundle>':
16:41:00.204 -> Status : FAILED_TO_PUBLISH [8]
16:41:00.204 -> Re-publish attempts: 1 out of 1
The sender's Publishing Queue shows Failed to Publish, the endpoint row shows Publishing Bundle, and Publish end is blank. The receiver finishes about a minute later with Success / Everything ok. The sender never updates.
Expected: a bundle the receiver reports as still publishing is not finalized as failed. The sender keeps polling until the receiver reports a terminal status, and records SUCCESS when the receiver succeeds.
Default-settings variant (matches the production sequence): with MAX_NUM_TRIES=3, make the receiver's first attempt fail (e.g. hold a FOR UPDATE row lock on a contentlet_version_info row the bundle updates, then pg_terminate_backend the blocked receiver connection), keep the lock held so the re-send blocks, and observe FAILED_TO_PUBLISH 3 out of 3 one poll after the re-send. Release the lock and the receiver reports SUCCESS.
Acceptance Criteria
-
updateBundleStatus()does not finalize a bundle asFAILED_TO_PUBLISHwhile any receiver group reportsPUBLISHING_BUNDLE, regardless ofnumTries. - A bundle whose receiver later reports
SUCCESS/SUCCESS_WITH_WARNINGSends in that status on the sender, and its push history (publishing_pushed_assets) is kept. - Polls that see a receiver status no longer consume the re-send budget, so
PUBLISHER_QUEUE_MAX_TRIEScounts sends (default 3 allows more than one re-send after a receiver failure). - Long-running receiver processing is still bounded by some safety net (coordinate with dotCMS/core#38109), so a bundle cannot sit in a non-terminal status forever.
- Integration test: with
MAX_NUM_TRIES=1and a receiver that reportsPUBLISHING_BUNDLEon the first poll andSUCCESSon a later one, the sender ends inSUCCESS.
dotCMS Version
- 26.10.26-01 (customer production, observed)
- 26.10.29-01 (local reproduction)
- Current
main: same code (PublisherQueueJob.java:504-505,:580-584;PushPublisher.java:183)
The getCountGroupPublishing() > 0 clause dates back to at least 2019 (dotCMS/core#17626 era).
Severity
Medium - Some functionality impacted
Links
- 主要语言
- Java
- 星标
- 972
- 派生
- 493
- 平均合并
- 2 天 12 小时
- 30 天内合并 PR
- 228
环境准备
在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
dotCMS/core 的其他 Issue
-
dotCMS : UVE OKR : Customer Support Team : Maintenance Type : Defect
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 2 天内回复
-
OKR : Customer Support Team : Maintenance Type : Defect
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 2 天内回复
-
Opening content from Content Search replaces the "Search" breadcrumb in the new Edit Content screen未关闭dotCMS: New Edit Contentlet OKR : Customer Support Team : Falcon Type : Defect
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 2 天内回复
-
Escape the Binary field hint in the legacy content editor so an apostrophe doesn't break the field未关闭OKR : Customer Support Team : Maintenance Type : Defect
难度 2/5 1-3 小时 新手友好度 79/100
维护者通常 2 天内回复
-
dotCMS : Permissions OKR : Customer Support Team : Maintenance Type : Defect
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 2 天内回复
相似的 Issue
-
desktop grp:about type:graphics-bug ui-public
难度 1/5 1 小时以内 新手友好度 68/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 74/100
维护者通常 2 天内回复
-
enhancement
难度 2/5 1-3 小时 新手友好度 70/100
it-at-m/Wahllokalsystem#3846 ·
维护者通常 2 天内回复
-
Area/Connector module/solace Priority/Low Type/Bug
难度 2/5 1-3 小时 新手友好度 77/100
ballerina-platform/ballerina-library#9849 ·
维护者通常 1 天内回复