Background reindex workers can overwhelm search capacity during large backlogs
I maintainer di solito rispondono entro 2 giorni
Nessuno ha ancora preso questa issue.
- #38586 di @wezell — chiusa senza merge
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 12/100
- Tipo di issue
- Bug
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Stack tecnologico
- elasticsearch, java
- Ambito
- backend, distributed-systems, performance, search
Direzione di ricerca
Start from the reindex journal code that reads the MAX_REINDEXING_NODES and REINDEX_NODE_LIMIT_MIN_PRIORITY settings and from ReindexEntry.isReindex(), which has a fixed boundary that must not be used for the configured cutoff. Follow participant selection, journal fetch, prefetch and stalled-share takeover, then run the reindex-package unit tests. Done means two of six nodes admit capped work after membership converges, the other four still process normal work, and no journal entries are stranded. The fix is already in PR #38586, which is closed and unmerged, so coordinate with maintainers before starting.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Defect Description
Large structure-wide and full-reindex backlogs can cause every serving application node to prepare documents and submit bulk indexing work concurrently, competing with production searches. Repeated production outages have involved exhausted request pools waiting on Elasticsearch searches while background indexing was also under pressure.
The observed deployment has six application pods and had approximately 200,000 queued content items. This fix must let operators limit structure/full-reindex participation to two workers while normal queued saves remain eligible on all six pods, without removing serving capacity or stranding journal entries. This is a production outage bug fix, not a new-feature request.
Status: IN PROGRESS — fix PR #38586 open; review/CI pending
Observed vs. Expected
- Observed: No configurable worker-participation limit; all six application nodes may drain a large backlog concurrently. Request pools can become exhausted waiting on searches. Reindexing is a plausible contributor, not an established sole cause of every outage.
- Expected: With
DOT_MAX_REINDEXING_NODES=2consistently configured, two selected nodes admit structure/full-reindex work after membership converges, all six remain eligible for normal queued saves and serving requests, and queued work remains recoverable during failover.
Reproduction / Regression Scenario
- Start with six eligible application nodes sharing the same search backend and a large background journal backlog.
- Run production-like searches while the backlog drains.
- Verify that current behavior has no way to restrict structure/full-reindex work to two workers while keeping normal queued saves eligible on all six pods.
- With the fix, verify priority-aware two-worker admission, complete journal coverage, exclusion of prefetched bulk work, continued normal-save processing on all six pods, and replacement of a failed participant.
Implementation
- Fix PR: #38586 (
issue-38579-limit-reindex-workers, commitc835f61df2). - Local validation: 80 reindex-package unit tests passed; the same 12 SQL-routing tests also passed against an isolated PostgreSQL 18 container. Five existing deduplication tests passed independently. Code/spec/documentation convergence is clean; six-pod peak-load validation and the full integration suite remain outstanding.
- The full defect specification is embedded below; reviewers do not need a separate document to understand the contract.
- Use the repository's urgent-incident path. Do not block the outage fix on approval of spec-only PR #38580.
- Preserve the repository's regression-test approval and Red-before-implementation requirements.
- No production deployment or configuration change has been made yet.
Defect Specification: Bound Background Reindex Workers
Created: 2026-11-12
Status: Outage bug fix — implemented in PR #38586; local regression checks passed, review/CI pending
Issue: dotCMS/core#38579
Input: Add MAX_REINDEXING_NODES so a six-node dotCMS cluster can restrict structure/full-reindex work to two selected workers while all six nodes remain eligible for normal queued saves and continue serving requests.
Configuration Contract
| Setting | Default | Meaning |
|---|---|---|
MAX_REINDEXING_NODES |
2 |
Maximum participants for capped work; explicit nonpositive values disable the node limit. An absent setting uses two workers. |
REINDEX_NODE_LIMIT_MIN_PRIORITY |
200 |
Inclusive minimum journal priority subject to the node limit. Container name: DOT_REINDEX_NODE_LIMIT_MIN_PRIORITY. |
The admission rule is: apply the limited participant set only when MAX_REINDEXING_NODES > 0 and an otherwise-eligible entry has priority >= REINDEX_NODE_LIMIT_MIN_PRIORITY. All lower-priority entries remain distributed across the full eligible application membership. Neither setting changes existing failed-entry/retry eligibility.
Examples with MAX_REINDEXING_NODES=2:
- Cutoff
0: all eligible journal work is limited to two participants. - Cutoff
100: normal, structure, and full-reindex work are limited; ASAP work remains eligible on all nodes. - Cutoff
200(default): structure and full-reindex work are limited; ASAP/normal work remains eligible on all nodes. - Cutoff
300: full-reindex work is limited; structure work also remains eligible on all nodes.
Unless a scenario explicitly sets another cutoff, all priority classifications and examples below assume the default cutoff of 200. These are numeric, inclusive priority comparisons, not a comparison against ReindexEntry.isReindex(). Operators should normally choose the category boundaries above because retry counts increment an entry's numeric priority. All nodes in one application cluster must use the same pair of settings; this does not promise instantaneous cluster-wide configuration changes. The node maximum defaults to two, so this changes existing behavior on upgrade. Operators requiring unrestricted participation must explicitly set MAX_REINDEXING_NODES=0; a one-node deployment uses its one available node. Mixed-version rollout requires particular care because old nodes do not enforce the new rule.
Problem and Scope
A large reindex backlog lets every application node contribute document preparation and bulk indexing work against the same search infrastructure. Operators need to reduce structure/full-reindex participation independently of the number of serving application nodes, without restricting normal queued saves to those same workers.
Changing REINDEX_THREAD_CONCURRENT_REQUESTS from 1 to 0 is not an equivalent worker limit. The Elasticsearch bulk client permits one in-flight request per processor for either value; zero prevents the producer from preparing the next bulk while the current request is outstanding. It does not promise to halve requests or indexing throughput.
This fix limits otherwise-eligible journal work with priority at or above the configured REINDEX_NODE_LIMIT_MIN_PRIORITY. At the default cutoff 200, this includes structure-wide updates, their retries, and full reindexes (300), while ASAP/normal work and its retries remain eligible on all application nodes. Queue action (update versus delete) does not bypass the configured priority rule. Direct synchronous indexing and explicit per-request indexing policy overrides are unchanged.
Do not use literal >100 or >200 comparisons: retries increment priority, so a normal retry can be 101 and a structure retry 201. The default boundary is >=200, but it is operator-configurable; ReindexEntry.isReindex() has a fixed >=300 boundary and must not substitute for the configured cutoff.
User Scenarios & Testing
User Story 1 — Limit background participation without reducing serving capacity (Priority: P1)
As an operator, I can configure MAX_REINDEXING_NODES=2 across a six-node application cluster so only two selected nodes admit structure/full-reindex work after membership converges, while all six continue processing their normal-priority journal shares and serving application requests.
Independent Test: Give six simulated nodes the same eligible membership and a nonempty journal. Verify that the same two nodes participate in structure/full-reindex work, the other four still process normal work but admit no capped work, and every eligible journal entry remains reachable by the participants for its priority class.
Acceptance Scenarios:
- Given six eligible nodes with the same configuration and membership, when the maximum is two, then every node agrees on the same two structure/full-reindex participants, while all six remain normal-work participants.
- Given a node not selected for structure/full-reindex work, when it checks for journal work, then it continues processing normal-priority entries but neither fetches nor consumes capped entries, including capped entries already in local prefetch.
- Given four nodes not selected for structure/full-reindex work, when they perform stalled-share recovery, then recovery cannot admit capped work on those nodes; normal-work recovery remains available.
- Given a mixed-priority backlog containing updates and deletes, when the priority-appropriate participants drain it, then limiting bulk participation does not strand entries or delete journal entries merely because their former owner is excluded from bulk work.
User Story 2 — Protect clusters by default and fail over to available workers (Priority: P1)
As an operator, I get a default maximum of two participants for priorities at/above the configured cutoff, can explicitly opt out of the limit, and do not need to manually reassign queued work when a participant fails.
Independent Test: Exercise absent, zero, negative, one, two, and oversized limits; then remove a selected node and refresh membership. Verify the expected participant set and continued queue progress.
Acceptance Scenarios:
- Given an absent maximum, when a six-node cluster indexes capped work, then the default maximum is two; given an explicit zero or negative maximum, then unrestricted participation is restored.
- Given a positive maximum larger than the eligible membership, when participants are selected, then all eligible nodes can participate.
- Given two selected participants and other healthy nodes, when a participant is removed by existing membership detection, then membership refresh selects a replacement without deleting pending journal entries.
- Given a formerly selected node with prefetched work, when its refreshed membership excludes it, then it stops admitting capped work, continues normal-work processing, and its unprocessed capped entries remain recoverable by the selected workers.
- Given membership has converged after a change, when selected workers continue draining the journal, then old cursors or cached ownership do not permanently skip newly assigned entries.
Requirements
Functional Requirements
- FR-001: Add integer configuration properties
MAX_REINDEXING_NODES(default2, container nameDOT_MAX_REINDEXING_NODES) andREINDEX_NODE_LIMIT_MIN_PRIORITY(default200, container nameDOT_REINDEX_NODE_LIMIT_MIN_PRIORITY). An absent maximum uses two workers; an explicit nonpositive maximum disables the node limit. The cutoff accepts nonnegative integers; a negative or malformed cutoff must be reported and fall back to200. A malformed maximum must be reported and fall back to2. Invalid live values must not stop indexing. - FR-002: For a positive maximum, the effective participant count for entries at/above the configured cutoff MUST be the smaller of the maximum and eligible membership size. Membership for entries below the cutoff MUST remain unrestricted. Selection MUST use a consistent deterministic order, preserving existing oldest-server ordering and its server-ID tie-breaker.
- FR-003: All nodes with the same membership and pair of settings MUST derive the same capped-work participant list. A node excluded for capped work MUST NOT append itself back into that limited list. Serving membership and membership for below-cutoff journal work MUST remain intact.
- FR-004: Admission, journal ownership, and stalled-share takeover MUST agree on the effective participants for each configured priority class: all eligible nodes below the cutoff, and the limited selection at/above the cutoff. Neither simply pausing four entire workers nor using six-way capped-work partitions with only two consumers is sufficient. Provider bulk sizing continues to use the full membership, as required by FR-008; worker limiting must not enlarge requests.
- FR-005: Bulk exclusion MUST be checked before admitting capped entries, including entries from local prefetch or mixed-priority batches. Already-started work MAY finish; exclusion MUST NOT delete persisted pending work or prevent normal-work admission.
- FR-006: Membership changes MUST preserve eventual processing of eligible entries under existing retry and acknowledgement semantics. Replacement workers MUST use the existing membership/liveness mechanism, not a separate election or locking service.
- FR-007: The priority-aware limit MUST apply irrespective of the active search provider. It MUST NOT change direct synchronous indexing, explicit request indexing policies, search routing, serving-node membership, normal-work membership, or the search provider migration phase.
- FR-008: Preserve existing bulk request sizing based on the full eligible application membership; the participation cap MUST NOT automatically enlarge requests. With six application nodes and a bulk-action setting of
20, retain the existing count threshold of3, even when only two nodes admit capped work. Limit priority-aware journal ownership/admission rather than shrinking the shared membership list used by provider adapters. Mixed-priority batching MUST NOT allow capped entries to bypass their admission rule. - FR-012: Both settings MUST be read through
Config.getIntProperty("MAX_REINDEXING_NODES", 2)andConfig.getIntProperty("REINDEX_NODE_LIMIT_MIN_PRIORITY", 200)before each new batch. Read once for that batch, not once at class initialization or consumer construction; do not retain static-final/Lazy setting values or introduce a new configuration mechanism. Changes visible through the existing Config resolution must affect the same consumer's next batch and its prefetched work without restart. In-flight work may finish; normal cluster configuration propagation rules still apply. Document that environment overrides outrank runtime system-table values. - FR-009: Documentation MUST explain that this is a worker-admission limit, not a documents-per-second limiter or a strict distributed bound on all outstanding HTTP requests. Existing in-flight work and membership-cache convergence can temporarily overlap during failover or rolling deployment.
- FR-010: Configuration documentation MUST distinguish background participation from
REINDEX_THREAD_CONCURRENT_REQUESTS,REINDEX_THREAD_SLEEP, andREINDEX_THREAD_MINIMUM_RUNTIME_IN_SEC. The latter two do not pace active queue ingestion in the investigated release. - FR-011: Regression tests MUST cover cutoffs
0,100,200, and300; entries below, exactly equal to, and above each cutoff; priorities0,100,101,199,200,201, and300; disabled/oversized node limits; invalid cutoff fallback; and existing error/retry eligibility. Below-cutoff work MUST NOT be blocked behind local prefetch containing only capped entries. At the default cutoff, normal saves and their retries remain unrestricted.
Edge Cases
-
Runtime edits to either setting on an existing consumer, including the unlimited opt-out and changes after prefetch, are checked at the next batch boundary. A setting snapshot is consistent within one batch.
-
One eligible node with a maximum of two: that node remains able to index.
-
No eligible membership from the existing membership source: retain existing safe behavior; never divide by zero or silently delete work.
-
Equal startup times: the server-ID tie-breaker produces identical selection across nodes.
-
Membership replacement while local prefetch is nonempty: relinquished entries remain in the journal and become reachable by the new owners.
-
A bulk-excluded node observes a stalled participant: recovery does not promote that node for capped work; normal-priority recovery remains available.
-
A local prefetch contains structure work followed by normal work: excluded bulk entries cannot stop that node from finding and processing normal work.
-
A batch or identifier has entries in both priority classes: preserve existing acknowledgement, deduplication, and update/delete ordering semantics without bypassing bulk admission.
-
An active full reindex reaches an empty local queue: local exclusion or an empty local share does not falsely indicate global completion.
-
Inconsistent settings during rollout: a strict global maximum is not guaranteed until configuration and membership agree across nodes; document this operational constraint.
Key Entities
- Eligible membership: Application nodes considered available for reindexing by the existing membership source.
- Effective participants: The consistently ordered subset permitted to admit background indexing batches.
- Journal entries: Persisted background update/delete work, which must remain recoverable when participation changes.
Success Criteria
The worker-limit scenarios below use a cutoff of 200. The same admission, failover, and coverage guarantees must also hold with operator-selected cutoffs; in particular, equality with the configured cutoff is capped, and lower numeric priorities are not.
- SC-001: With six eligible nodes, a stable shared membership, and a maximum of two, exactly two nodes admit structure/full-reindex work and all six remain eligible for normal-priority journal work and serving requests.
- SC-002: Regression tests demonstrate complete journal coverage under the two-worker selection and after replacement of a selected worker, without deleting unprocessed work.
- SC-003: Tests prove bulk-excluded nodes cannot process capped prefetched entries or bypass the cap through stalled-share recovery, while still making progress on ordinary saves.
- SC-004: An unset maximum selects at most two workers for capped work; explicit nonpositive maximum values restore unrestricted participation. A maximum above membership size does not reduce availability. A one-node deployment still has one worker.
- SC-006: On the same running consumer, changing the maximum, disabling/re-enabling it, raising/lowering the cutoff, and reducing eligibility after prefetch MUST affect the next batch without reconstructing the consumer or restarting a pod.
- SC-005: Documentation accurately describes both settings, the inclusive configurable cutoff and examples, deployment consistency requirements, failover behavior, and preservation of existing bulk request sizing.
Legacy Considerations
The background queue implementation is active legacy code under com.dotmarketing.common.reindex; membership and provider adapters also consume its participant list. Preserve existing queue acknowledgement, update/delete ordering, retry, provider migration, and full-reindex completion behavior. No database schema or index-mapping migration is required.
The initial use case is six application pods with a maximum of two for structure/full-reindex work; normal queued saves remain distributed across all six. The maximum of two is the new default for every installation, not a tenant-only opt-in. All nodes in one application cluster are expected to use the same maximum and cutoff. The default cutoff preserves ASAP/normal-save eligibility on all nodes; operators can deliberately change that scope by selecting another cutoff.
Existing membership detection and caching remain authoritative. This fix does not introduce distributed leases or promise instantaneous exclusion of a failed or formerly selected worker.
Non-Goals
- Deploying the change or setting a production cluster's maximum.
- Changing the configured bulk-action budget, bulk byte limit, fetch count, or concurrent-request setting automatically; limiting workers must not implicitly enlarge requests.
- Introducing documents-per-second pacing, CPU-adaptive throttling, or a new scheduler.
- Increasing OpenSearch replicas, changing instance sizes, or changing storage settings.
- Fixing bulk-processor close-timeout handling or changing HTTP retry behavior.
- Guaranteeing a particular throughput reduction, reindex completion time, or prevention of every search-capacity incident.
Review and Verification Gates
This issue is the canonical defect specification. This is an outage remediation, not a new-feature approval exercise: follow the urgent-incident exception in docs/core/SPEC_KIT_QUICK_START.md, which permits the fix before the separate defect-spec PR. Spec-only PR #38580 is not a prerequisite for the fix. Developer approval of regression tests and confirmation that they fail for the intended reason remain required before production code changes.
Before production use, compare two-worker queue-drain speed and search CPU/latency under a controlled backlog. Fewer bulk workers can delay structure propagation and full reindex completion; ordinary saves are not deliberately restricted to those workers, though all workloads still share search infrastructure. Bulk request sizing is unchanged, and a worker cap is not itself a documents-per-second ceiling or a guarantee of proportional pressure reduction.
Save Indexing Policy — Existing Behavior and Follow-up
Verified against the running release source:
- Backend defaults are already
INDEX_POLICY_SINGLE_CONTENT=DEFERandINDEX_POLICY_DEPENDENCIES=DEFER; container overrides areDOT_INDEX_POLICY_SINGLE_CONTENTandDOT_INDEX_POLICY_DEPENDENCIES. - Request policy precedence is query parameter
indexPolicy, then the contentlet payload'sindexPolicy, then the configured default. ExplicitWAIT_FOR/FORCErequests are supported and are not silently overridden by this worker-limit fix. - Normal queued saves use priority
100; this fix keeps their processing eligible across all six pods. - Some UI workflow methods (
DotWorkflowActionsFireService.fireToandfireDefaultAction) hardcodeWAIT_FOR. Auditing implicit UI overrides is separate from limiting structure/full-reindex workers; changing those callers requires read-after-write regression coverage. - No new indexing-policy flag is necessary to provide the existing default and per-request override. Runtime system-table overrides and every client save path have not been exhaustively audited.
Related
- Spec-only PR: #38580 (supporting record, not an implementation prerequisite).
- Lingua principale
- Java
- Stelle
- 972
- Fork
- 493
- Merge medio
- 2g 12h
- PR unite (30g)
- 228
Preparare l'ambiente
Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di dotCMS/core
-
dotCMS : UVE OKR : Customer Support Team : Maintenance Type : Defect
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
I maintainer di solito rispondono entro 2 giorni
-
OKR : Customer Support Team : Maintenance Type : Defect
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 2 giorni
-
Opening content from Content Search replaces the "Search" breadcrumb in the new Edit Content screenApertadotCMS: New Edit Contentlet OKR : Customer Support Team : Falcon Type : Defect
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 2 giorni
-
Escape the Binary field hint in the legacy content editor so an apostrophe doesn't break the fieldApertaOKR : Customer Support Team : Maintenance Type : Defect
Difficoltà 2/5 1-3 ore Idoneità per principianti 79/100
I maintainer di solito rispondono entro 2 giorni
-
dotCMS : Permissions OKR : Customer Support Team : Maintenance Type : Defect
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
I maintainer di solito rispondono entro 2 giorni
Issue simili
-
desktop grp:about type:graphics-bug ui-public
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 68/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
I maintainer di solito rispondono entro 2 giorni
-
enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
it-at-m/Wahllokalsystem#3846 ·
I maintainer di solito rispondono entro 2 giorni
-
Area/Connector module/solace Priority/Low Type/Bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
ballerina-platform/ballerina-library#9849 ·
I maintainer di solito rispondono entro 1 giorno