Add local resumable search backfill and verification runner
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 25/100
Direzione di ricerca
Start by reading Purchasing.Core/Services/OrderIndexSource.cs, IndexService.cs, and IndexBackend.cs to understand the shared extraction, indexing, and reconciliation behavior; then inspect the referenced job entry points and existing tests. Build a bounded local .NET runner and runbook around those shared APIs, covering resumable ranges, failure handling, all nine indexes, and final verification. Done means focused behavioral tests pass and verification reports readiness only when expected IDs and normalized contents match across the intended destination.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Problem and outcome
Provide a standalone .NET command-line runner that an operator runs locally to populate and verify the complete Purchasing TEST search dataset in Elastic Cloud from live TEST SQL. Reuse the shared projections, mappings, serialization, document IDs, and indexing behavior introduced by #675 / #676.
The runner must support bounded execution, interruption and resume, explicit year/date ranges, and overlapping reruns. Completion means verified document IDs and contents across all nine indexes, not merely a successful bulk request or matching document counts.
Implement this as a separate change from #676. The initial operational plan is for TEST; it does not define or authorize production cutover.
Selected operating model
- Read the same live TEST database used by the TEST application.
- During initial backfill, deployed jobs continue writing Bonsai and the local runner writes the isolated Purchasing Elastic Cloud destination. They must not write concurrently to the same destination.
- Run automatic bounded batches, optionally selecting orders by creation year/date range. Overnight execution is acceptable.
- Use a small local JSON checkpoint to resume a particular run. No SQL tracking table is required.
- Enter a controlled TEST maintenance window for final reconciliation and verification. Pause TEST data edits and indexing, including relevant background mutations/writers, so the expected SQL projections and target remain stable.
- After verification succeeds, report readiness for manual handoff. An operator enables scheduled Elastic Cloud writes and completes any separately selected read-routing change before resuming normal operation.
The runner does not change deployed configuration, disable jobs, pause the application, scale SQL, deploy code, or switch reads automatically. Apply writer separation to order indexing, lookup rebuilding, and AggieEnterprise commodity rebuilding.
Why IDs and ordinary catch-up are insufficient
Stable IDs prevent duplicate identities but do not prevent stale overwrites. For example, the backfill reads quantity 2; a later edit and scheduled job index quantity 3; then the slower backfill writes quantity 2. Last completion wins, not newest source state.
Current child reconciliation refreshes, deletes documents for selected order IDs, then writes replacements. Overlapping writers can delete newer children or resurrect removed children. Consequently, existence checks and create-only bulk operations are not a replacement for the selected writer separation.
The repository schedules order indexing every five minutes with a rolling ten-minute lookback, without a persistent watermark. Order candidates come from OrderTracking.DateCreated; recent comments have a separate pass. A long-running backfill cannot rely on the next ordinary scheduled run to repair its entire duration.
The tracking feed also misses some indexed changes. Paid/received-note edits can update indexed line-item fields without creating OrderTracking, and changes to joined user/workgroup/vendor data introduce further gaps. Replaying tracking since backfill start may reduce remaining work but is not proof of completeness. Final reconciliation must cover the entire intended dataset independently of that feed.
Selection, extraction, and bounded writes
- Support a complete run and optional order-creation year/date filters, with documented boundary and timezone semantics.
- Iterate orders in stable ascending-ID batches rather than loading all orders or projections into memory. Establish a bounded baseline; include orders added after baseline selection during final reconciliation.
- A selected order brings all related projections, including records created in later years. Do not independently filter comments/actions/children by their creation year.
- Reuse the shared extraction and serialization rules. Expose the smallest necessary shared API rather than copying SQL and mappings into the runner.
- Preserve successful-empty versus failed-extraction semantics. A successful empty projection removes stale target documents; an extraction failure preserves existing documents and remains retryable.
- Continue independent work that can succeed, retain failed work, and report failure if required work remains incomplete.
- Never delete documents outside the selected order range or intended Purchasing namespace during a scoped run.
- Do not call the existing full-index
recreateoperation once per year. Each replacement would discard previously loaded years. Range execution must accumulate/reconcile data safely in the selected destination.
The dataset comprises five order projections: history, line items, custom answers, comments, and tracking. It also includes four independent lookup indexes: accounts, buildings, commodities, and vendors.
Resume and repeatability
Overlapping ranges mean sequential executions with overlapping selections, not simultaneous runners writing the same destination. Resume and a new overlapping run have different meanings:
- Resume the same run: skip batches confirmed complete, retry failed/incomplete batches, and continue the remaining selection.
- Start a new range/run: deliberately reread and reconcile the selected orders, even if their documents already exist. Overlap must be safe and repeatable.
Checkpoint requirements:
- Store a small local JSON manifest/checkpoint, saved atomically.
- Bind it to the source database identity, target endpoint/index namespace, selected range, and compatible projection/schema identity. Reject incompatible resumes.
- Do not store credentials, connection strings, or document payloads.
- Record completion only after every required projection and write for that batch succeeds. Track completed/failed lookup-index work as well as order batches so resume cannot overlook unfinished lookups.
- Preserve failures behind the highest processed ID. A single “highest ID seen” value must not skip failed earlier work.
- An interrupted write/checkpoint transition must be safe to replay. Existing documents do not establish batch completion: history may have succeeded while child projections failed.
- Provide useful progress and failure diagnostics without logging sensitive payloads.
The checkpoint describes completed baseline work. It is not a freshness guarantee and must not exempt previously completed batches from final reconciliation.
Final reconciliation, verification, and handoff
During the controlled TEST maintenance window:
- Confirm the intended source and destination and that relevant SQL mutations and target writers are quiescent. Deployed enablement/settings must be inspected; repository schedules alone are insufficient evidence.
- Reconcile the complete intended dataset from current SQL, including inserts, updates, deletes, changed child sets, and lookup data since the initial backfill. Clear stale documents when their expected projection is empty, without interpreting extraction errors as empty results.
- Verify expected IDs and normalized projected contents across all nine indexes. Keep comparison and enumeration of unexpected target documents bounded in memory. Detect missing, stale, and unexpected documents. Use the actual shared projection rules; raw Orders counts or an assumed one-history-document-per-order rule are incorrect because SQL joins can legitimately exclude rows.
- Report readiness only when required extraction, writes, and verification all succeed. Report unresolved failures/mismatches explicitly and exit unsuccessfully if work remains.
- Keep the maintenance state controlled through manual configuration handoff, then resume normal operation. Read routing remains an independent operator decision.
Scope deletion/reconciliation to the selected Purchasing destination. Other applications' indexes must remain untouched. A failed or interrupted final pass must be safely rerunnable and must not announce readiness.
Capacity and source decisions
Live TEST SQL is the selected first source. A BACPAC/local Docker copy is an alternative for later reconsideration, not a prerequisite or initial implementation requirement.
Measure representative SQL extraction batches and indexing throughput before deciding whether to temporarily increase SQL capacity to P4 or higher. Capture enough evidence to distinguish SQL/query, network, and Elasticsearch bottlenecks. Measure the full final reconciliation/verification time before scheduling the maintenance window; it is not assumed to be brief.
The selected destination must be inspected for isolation, capacity, and application-key permissions before a live run. No particular existing deployment is reserved by this issue. Use external configuration for credentials and endpoints.
Implementation and a runbook are in scope. Live scaling, destination/configuration changes, job disabling, maintenance, and bulk backfill execution require an explicit operational go-ahead; this issue is not that authorization.
Acceptance criteria and focused verification
- A local .NET runner reuses the shared indexing implementation and supports a complete dataset plus optional year/date selection.
- Selection/extraction/writes remain bounded for a large dataset.
- Overlapping explicit runs are repeatable and retain previously loaded unrelated years.
- Resume survives interruption and never skips earlier failures because later batches succeeded.
- Checkpoint identity mismatches are rejected and checkpoints contain no secrets.
- All related records for selected orders are included regardless of child creation dates.
- Successful empty projections remove stale documents; failed SQL projections preserve them and remain retryable.
- Final reconciliation includes changes/new orders after baseline completion and covers all nine indexes.
- Verification compares expected IDs and normalized contents, detects extra documents, and cannot report readiness with unresolved failures.
- Reconciliation/deletion stays within the intended destination and selection.
- A runbook covers configuration, initial writer separation, benchmark procedure, interruption/resume, TEST maintenance, failure recovery, final verification, and manual handoff.
- Focused behavioral tests cover these failure and resume paths. Avoid tests that merely grep source text. Report exactly which tests and live scenarios ran; do not automatically restart the full validation pipeline for this follow-up.
Out of scope
- Unrestricted simultaneous writers to the same destination, distributed locks, fencing, or cross-process coordination.
- A comprehensive change-feed/CDC redesign or durable recovery for deployed jobs.
- SQL tracking tables for the local runner.
- Automatic deployment, scaling, maintenance, configuration changes, or read cutover.
- Production maintenance/cutover policy. The agreed TEST pause must not be generalized to production.
- Timestamp display accuracy, tracked separately in #677. Preserve the restored date displays.
Implementation references
Base this separate runner change on the shared implementation from #676, or on master after that PR merges. Do not expand #676 with runner commits.
- Search backend configuration and limits
Purchasing.Core/Services/OrderIndexSource.cs: batched SQL projections and extraction failures.Purchasing.Core/Services/IndexService.cs: batching, lookup SQL, and existing full-index operations.Purchasing.Core/Services/IndexBackend.cs: identities, mappings, serialization, reconciliation, and alias replacement.Purchasing.Core/Services/SearchBackendConfiguration.cs: backend selection/settings.Purchasing.Jobs.UpdateOrderIndexes,Purchasing.Jobs.UpdateLookupIndexes, andPurchasing.Jobs.AggieEnterprise: deployed writer inventory.Purchasing.Tests/ServiceTests/DualIndexingTests.csandSearchBackendIntegrationTests.cs: existing behavioral helpers and integration patterns.PurchasingTP.AzureDb/dbo/Tables/Orders.sql,OrderTracking.sql, andFunctions/udf_GetOrderHistoryForOrderIds.sql: source identities, timestamps, and projection eligibility.
Related: #675, #676, #677.
- Lingua principale
- C#
- Stelle
- 7
- Fork
- 5
- Merge medio
- 9h 3m
- PR unite (30g)
- 5
Preparare l'ambiente
Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Nessuna guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di ucdavis/Purchasing
-
remove elmahAperta
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 68/100
ucdavis/Purchasing#436 ·
-
Refactor configsForse già presa @jSylvestre l’ha presa 10 giorni fa. Aperta
ucdavis/Purchasing#683 · 1 assegnatario ·
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 25/100
ucdavis/Purchasing#677 ·
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 15/100
ucdavis/Purchasing#668 ·
-
AE API field length issueForse di nuovo libera @jSylvestre l’ha presa 39 giorni fa e non c’è nessuna pull request aperta. Aperta
ucdavis/Purchasing#665 · 1 assegnatario ·
Tutte le issue di ucdavis/Purchasing
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
I maintainer di solito rispondono entro 1 giorno
-
Bug IronPython Python 3
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
pyrevitlabs/pyRevit#4184 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
yusei36/KeePassPasskey#66 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
🐛 bug 🕒 low effort good first issue
Difficoltà 2/5 1-3 ore Idoneità per principianti 85/100
guibranco/Sankhya-SDK-dotnet#654 ·
I maintainer di solito rispondono entro 1 giorno
-
巫女のセリフに声を付けるApertaassets claude-proposal
Difficoltà 2/5 1-3 ore Idoneità per principianti 64/100
Fialuxe/MouthOfGod_System#358 ·
I maintainer di solito rispondono entro 1 giorno