GS: Failure-safe full refresh and large-result transfer

Open
#402 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
google-cloud, typescript
Domain
backend, cloud

Research direction

Read docs/design/google-sheets/full-refresh.md, docs/design/google-sheets/large-results.md, and docs/design/google-sheets/failure-recovery.md, then review prerequisites #399 and #401. Done means implementing the listed refresh, staging, publication, limits, cancellation, recovery, and checkpoint acceptance criteria without truncation or changing committed live data prematurely.

Written by the indexing model from the issue text.

Description

cool feature

Parent: #398
Depends on: #399, #401

Design:

  • docs/design/google-sheets/full-refresh.md
  • docs/design/google-sheets/large-results.md
  • docs/design/google-sheets/failure-recovery.md

Scope

Implement manual full refresh with streaming ClickHouse reads, canonical value conversion, hidden staging, stable-sheet publication, limits, progress, cancellation, and failure recovery.

Acceptance criteria

  • The complete query result is never required in browser memory.
  • Google writes use byte-bounded explicit ranges and RAW semantics.
  • Existing live data remains unchanged until all query and staging work succeeds.
  • Final publication preserves the managed numeric sheet ID and atomically updates state.
  • Old trailing rows/columns are cleared and formatting/metadata are refreshed.
  • Ambiguous publication is reconciled by generation/refresh ID before retry.
  • Row, cell, byte, duration, and spreadsheet-capacity limits are enforced without truncation.
  • Cancellation and closed-tab recovery leave committed data intact and clean orphan staging later.
  • Successful full refresh can initialize/reset an append checkpoint when eligible.

Non-goals

Append page selection, automatic scheduling, or in-place partial replacement when staging cannot fit.

Dominant language
TypeScript
Stars
8
Forks
2
Avg merge
1h 34m
Merged PRs (30d)
6

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Altinity/altinity-sql-browser

All issues in Altinity/altinity-sql-browser

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.