Alternate exit criteria for DataIterators

Open
#335 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
go, sql
Domain
backend, databases

Research direction

Start by reading the DataIterator and Cursor flow, then inspect CopyFilter.BuildSelect and Ferry's use of dataIterator.Run(). The change should support object-ID batches across property tables and stop only when no object IDs remain, rather than when one query yields zero rows.

Written by the indexing model from the issue text.

Description

For some context, the schema I'm trying to use Ghostferry with is based on EAV. Each table is a property table and an object will typically correspond to multiple records across various tables.

I'm also trying to implement sharding support with an implementation of CopyFilter.

My CopyFilter implementation generates object IDs for a given shard in batches and uses those IDs in BuildSelect to form the base SQL query to copy records, e.g.

SELECT ... FROM ... WHERE pagination_key IN [object_id1...object_id20]

I'm currently running into a problem where this query sometimes yields 0 rows for a property table for a batch of object IDs, leading the Cursor to terminate iteration early even though there are still objects in the shard left to process.

Ideally, the flow I need looks something like:

  1. Generate a batch of object IDs
  2. Copy all records across all property tables corresponding to these object IDs
  3. Terminate DataIterators if no more object IDs remain in shard

I don't believe this is possible without opening a PR against ghostferry, but let me know if I'm missing something. If it isn't possible, do you have any suggestions on how to implement this?

I'm thinking I'll need to have the DataIterator understand these object ID batches and do multiple dataIterator.Run()s for each batch before exiting. I see a few options here:

  • A: Define an interface on DataIterator that supports different termination conditions
  • B: Put the current DataIterator implementation behind an interface so Ferry can support an alternate implementation

Any guidance here would be appreciated - thanks for your time.

Dominant language
Go
Stars
855
Forks
74
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Shopify/ghostferry

All issues in Shopify/ghostferry

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.