Goal: Facilitate Pipeline Creation
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- data-engineering
Research direction
Start by reading the linked reconciliation.py, provider.py, and transformers.py examples to understand the current adapter patterns. Then define the generic pipeline, data-source, reconciliation, and transformer abstractions and determine the strategy-specific methods they require. Done would require an agreed design, implementation, and comprehensive tests for the supported ingestion strategies.
Written by the indexing model from the issue text.
Description
There are some repetitive parts of a general pipeline which could be further improved.
Right now, we provide building blocks for different ingestion structures. But we could further improve it with the correct abstractions on the pipeline execution. Examples:
- Data reconciliation strategies
- By last inserted (example)
- Full compare
- Gap filling
- By key
- ...
- Data Source abstract class
- list_all
- get_all
- get_by_date
- get_by_key (example: https://github.com/trufnetwork/adapters/blob/main/src/tsn_adapters/tasks/argentina/provider.py#L23 but a lot to improve)
- Transformer abstract class to prepare for TN (example)
and it could be well structured to require only the correct methods depending on the strategies. The intention is to create a well-tested generic structure that makes it easier to include new adapters
- Dominant language
- Python
- Stars
- 0
- Forks
- 3
- Avg merge
- 2h 9m
- Merged PRs (30d)
- 1
Getting set up
We have not checked this project's setup files yet. Start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from trufnetwork/adapters
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
trufnetwork/adapters#213 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 52/100
trufnetwork/adapters#205 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
trufnetwork/adapters#202 ·
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
trufnetwork/adapters#151 · 1 comment ·
-
Difficulty 4/5 3-5 days Newbie friendliness 20/100
trufnetwork/adapters#72 ·
All issues in trufnetwork/adapters
Similar issues
-
bug status/needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
prowler-cloud/prowler#12887 · 1 comment ·
Maintainers usually reply within 1 day
-
area: desktop platform: macos priority: p3 status: ready type: enhancement
Difficulty 1/5 Under an hour Newbie friendliness 92/100
use-agent-os/agent-os#3484 ·
Maintainers usually reply within 2 days
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
open-telemetry/opentelemetry-python-contrib#5113 · 2 comments · 2 reactions ·
Maintainers usually reply within 1 day
-
external
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
langchain-ai/docs#6255 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day