Problem: can't discover stream identifiers
Nobody has claimed this yet.
Assessment
This issue has not been assessed yet.
Description
Separate comment to hide and add to specs if appropriate
I also have a request that I don't know how to put in the specs properly, and I need a lot of input from the data team (@joequant, should I tag anyone else?).
The issue is that we explicitly need a way to discover and share stream identifiers across our teams.
I see that
truflation-ingestorsdefine a property callednamefor eachExportDetails. Is that the identifier I'm expecting?If so, it's hard to follow the output from the code; there's no centralized list (that I know of) of all the pipelines available, then it's hard to agree on an identifier for each.
While previously ingesting from the DB, we've used the table names as identifiers for primitive streams (which seems reasonable). For transformed data, in our case, composed streams (e.g., Transport), we've been using a hash of category names, which are very arbitrary, and no one says it won't change a space or a comma.
As we're targeting building a new destination and eventually migrating to use this destination, having mappable identifiers from the current system is a must for a smooth migration process. It would help us detect drifts much more easily.
How do you (the data team) think this could be handled? Could we commit a list to any of these repos mapping all output exports and maybe the pipeline name or localization that originated them?
Could this problem be incorporated into this goal to start the identifiers correctly already?
Originally posted by @outerlook in #493
- Dominant language
- Go
- Stars
- 7
- Forks
- 3
- Avg merge
- 3h 2m
- Merged PRs (30d)
- 13
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from trufnetwork/node
-
type: goal
Difficulty 5/5 Over a week Newbie friendliness 35/100
trufnetwork/node#1436 ·
-
trufnetwork/node#1435 · 1 assignee ·
-
trufnetwork/node#1429 · 2 comments · 1 assignee ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
trufnetwork/node#1313 · 2 comments · 1 reaction ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
trufnetwork/node#1200 · 1 comment ·
All issues in trufnetwork/node
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 60/100
github/gh-aw-mcpg#13748 ·
-
agentic-workflows
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
googleapis/librarian#7670 · 2 comments ·