Problem: can't discover stream identifiers

Separate comment to hide and add to specs if appropriate

I also have a request that I don't know how to put in the specs properly, and I need a lot of input from the data team (@joequant, should I tag anyone else?).

The issue is that we explicitly need a way to discover and share stream identifiers across our teams.

I see that truflation-ingestors define a property called name for each ExportDetails. Is that the identifier I'm expecting?

If so, it's hard to follow the output from the code; there's no centralized list (that I know of) of all the pipelines available, then it's hard to agree on an identifier for each.

While previously ingesting from the DB, we've used the table names as identifiers for primitive streams (which seems reasonable). For transformed data, in our case, composed streams (e.g., Transport), we've been using a hash of category names, which are very arbitrary, and no one says it won't change a space or a comma.

As we're targeting building a new destination and eventually migrating to use this destination, having mappable identifiers from the current system is a must for a smooth migration process. It would help us detect drifts much more easily.

How do you (the data team) think this could be handled? Could we commit a list to any of these repos mapping all output exports and maybe the pipeline name or localization that originated them?

Could this problem be incorporated into this goal to start the identifiers correctly already?

Originally posted by @outerlook in #493