apache/pulsar

To optimize parallelism config in config file

Open

#14 281 ouverte le 14 févr. 2022

Voir sur GitHub
 (2 commentaires) (0 réactions) (0 assignés)Java (3 485 forks)batch import
Stalehelp wantedlifecycle/staletype/enhancement

Métriques du dépôt

Stars
 (13 697 stars)
Métriques de merge PR
 (Merge moyen 3j 10h) (146 PRs mergées en 30 j)

Description

Is your enhancement request related to a problem? Please describe. in config file of pulsar io , user can define the parallelism parameter, but there is a problem, for example In Mongodb Source, if this param is set to be 100000, when run in local run mode,100000 threads generated, but apparently this is wrong, how about the config set to be 10000000000? maybe u can say, this is set by user ,but we can build pulsar io stronger as posssible to reduce errors because users do not know more details , and they are not expected to be expert of all Storages.

besides this, in Mongo Source ,the parallelism just shoud be 1, if above 1, the result will be wrong

so, based on the 2 reasons, the parallelism config should be decided finally by the Source Class , just like kafka connect because the author of the Source Class knows concretely how the source runs

Describe the solution you'd like A clear and concise description of what you want to happen.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

Guide contributeur