apache/pulsar

To optimize parallelism config in config file

Open

#14.281 aberto em 14 de fev. de 2022

Ver no GitHub
 (2 comments) (0 reactions) (0 assignees)Java (3.485 forks)batch import
Stalehelp wantedlifecycle/staletype/enhancement

Métricas do repositório

Stars
 (13.697 stars)
Métricas de merge de PR
 (Mesclagem média 3d 10h) (146 fundiu PRs em 30d)

Description

Is your enhancement request related to a problem? Please describe. in config file of pulsar io , user can define the parallelism parameter, but there is a problem, for example In Mongodb Source, if this param is set to be 100000, when run in local run mode,100000 threads generated, but apparently this is wrong, how about the config set to be 10000000000? maybe u can say, this is set by user ,but we can build pulsar io stronger as posssible to reduce errors because users do not know more details , and they are not expected to be expert of all Storages.

besides this, in Mongo Source ,the parallelism just shoud be 1, if above 1, the result will be wrong

so, based on the 2 reasons, the parallelism config should be decided finally by the Source Class , just like kafka connect because the author of the Source Class knows concretely how the source runs

Describe the solution you'd like A clear and concise description of what you want to happen.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

Guia do colaborador