Create API-enforced, configurable upper bounds on number of 'waiting' `Task`s on a `Transformation`, number of `Task`s ACTIONed on a `TaskHub`
#134 创建于 2023年4月27日
仓库指标
- 星标
- (35 个星标)
- PR 合并指标
- (PR 指标待抓取)
描述
Currently, there exist no enforced limit on the number of Tasks a user can create for a given Transformation, meaning it is very easy to (accidentally or otherwise) create very large (thousands or more) Tasks on a single Transformation. There is also no enforced limit on the number of such Tasks that can have ACTIONSrelationships to a givenTaskHub`, which can choke the compute services due to the bottleneck of the claiming code (reduced by #121, but not able to scale infinitely).
To address this, we should create server-side configurable limits to:
- the number of
waitingTasks allowed on aTransformationat a time - the number of
ACTIONSrelationships on aTaskHubat a time
User-client side, this will mean that when a user calls AlchemiscaleClient.create_tasks(...), they will get back ScopedKeys for Tasks created until no more can be created for that Transformation; if a Task can't be created due to the limit, this will be reflected in the list of ScopedKeys returned, with a None given in its place.
Likewise, when a user calls AlchemiscaleClient.action_tasks(...), the method will action as many of the Tasks as it can up till the limit, then a None will be given for those Tasks that were not actioned.