Create API-enforced, configurable upper bounds on number of 'waiting' `Task`s on a `Transformation`, number of `Task`s ACTIONed on a `TaskHub`
#134 geöffnet am 27.04.2023
Repository-Metriken
- Stars
- (35 Sterne)
- PR-Merge-Metriken
- (PR-Metriken ausstehend)
Beschreibung
Currently, there exist no enforced limit on the number of Tasks a user can create for a given Transformation, meaning it is very easy to (accidentally or otherwise) create very large (thousands or more) Tasks on a single Transformation. There is also no enforced limit on the number of such Tasks that can have ACTIONSrelationships to a givenTaskHub`, which can choke the compute services due to the bottleneck of the claiming code (reduced by #121, but not able to scale infinitely).
To address this, we should create server-side configurable limits to:
- the number of
waitingTasks allowed on aTransformationat a time - the number of
ACTIONSrelationships on aTaskHubat a time
User-client side, this will mean that when a user calls AlchemiscaleClient.create_tasks(...), they will get back ScopedKeys for Tasks created until no more can be created for that Transformation; if a Task can't be created due to the limit, this will be reflected in the list of ScopedKeys returned, with a None given in its place.
Likewise, when a user calls AlchemiscaleClient.action_tasks(...), the method will action as many of the Tasks as it can up till the limit, then a None will be given for those Tasks that were not actioned.