reduce padding overhead when using buckets
#5,011 opened on 2023幎3æ6æ¥
Repository metrics
- Stars
- Â (29,107 stars)
- PR merge metrics
-  (30d ã« merged PR ã¯ãããŸãã)
説æ
ð Feature Request
The current get_buckets function ensures that you have the same number of samples in each bucket. This can result in many unnecessary padding. The feature would enable the user to set an optimal bucket size to reduce padding.
Motivation
padding = unrequited work -> bad performance. While finding the optimal bucket can take 1-5 minutes it can reduce training time by 10-50%.
Pitch
Add support for using Integer programming to set the optimal bucketing size. The implementation I had in mind depends on pulp. I can share the code for optimal bucketing.
Alternatives
k-means algorithm can give a suboptimal but faster approximation