NicolasHug/Surprise

measures param for grid search and cross_validate should support callables

Open

#166 创建于 2018年4月6日

在 GitHub 查看
 (0 评论) (0 反应) (0 负责人)Python (6,098 star) (1,043 fork)batch import
enhancementgood first issue

描述

For now only strings are accepted as the measures parameter in GridSearchCV, RandomizedSearchCV, and cross_validate. It's thus impossible to use those with measures that take specific parameters as input (e.g. #156 ), or to use custom measures.

We should then accept callables in addition to strings.

Each callable should only take the predictions parameter. In order to handle measures with mulptile parameters, we could implement a make_measure helper (much like sklearn make_scorer) which would simply perform some partial application on the parameters, and set a greater_is_better parameter (this would clean a bit the current code).

Any other option is welcome

贡献者指南

技术栈
python
领域
datamachine learning
议题类型
feature
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
3
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
1-2 days
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
stale
清晰度议题是否清楚说明期望改动、验收标准和下一步。
mostly clear
前置要求
PythonFamiliarity with Surprise libraryUnderstanding of callables and partial application
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
60
研究方向
Investigate the current implementation of GridSearchCV, RandomizedSearchCV, and cross validate in the Surprise source code, likely in grid search.py and evaluate.py. Understand how the 'measures' parameter is validated and processed. Then implement support for callables by creating a make measure helper function similar to sklearn's make scorer, which accepts a callable with only 'predictions' parameter and sets a 'greater is better' attribute. Update the classes to accept both strings and callables, and consider extending the helper to handle partial application for measures with multiple parameters.