Use recipes_map_cols() in bake() methods that transform columns in place
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 58/100
- Issue type
- Refactor
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- r
- Domain
- performance
Research direction
Wait for the recipes release exporting recipes_map_cols(), then inspect the listed bake() methods in R/lencode.R, R/lencode_glm.R, R/lencode_bayes.R, R/lencode_mixed.R, R/collapse_stringdist.R, R/discretize_cart.R, and R/discretize_xgb.R. Compare their per-column assignments with the helper's intended usage and verify that baking produces the same results while avoiding repeated data-frame copies. Also review R/collapse_cart.R separately, since convert_keys() prevents a direct replacement.
Written by the indexing model from the issue text.
Description
recipes is gaining an exported recipes_map_cols() helper (tidymodels/recipes#1543, tidymodels/recipes#1556, tidymodels/recipes#1558) for use in bake() methods of steps that transform columns in place. Assigning one column at a time in a for loop (new_data[[col_name]] <- ...) copies the whole data frame on each iteration, which is quadratic in the number of columns. recipes_map_cols() computes all the new columns first and assigns them in bulk. In recipes, baking 8,000 factor columns through step_other() dropped from 5.14s to 1.26s.
The following bake() methods in embed use the slow pattern and could be converted:
step_lencode()(R/lencode.R:296)step_lencode_glm()(R/lencode_glm.R:209)step_lencode_bayes()(R/lencode_bayes.R:272)step_lencode_mixed()(R/lencode_mixed.R:259)step_collapse_stringdist()(R/collapse_stringdist.R:203)step_discretize_cart()(R/discretize_cart.R:270)step_discretize_xgb()(R/discretize_xgb.R:545)
The two discretize_* steps are the worst of these: they take a full binned_data <- new_data copy and then assign into it once per column, so they pay the copy twice per iteration.
Separately, step_collapse_cart() (R/collapse_cart.R:165) loops per column too, but each iteration calls convert_keys(), which rebuilds the data frame via a join. recipes_map_cols() doesn't drop in directly there, though it is quadratic in its own way and worth a look.
This is blocked on the recipes release that exports recipes_map_cols().
- Dominant language
- R
- Stars
- 146
- Forks
- 23
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from tidymodels/embed
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 90/100
tidymodels/embed#277 ·
-
feature
Difficulty 5/5 Over a week Newbie friendliness 45/100
tidymodels/embed#275 · 5 comments ·
-
feature
Difficulty 5/5 Over a week Newbie friendliness 25/100
tidymodels/embed#274 ·
-
documentation
Difficulty 4/5 3-5 days Newbie friendliness 35/100
tidymodels/embed#256 ·
-
feature
Difficulty 5/5 Over a week Newbie friendliness 30/100
tidymodels/embed#255 ·
All issues in tidymodels/embed
Similar issues
-
documentation pkg infrastructure
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
epiverse-trace/epiparameter#511 ·
-
function:write_dwc
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100