MemoryError with IV2SLS
維護者通常 1 天內回覆
還沒有人認領這個 Issue。
評估
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 新手友好度
- 35/100
- Issue 類型
- 缺陷
- 描述清晰度
- 基本清楚
- 活躍度
- 停滯
- 技術堆疊
- numpy, python
- 領域
- performance
研究方向
Start at the IV2SLS entry point and inspect the weight handling around self._wz = self._z * w, comparing the weights=None path with explicitly supplied float32 weights. The reported failure is the allocation of a float64 array for a large design matrix; done means explaining or addressing that excessive memory use so the described regression can run.
由索引模型根據 Issue 內容生成。
描述
I'm trying to run a 2SLS to estimate price elasticity with IV2SLS. This is what my data looks like:
| ln_q | ln_p | .... weather variables ... | ... instruments ... |... user id dummies ...|
all data is np.float32. My data array is approx. (200000, 20000) which is about 16GB.
Using linearmodels IV2SLS I set up my model like:
dependent = ln_q
endog = weather variables + user id dummies
exog = ln_p
instruments = instruments
results = IV2SLS(dependent, endog, exog, instruments).fit()
When running with the full dataset I consistently get the error:
Unable to allocate 27.8GiB of memory to an array with shape (202507, 18450) and data type float64 and it looks like this line is the culprit:
self._wz = self._z * w
which is where weights are assigned.
I'm running 64-bit python on a machine with 128 GB of RAM. I've tried to circumvent this issue by passing my own weights:
results = IV2SLS(dependent, endog, exog, instruments, weights=np.ones(dependent.shape, dtype=np.float32)).fit()
but still get the same MemoryError even when I explicitly pass my own weights of data type float32.
32 GB of RAM usage just to create an array of 1s when weights = None seems like an awful lot of memory usage to essentially keep the input values unchanged. Further, why is it getting recast to float64, when all my other data is of data type float32 and I explicitly pass weights of datatype float32?
Why is an array of ~16GB using >100GB of RAM in this process? What can I do to get this regression to run?
- 主要語言
- Python
- 星號
- 1.1k
- 分支
- 203
- 平均合併
- 3 小時 26 分鐘
- 30 天內合併 PR
- 115
環境準備
這個專案沒有提供開發容器、Dockerfile 或貢獻指南，環境需要你自己搭建：先看它的 README，通用步驟見我們的新手貢獻指南。
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
bashtage/linearmodels 的其他 Issue
-
enhancement
難度 5/5 一週以上 新手友好度 15/100
bashtage/linearmodels#832 ·
維護者通常 1 天內回覆
-
bug
難度 4/5 3-5 天 新手友好度 25/100
bashtage/linearmodels#831 · 1 則留言 ·
維護者通常 1 天內回覆
-
IV: LIML kappa with an endogenous regressor spanned by the instruments, kappa's unit dependence, scale-dependent instrument rank check可能已有人在做 關聯的 PR 仍在進行中或已合併。 未關閉bug
難度 4/5 3-5 天 新手友好度 25/100
bashtage/linearmodels#823 ·
維護者通常 1 天內回覆
-
enhancement
難度 5/5 一週以上 新手友好度 18/100
bashtage/linearmodels#817 ·
維護者通常 1 天內回覆
-
bug
難度 4/5 3-5 天 新手友好度 48/100
bashtage/linearmodels#780 ·
維護者通常 1 天內回覆
查看 bashtage/linearmodels 的全部 Issue
相似的 Issue
-
PluginProvider.storeInstance races and clobbers the dedup cache for concurrent loads可能已有人在做 @Wilson0511 今天認領。 未關閉needs-triage
難度 2/5 1-3 小時 新手友好度 62/100
維護者通常 2 天內回覆
-
community documentation good first issue
難度 2/5 1-3 小時 新手友好度 75/100
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 78/100
-
bug(sdk): add_toleration_json silently drops camelCase tolerationSeconds for static dict/list input可能已有人在做 @ravikumar266 今天認領。 未關閉area/sdk kind/bug
難度 2/5 1-3 小時 新手友好度 75/100
kubeflow/pipelines#15347 · 3 則留言 ·
維護者通常 1 天內回覆
-
predict_file with crop_model duplicates predictions once per image可能已有人在做 關聯的 PR 仍在進行中或已合併。 未關閉
難度 2/5 1-3 小時 新手友好度 62/100
weecology/DeepForest#1445 ·