Forecasting incrementally with patsy

未关闭
#126 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
停滞
技术栈
pandas, python
领域
data

调研方向

从 dmatrix 和 incr_dbuilder 入口点开始,重现所提供的不平衡面板示例。调查增量评估如何处理跨 chunk 的依赖关系,并确定行选择如何保留滞后行为和面板行为。当针对预测场景定义并演示了所请求的仅行评估时,即视为完成。

由索引模型根据 Issue 内容生成。

描述

This is either a feature request or a request for help with current functionality. I am doing some work with unbalanced panel data work that involves using patsy to forecast some series. Here's a basic example:

import io, pandas, patsy

#raw panel data indexed on ID, YEAR. Y is the forecast variable of interest. There are no gaps in the data for an individual entity but the panel is potentially unbalanced (meaning different start/end dates).
data = '''ID,YEAR,Y,B,C,D
1,1999,0,2,3,4
1,2000,.,2,3,4
1,2001,.,2,3,4
1,2002,.,2,3,4
2,1996,1,2,3,4
2,1997,.,2,3,4
3,1998,3,2,3,4
3,1999,3,2,3,4
3,2000,.,2,3,4
3,2001,3,2,3,4
'''
data = io.StringIO(data)
df = pandas.read_csv(data, index_col=['ID','YEAR'], na_values=['.'])
print(df)

def lag(series, n=1):
    return series.groupby(level=0).shift(n)

formula = '1+lag(Y)+B+C+D' #This is the forecast equation for Y
x = patsy.dmatrix(formula,df, return_type='dataframe')
params = pandas.Series([1,2,3,4,5], index=x.columns) #these are the coefficients on the forecast vars

#Now forecast year by year
for yr in range(1997,2010):
    ind = df.index.get_level_values('YEAR')==yr
    x = patsy.dmatrix(formula,df, return_type='dataframe').reindex(df.index) 
    x = x.loc[ind]
    df.loc[ind, 'Y'] = df.loc[ind, 'Y'].fillna(x@params)
    print('================')
    print(yr)
    print(df)

Note that to produce the entire forecast we need to call dmatrix over and over. The problem that I'm having is that it is quite inefficient to have to call dmatrix on the entire DataFrame repeatedly, but because the forecast formula can contain arbitrary numbers of lags I can't just pass in a df filtered to the current year (or a set number of lags from the current year). What would be ideal is if I could replace

    ind = df.index.get_level_values('YEAR')==yr
    x = patsy.dmatrix(formula,df, return_type='dataframe').reindex(df.index) 
    x = x.loc[ind]

with version of dmatrix that takes a boolean rows and only evaluates and returns the rows that are needed

    ind = df.index.get_level_values('YEAR')==yr
    x = patsy.dmatrix(formula,df, return_type='dataframe', rows=ind) #evaluates only on rows there ind==True and returns a dataframe with only those rows

I thought incr_dbuilder might be able to handle this, but it seems that it expects each chunk returned is completely separate from previous chunks. That won't work in the time series/panel context.

主要语言
Python
星标
990
派生
113
平均合并
2 天 8 小时
30 天内合并 PR
3

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南,环境需要你自己搭建:先看它的 README,通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

pydata/patsy 的其他 Issue

查看 pydata/patsy 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。