pandas: mean imputation corrupts Argentina's GDP per capita in the later plot

未关闭 适合新手
  • 哪里不对？
#658 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
82/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
pandas, python
领域
data, documentation

调研方向

从lectures/pandas.md中缺失值的填补及后续的人均GDP计算开始，然后使用QuantEcon/data-lectures中的test_pwt.csv重现图表。决定是保留损坏的值，作为关于静默失败的教训，还是更改被填补的数据。完成标准是：讲义不再将阿根廷错误的人均GDP呈现为有效值，并且图表或说明能清楚展示结果。

由索引模型根据 Issue 内容生成。

描述

In lectures/pandas.md, the missing value imputation step silently corrupts the GDP per capita figures that the lecture plots later.

What happens

  1. "Let's randomly insert some NaN values" sets df.iloc[idx] = np.nan for idx in zip([0, 3, 5, 6], [3, 4, 6, 2]). The first pair is Argentina's POP.
  2. df = df.fillna(df.iloc[:,2:8].mean()) replaces it with the mean population of the other countries, about 196 million, against Argentina's true 37 million. India and the United States pull the mean far up.
  3. "Standardization and Visualization" then computes GDP percap = total GDP * 1e6 / population from this df and plots it.

Argentina's GDP per capita comes out at about 1,500 instead of 7,900, so the sorted bar chart shows Argentina as poorer than India. Nothing in the output signals that anything went wrong.

country true in lecture
Argentina 7,903 1,504
all other countries unchanged unchanged

(Reproduced against test_pwt.csv in QuantEcon/data-lectures, following the lecture's code.)

The other imputed cells (Israel XRAT, South Africa cc, US year) are not used in the GDP per capita calculation, so only Argentina is affected. Imputing year with a column mean is also odd, but harmless here.

Possible fixes

  • Insert the NaNs only in columns not used later (for example cc and cg), or reload the data before "Standardization and Visualization".
  • Alternatively, keep the bug and turn it into a lesson: point out that the code runs without complaint but gives a wrong answer, because imputation is a modeling choice whose errors flow through to everything computed from it. The course version of this lecture does this in a short "silent failures" section: https://jstac.github.io/py_for_ds_public/working_with_data.html

cc @mmcky

主要语言
JavaScript
星标
73
派生
34
平均合并
28 分钟
30 天内合并 PR
5

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

QuantEcon/lecture-python-programming 的其他 Issue

查看 QuantEcon/lecture-python-programming 的全部 Issue

相似的 Issue

更多 JavaScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。