Performance suggestion: do not run unselected plugins/checks
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 35/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 停滞
- 技术栈
- python
- 领域
- performance, tooling
调研方向
从 src/flake8/checker.py 开始，重点查看 FileChecker 以及 issue 中展示的插件执行路径，并追踪检查运行后如何应用 --select。比较有无 --select 时的详细执行情况；当未选中的检查或插件被跳过，同时选中的诊断结果仍然正确且运行时间得到改善时，即表示完成。
由索引模型根据 Issue 内容生成。
描述
In GitLab by @hugovk on Jun 5, 2020, 01:45
Please read this brief portion of documentation before going any further: http://flake8.pycqa.org/en/latest/internal/contributing.html#filing-a-bug
Please describe how you installed Flake8
$ pip install -U flake8
$ brew install flake8
# etc.
Please provide the exact, unmodified output of flake8 --bug-report
{
"dependencies": [],
"platform": {
"python_implementation": "CPython",
"python_version": "3.8.3",
"system": "Darwin"
},
"plugins": [
{
"is_local": false,
"plugin": "flake8_2020",
"version": "1.6.0"
},
{
"is_local": false,
"plugin": "mccabe",
"version": "0.6.1"
},
{
"is_local": false,
"plugin": "pycodestyle",
"version": "2.6.0"
},
{
"is_local": false,
"plugin": "pyflakes",
"version": "2.2.0"
}
],
"version": "3.8.2"
}
Please describe the problem or feature
I noticed that Flake8 takes the same time to run with --select as without. As shown using -vv verbosity, it runs all the plugins and checks regardless of --select, and only reports the selected ones afterwards.
Flake8 can sometimes take a long time to run on large codebases, and if it was possible to only run the selected checks, that would save a lot of time, CPU and power.
Would it be possible to only run selected checks/plugins? Rather than running them anyway and discarding that work when reporting?
Docs
For reference, my emphasis.
flake8 --help says --select is for which ones to enable:
--select errors Comma-separated list of errors and warnings to enable. For example, ``--select=E4,E51,W234``.
(Default: ['E', 'F', 'W', 'C90'])
The docs are a bit more explicit:
Specify the list of error codes you wish Flake8 to report.
https://flake8.pycqa.org/en/latest/user/options.html#cmdoption-flake8-select
Example
An example running on the TensorFlow codebase:
$ time flake8
...
flake8 323.91s user 4.31s system 98% cpu 5:32.78 total
$ time flake8 --select YTT
...
flake8 --select YTT 318.62s user 3.80s system 99% cpu 5:25.51 total
Both about the same, around 5m20s.
With an ugly hack (I know this mixes plugin names with error codes, but it's just to get a rough idea, and there's other places to skip too):
diff --git a/src/flake8/checker.py b/src/flake8/checker.py
index d993cb9..9ed986d 100644
--- a/src/flake8/checker.py
+++ b/src/flake8/checker.py
@@ -486,6 +486,8 @@ class FileChecker(object):
return
for plugin in self.checks["ast_plugins"]:
+ if plugin["name"] != "YTT":
+ continue
checker = self.run_check(plugin, tree=ast)
# If the plugin uses a class, call the run method of it, otherwise
# the call should return something iterable itself
$ time flake8 --select YTT
flake8 --select YTT 276.90s user 3.17s system 98% cpu 4:43.00 total
About 4m30s, nearly a minute and ~13% faster.
- 主要语言
- Python
- 星标
- 3.8k
- 派生
- 356
- 平均合并
- 3 小时 25 分钟
- 30 天内合并 PR
- 5
环境准备
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
PyCQA/flake8 的其他 Issue
-
难度 3/5 1-2 天 新手友好度 76/100
-
难度 3/5 1-2 天 新手友好度 45/100
-
难度 3/5 1-2 天 新手友好度 35/100
-
难度 3/5 1-2 天 新手友好度 42/100
-
难度 3/5 1-2 天 新手友好度 35/100
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 72/100
letsencrypt/cp-cps#353 ·
-
难度 2/5 1-3 小时 新手友好度 68/100
-
难度 2/5 1-3 小时 新手友好度 84/100
PedestrianDynamics/pyFDS-Evac#394 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
DOI-USGS/pywatershed#421 ·
-
难度 2/5 1-3 小时 新手友好度 78/100
python-pillow/Pillow#10087 · 1 条评论 ·
维护者通常 1 天内回复