Team schema results in excess queries and exceeds API rate limit
还没有人认领这个 Issue。
评估
调研方向
Start in tap_github/init.py at get_all_teams, get_all_team_memberships, and authed_get_all_pages, as identified by the traceback. Reproduce a sync with nested teams and inspect the membership requests; done means team extraction no longer makes thousands of redundant calls or hits the GitHub API rate limit.
由索引模型根据 Issue 内容生成。
描述
@KrisPersonal @dmosorast thanks so much for the fix to unblock using the team schema! I can now get further; however, it does thousands of HTTP calls for "memberships", around 4k in my case (and many calls against team_members), and eventually hits:
API rate limit exceeded, please try after 3106 seconds.
We use nested teams - I'm wondering if that's something that might be causing some sort of unexpected behavior, it seems to be stuck in a loop?
2021-12-01 00:31:45,568Z tap - CRITICAL API rate limit exceeded, please try after 3106 seconds.
2021-12-01 00:31:45,568Z tap - Traceback (most recent call last):
2021-12-01 00:31:45,568Z tap - File "tap-env/bin/tap-github", line 33, in <module>
2021-12-01 00:31:45,568Z tap - sys.exit(load_entry_point('tap-github==1.10.1', 'console_scripts', 'tap-github')())
2021-12-01 00:31:45,568Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 229, in wrapped
2021-12-01 00:31:45,568Z tap - return fnc(*args, **kwargs)
2021-12-01 00:31:45,568Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 1100, in main
2021-12-01 00:31:45,568Z tap - do_sync(args.config, args.state, catalog)
2021-12-01 00:31:45,568Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 1088, in do_sync
2021-12-01 00:31:45,568Z tap - state = sync_func(stream_schemas, repo, state, mdata, start_date)
2021-12-01 00:31:45,569Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 367, in get_all_teams
2021-12-01 00:31:45,569Z tap - for team_memberships_rec in get_all_team_memberships(team_slug, schemas['team_memberships'], repo_path, state, mdata):
2021-12-01 00:31:45,569Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 404, in get_all_team_memberships
2021-12-01 00:31:45,569Z tap - 'https://api.github.com/orgs/{}/teams/{}/memberships/{}'.format(org, team_slug, username)
2021-12-01 00:31:45,569Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 214, in authed_get_all_pages
2021-12-01 00:31:45,569Z tap - r = authed_get(source, url, headers)
2021-12-01 00:31:45,569Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 209, in authed_get
2021-12-01 00:31:45,569Z tap - rate_throttling(resp)
2021-12-01 00:31:45,569Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 196, in rate_throttling
2021-12-01 00:31:45,569Z tap - raise RateLimitExceeded(message) from None
2021-12-01 00:31:45,569Z tap - tap_github.RateLimitExceeded: API rate limit exceeded, please try after 3106 seconds.
- 主要语言
- Python
- 星标
- 76
- 派生
- 84
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 没有贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
singer-io/tap-github 的其他 Issue
-
Stitch - Projects (classic) has been deprecated in favor of the new Projects experience可能已有人在做 @skuttleman 于 16 天前认领。 未关闭
难度 3/5 1-2 天 新手友好度 38/100
singer-io/tap-github#215 · 已指派 2 人 ·
-
Re-add `files` and `stats` to `PrCommits` stream可能重新可做 关联的 PR 已关闭且未合并。 未关闭
难度 2/5 1-3 小时 新手友好度 48/100
singer-io/tap-github#202 ·
-
难度 4/5 3-5 天 新手友好度 30/100
singer-io/tap-github#194 · 4 条评论 ·
-
难度 4/5 3-5 天 新手友好度 20/100
singer-io/tap-github#186 · 1 条评论 · 1 个 reaction ·
-
难度 4/5 3-5 天 新手友好度 38/100
singer-io/tap-github#185 ·
查看 singer-io/tap-github 的全部 Issue
相似的 Issue
-
changelog investigate
难度 2/5 1-3 小时 新手友好度 62/100
ramnes/notion-sdk-py#409 ·
-
good first issue help wanted
难度 2/5 1-3 小时 新手友好度 72/100
lindicaphxag-tech/kaggle#28 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 62/100
BSData/horus-heresy-3rd-edition#3211 ·
维护者通常 1 天内回复
-
bug needs-triage
难度 2/5 1-3 小时 新手友好度 70/100
维护者通常 1 天内回复
-
bug tests
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 1 天内回复