Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Team schema results in excess queries and exceeds API rate limit

Open
  • Is something wrong?
#151 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
32/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
github, python
Domain
api, backend

Research direction

Start in tap_github/init.py at get_all_teams, get_all_team_memberships, and authed_get_all_pages, as identified by the traceback. Reproduce a sync with nested teams and inspect the membership requests; done means team extraction no longer makes thousands of redundant calls or hits the GitHub API rate limit.

Written by the indexing model from the issue text.

Description

@KrisPersonal @dmosorast thanks so much for the fix to unblock using the team schema! I can now get further; however, it does thousands of HTTP calls for "memberships", around 4k in my case (and many calls against team_members), and eventually hits:

API rate limit exceeded, please try after 3106 seconds.

We use nested teams - I'm wondering if that's something that might be causing some sort of unexpected behavior, it seems to be stuck in a loop?

2021-12-01 00:31:45,568Z    tap - CRITICAL API rate limit exceeded, please try after 3106 seconds.
2021-12-01 00:31:45,568Z    tap - Traceback (most recent call last):
2021-12-01 00:31:45,568Z    tap -   File "tap-env/bin/tap-github", line 33, in <module>
2021-12-01 00:31:45,568Z    tap -     sys.exit(load_entry_point('tap-github==1.10.1', 'console_scripts', 'tap-github')())
2021-12-01 00:31:45,568Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 229, in wrapped
2021-12-01 00:31:45,568Z    tap -     return fnc(*args, **kwargs)
2021-12-01 00:31:45,568Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 1100, in main
2021-12-01 00:31:45,568Z    tap -     do_sync(args.config, args.state, catalog)
2021-12-01 00:31:45,568Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 1088, in do_sync
2021-12-01 00:31:45,568Z    tap -     state = sync_func(stream_schemas, repo, state, mdata, start_date)
2021-12-01 00:31:45,569Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 367, in get_all_teams
2021-12-01 00:31:45,569Z    tap -     for team_memberships_rec in get_all_team_memberships(team_slug, schemas['team_memberships'], repo_path, state, mdata):
2021-12-01 00:31:45,569Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 404, in get_all_team_memberships
2021-12-01 00:31:45,569Z    tap -     'https://api.github.com/orgs/{}/teams/{}/memberships/{}'.format(org, team_slug, username)
2021-12-01 00:31:45,569Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 214, in authed_get_all_pages
2021-12-01 00:31:45,569Z    tap -     r = authed_get(source, url, headers)
2021-12-01 00:31:45,569Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 209, in authed_get
2021-12-01 00:31:45,569Z    tap -     rate_throttling(resp)
2021-12-01 00:31:45,569Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_github/__init__.py", line 196, in rate_throttling
2021-12-01 00:31:45,569Z    tap -     raise RateLimitExceeded(message) from None
2021-12-01 00:31:45,569Z    tap - tap_github.RateLimitExceeded: API rate limit exceeded, please try after 3106 seconds.
Dominant language
Python
Stars
76
Forks
84
No merged PRs in 30d

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from singer-io/tap-github

All issues in singer-io/tap-github

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.