Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

GitLab inventory stops at the first page of projects on python-gitlab >= 4.0.0, and the documented `orgs` key is never read

Open
  • Is something wrong?
#89 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
72/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
backend, cli

Research direction

Start with scraper/gitlab/init.py:43, scraper/code_gov/init.py:73-82, README.md:98, and the python-gitlab requirement in pyproject.toml:34. Reproduce the paged fake-server case, then verify that inventories include all projects and that an orgs-only configuration filters through the groups API without breaking repos or exclude behavior.

Written by the indexing model from the issue text.

Description

On python-gitlab 4.0.0 and later, scraper inventories only the first page of projects (20 at GitLab's default page size) of a GitLab instance configured without repos, and exits 0.

scraper/gitlab/__init__.py:43 calls gl_session.projects.list(as_list=False). python-gitlab removed as_list in 4.0.0 (changelog: "as_list support is removed"), so it goes to the server as a query parameter and list() returns one page. pyproject.toml:34 only requires python-gitlab>=1.6.0, and a fresh install today resolves 8.5.0.

Related: README.md:98 documents "orgs" for GitLab, but the read at scraper/code_gov/__init__.py:73 has been commented out since f9de592 (2018), and :82 never passes it to query_repos. An orgs-only config gets the same unfiltered list as an empty one.

I ran the scraper CLI at b1bfe6e (0.16.0, same as the PyPI release) against a local fake GitLab v4 server that pages 20 at a time:

  • 25 projects, no repos: 20 releases on python-gitlab 8.5.0 and 4.0.0, from a single GET /api/v4/projects?as_list=False. The log says Number of Projects: 20, and the only sign is python-gitlab's UserWarning on stderr ("returned 20 of 25 items"). On 3.15.0 it requests page 2 and gets 25.
  • Three projects in three groups, orgs: ["public-group"]: all three come back, with the same requests as an empty config. Set alongside repos, orgs adds nothing.
  • Controls: repos and exclude each narrow the result to 1 project. Changing line 43 to all=True gives 25 of 25 on all three versions. A patched copy that reads orgs through the groups API returns only that group's 9 of 25.

Fixing the pagination alone would make an orgs config inventory every project /projects returns (25 of 25 in the same harness), so I'd fix the two together. I haven't run this against a real GitLab server, or tried python-gitlab older than 3.15.0.

Happy to open a PR that switches line 43 to all=True and reads orgs via groups.get(org).projects.list(all=True, include_subgroups=True), about 20 lines across the two files, or to just drop orgs from the README if you'd rather not support group filtering.

Dominant language
Python
Stars
59
Forks
28
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from llnl/scraper

All issues in llnl/scraper

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.