Fix or rewrite scraper engine and scrapers
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Domain
- data-engineering
Research direction
Start by locating the scraper engine, master scraper, individual scrapers, and manifest handling mentioned in the issue, then inspect how sources, screenshots, and file extensions are selected. Done means the listed Demozoo and engine bugs are fixed, other scrapers are checked, and a basic test suite covers the expected behavior.
Written by the indexing model from the issue text.
Description
Many years have passed since its first release.
Known demozoo bug:
- it always select the first secondary mirror assuming that is from scene.org. This is wrong and should be carefully checked
- demozoo is missing the screenshots, it always downloads only the first one
Known scrape engine bugs:
- sometimes it sets "screenshots" to None: it should set that field to empty list if no screenshot detected
- it doesn't handle very well other extensions than gb, therefore, it may be a good idea to fix this thing, sometimes in the manifest there is "gbc" but the engine has already renamed the file in gb
Other improvements:
- change the general logic to become more flexible (e.g. select best source, include other extensions)
- write a basic test suite
- test from scratch other scrapers, since they may have become buggy due to change in the master scraper
- Dominant language
- C
- Stars
- 95
- Forks
- 52
- Avg merge
- 7h 33m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from gbdev/database
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 1/5 Under an hour Newbie friendliness 35/100
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 30/100
-
GBDK Gallery Opensource
Difficulty 3/5 1-2 days Newbie friendliness 25/100
-
Hardware features Openschema
Difficulty 3/5 1-2 days Newbie friendliness 35/100
Similar issues
-
internal.h中，漏掉了1个定义。 Open
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
kovidgoyal/kitty#10516 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 80/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
zephyrproject-rtos/zephyr#120011 ·