mikf/gallery-dl

[documentation] custom site support

Open

#1,658 opened on Jun 29, 2021

View on GitHub
 (2 comments) (3 reactions) (0 assignees)Python (8,424 stars) (737 forks)batch import
documentationhelp wanted

Description

Manually adding certain custom domains to the list of gallery-dl's recognized URLs is possible since version 1.17.0, but hasn't been documented anywhere except in the commit message of 595bdaa4.

To describe this functionality:

gallery-dl can support any site that fits into one of its "basecategories" (gelbooru_v01, gelbooru_v02, moebooru, philomena, foolfuuka, foolslide, mastodon, shopify). Several sub-sites are already supported by default (see the bottom entries in supportedsites.md), but it is also possible to get it to recognize even more with the right config entries.

The general way is to add a new entry with arbitrary name (this will be used as category) inside a block named after the basecategory, which in turn is inside the "extractor" block. These entries must specify the root URL for this site, and can optionally set a custom pattern that gets used to match the site's domain. pattern must not contain a capture group.

For example the following would add support for two additional Gelbooru Beta 0.1.11 sites as well as two Shopify instances in addition to what is already supported by default:

{
    "extractor": {
        "gelbooru_v01": {
            "rozenmaidenbooru": {"root": "http://rm.booru.org"},
            "azureblade"      : {"root": "https://tab.booru.org"}
        },
        "shopify": {
            "hiutdenim": {"root": "https://hiutdenim.co.uk"},
            "ptfoods"  : {"root": "https://partakefoods.com",
                          "pattern": "(?:www\\.)?partakefoods\\.com"}
        }
    }
}

This could definitely be described / documented in a better, easier to understand way, but I'm not very good at this. So if anyone could help with improving this, I'd be much obliged. I'm also not entirely sure where this information should go. Maybe in the Configuration section of the README? Maybe as its very own section below Authentication?

Contributor guide

Tech stack
python
Domain
documentation
Issue type
documentation
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
basic understanding of JSONfamiliarity with gallery dl configuration
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65
Research direction
Examine the commit 595bdaa4 for the initial implementation details. Review the existing supportedsites.md and Configuration section in the README to decide where to place the new documentation. Look for other config examples in the repository. The maintainer suggested adding a section below Authentication, so consider that location.