💡 Feature request: Disable web crawlers by default
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 35/100
Research direction
Start by locating the code or template that serves robots.txt and check how its current crawling rules are chosen. Make robots.txt disallow all crawlers by default, with crawling available as an option; done when the default response blocks crawlers and the option permits them. The issue does not name a file or test, and its last comment was 322 days ago.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem? Please describe.
I was sent a DMCA (along with other Redlib instance owners) by WebToon and Google because of a post on Reddit that the web crawler visited through Redlib
Describe the feature you would like to be implemented
I'd like the crawler protection to be enabled by default
Describe alternatives you've considered
There aren't any alternatives other than captchas
Additional context / screenshot
I specifically just mean the robots.txt file should disallow all by default and have crawling as an option
- Dominant language
- Rust
- Stars
- 3.9k
- Forks
- 293
- Avg merge
- 14d 8h
- Merged PRs (30d)
- 2
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from redlib-org/redlib
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
redlib-org/redlib#604 ·
Maintainers usually reply within 3 days
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
redlib-org/redlib#571 · 1 comment ·
Maintainers usually reply within 3 days
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 61/100
redlib-org/redlib#557 ·
Maintainers usually reply within 3 days
-
🐛 Bug Report: Failed to parse page JSON data: expected value at line 1 column 1 | /r/LivestreamFail/comments/1xd628q/hila_klein_says_ethan_klein_has_been_having_an/.json?Possibly taken A pull request linked to this issue is open or already merged. Openbug
Difficulty 3/5 1-2 days Newbie friendliness 38/100
redlib-org/redlib#602 · 1 comment ·
Maintainers usually reply within 3 days
-
bug
Difficulty 4/5 1-2 days Newbie friendliness 52/100
redlib-org/redlib#600 · 7 comments · 7 reactions ·
Maintainers usually reply within 3 days
All issues in redlib-org/redlib
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
status:untriaged
Difficulty 1/5 Under an hour Newbie friendliness 72/100
midnightntwrk/midnight-zk#592 ·
Maintainers usually reply within 3 days
-
QuartoHub: have the Document name before the filename in the browser tabPossibly taken @cscheid claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
quarto-dev/q2#930 · 1 comment ·
Maintainers usually reply within 1 day
-
muse P3
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
HerringtonDarkholme/hd-lang#923 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day