Redditbot is currently blocked by AppSec and not excluded from bot challenges

オープン
  • 何か問題がありますか？
#4,685 コメント 7 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
30/100
issue の種類
機能追加
明瞭さ
説明が足りない
活発さ
活発
領域
security

調査の方向性

Start with the linked appsec-bot-challenge-exclude-social WAF configuration and compare how its existing social crawlers are verified. The issue does not identify a trustworthy Redditbot network attribute or settle whether a User-Agent-only definition is acceptable, so first establish a recommended verification method. Done means Redditbot can be allowed without relying on spoofable headers or broad cloud IP ranges.

索引モデルが issue の本文から書いたものです。

説明

area/bot-detection kind/enhancement needs/triage
What would you like to be added?

The Reddit link-preview crawler (redditbot) is currently being blocked/challenged by CrowdSec AppSec and the new bot detection feature.

It is not currently included in the appsec-bot-challenge-exclude-social WAF configuration.

I would like to add support for Redditbot as a legitimate social-media crawler, similar to the existing exclusions for services such as Twitter/X, LinkedIn, Discord, Pinterest, etc.

Observed User-Agent

The requests I am seeing use the following User-Agent:

Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)

The identifying part appears to be:

redditbot/1.0
Network verification

The challenge is that, unlike some of the other supported bots, I have not been able to identify a dedicated and trustworthy IP range for Redditbot that we could use for verification.

For example, we observed a request with the following headers:

User-Agent: Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)
X-Forwarded-Host: ...
X-Span: ...
X-Sampled: 1
X-Trace: ...
Traceparent: ...
X-Forwarded-Server: ...
X-Parent: 0

However, I don't consider these headers to be suitable authentication/verification mechanisms for a legitimate bot whitelist. I also don't want to whitelist broad cloud-provider IP ranges merely because Redditbot may originate from such infrastructure.

Question

What is the recommended way to add redditbot to the appsec-bot-challenge-exclude-social configuration?

In particular:

  1. Is there an official Redditbot IP range, ASN, reverse-DNS pattern, or other network attribute that CrowdSec recommends using to verify that requests genuinely originate from Reddit?
  2. If no trustworthy network verification is available, does CrowdSec support a User-Agent-only legitimate-bot definition for Redditbot? I assume not, as the user-agent can be spoofed by anyone and would bypass bot protection completely.
  3. Is there another recommended mechanism for allowing Reddit's URL/link-preview crawler without weakening AppSec bot protection?

I'd be happy to contribute a reddit.json definition if there is a recommended verification method.

Thanks!

Why is this needed?

Unlike other social media bots, Reddit is not included in the list of appsec-bot-challenge-exclude-social. Therefore, it is blocked and gets the bot verification challenge. Unable to fetch blog post seo content, images and sorts.

主要言語
Go
スター
15.1k
フォーク
726
平均マージ
2日 51分
マージ済み PR（30日）
18

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

crowdsecurity/crowdsec のほかの issue

crowdsecurity/crowdsec の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。