lobsters/lobsters

add Anubis with a conservative config

Open

#1,781 创建于 2025年11月3日

在 GitHub 查看
 (6 评论) (0 反应) (0 负责人)Ruby (961 fork)user submission
featurereqgood first issueperformance

仓库指标

Star
 (4,680 star)
PR 合并指标
 (平均合并 7天 4小时) (30 天内合并 20 个 PR)

描述

We haven't had a lot of problems from scrapers over the years because we have put a bunch of time into performance work, especially by using full-page caching for logged-out visitors to avoid even touching the app servers. That means we have the breathing room to take a very gradual, conservative approach to increasing mitigations like Anubis, which we should use to only challenge the riskiest traffic. An unusually high percentage of our users (unknown, but probably single-digit) cannot solve javascript proof-of-work challenges because they're using strict privacy/security controls, homebrewed browsers, or other unusual setups.

Saturday before last (2025-10-25) Lobsters was slow/mostly down because were were getting aggressively scraped by a bot that used many thousands of IPs to avoid rate-limiting.

In the immediate mitigation, I was able to block their IPs with iptables and a list of bad-actor IPs for ASINs owned by that network because cadey shared the list that's already in Anubis for them.

I also fixed the endpoint they overwhelmed in 5e29c8d60. It normally took ~8ms to look up a redirect, but the fact that it did 2-3 queries instead of serving a locally cached page allowed them to dogpile the database.

This probably isn't the only endpoint susceptible to abuse, nor the last aggressive scraper we'll see. So I'd like to add Anubis, conservatively configured to only challenge ASINs from its bad actor lists, which are here and here.

贡献者指南