gocolly/colly

cache and proxy

Open

#187 opened on Jul 10, 2018

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Go (24,898 stars) (1,837 forks)batch import
enhancementhelp wanted

Description

Hey guys,

I have an issue. In real world of scrappers when using proxies, checking if You have response code 200 or 500 is simply not enough. Plenty of proxies or even website itself can throw an output with code 200 which is invalid from developer point of view and yet its getting cached. Is there any way to apply some kind of custom logic which would determine if response is what we want? For example:

colly.CacheDir("some/dir", func(resp) {
   if bytes.Contains(resp.Body, []byte("some magic marker")) {
      return false // don't cache
   }
})

I'm new to colly and i was digging thru source code for a bit, but didn't find such of feature. It's also useful without proxies.

Contributor guide

Tech stack
go
Domain
backendtooling
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Go basicscolly framework understanding
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
The user proposes adding a custom validation function to Colly's caching mechanism. To implement, examine the existing CacheDir function in the colly source (likely in colly.go or cache.go). The callback should allow users to inspect the response and decide whether to cache it. A pull request introducing an optional validation callback parameter to CacheDir would be appropriate. Review any existing discussions in the issue comments for additional context.