gocolly/colly

cache and proxy

开放

#187 创建于 2018年7月10日

 (2 条评论) (0 个反应) (0 位负责人)Go (1,857 个派生)batch import
enhancementhelp wanted

仓库指标

星标
 (25,470 个星标)
PR 合并指标
 (平均合并 15小时 38分钟) (30 天内合并 2 个 PR)

描述

Hey guys,

I have an issue. In real world of scrappers when using proxies, checking if You have response code 200 or 500 is simply not enough. Plenty of proxies or even website itself can throw an output with code 200 which is invalid from developer point of view and yet its getting cached. Is there any way to apply some kind of custom logic which would determine if response is what we want? For example:

colly.CacheDir("some/dir", func(resp) {
   if bytes.Contains(resp.Body, []byte("some magic marker")) {
      return false // don't cache
   }
})

I'm new to colly and i was digging thru source code for a bit, but didn't find such of feature. It's also useful without proxies.

贡献者指南