invana/crawlerflow

Web Crawlers orchestration framework that lets you create datasets from multiple web sources using yaml configurations.