add option to limit number of pages crawled?

Open
  • Is something wrong?
#71 8 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
38/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
typescript
Domain
cli

Research direction

Start by locating the existing --max-crawl-depth option and the crawler loop that applies its limits. Add a page-count option whose value is the smaller of the requested limit and the pages allowed by other options, then verify that crawling stops at that limit.

Written by the indexing model from the issue text.

Description

Hey this is a cool tool. Here's a feature idea assuming you're open to it.

Currently one can use:

--max-crawl-depth  2

to get, say, index page + 1st linked pages.

But maybe there are a lot of linked pages, and you just need a representative sample which is more than 1 page but not tons of pages.

So maybe another option like:

--max-crawled-pages N # or just --max-pages ?

and the crawler stops after it exhausts all pages allowed by other options, or N, whichever is smaller.

One might then use it like:

lighthouse-parade --max-crawl-depth 2 --max-crawled-pages 20 example.com
Dominant language
TypeScript
Stars
373
Forks
15
Avg merge
15h 20m
Merged PRs (30d)
25

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from cloudfour/lighthouse-parade

All issues in cloudfour/lighthouse-parade

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.