Why does sitemap.xml get crawled before the passed URL
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- node.js, typescript
- Domain
- cli
Research direction
Start by tracing the crawl startup and queue ordering in the TypeScript CLI, then inspect how the simplecrawler dependency handles sitemap.xml and the requested URL. Reproduce the command shown in the issue and compare the enqueue and crawl events. Done means explaining the ordering and identifying a concrete behavior change, if one is required.
Written by the indexing model from the issue text.
Description
Ex:
$ lighthouse-parade htts://www.baptistjax.com
Created CSV file
Starting the crawl...
Crawled https://www.baptistjax.com/sitemap.xml [text/xml] (646288 bytes)
Crawled https://www.baptistjax.com/ [text/html; charset=utf-8] (289135 bytes)
Report is done for https://www.baptistjax.com/
Wrote report for https://www.baptistjax.com/
Crawled https://www.baptistjax.com/services [text/html; charset=utf-8] (246368 bytes)
Report is done for https://www.baptistjax.com/services
Wrote report for https://www.baptistjax.com/services
Crawled https://www.baptistjax.com/site-search [text/html; charset=utf-8] (244996 bytes)
Notice that `sitemap.xml' is crawled before the url I requested. Why? Maybe this is some internal logic of simplecrawler?
Possibly related to #3
- Dominant language
- TypeScript
- Stars
- 373
- Forks
- 15
- Avg merge
- 19h 16m
- Merged PRs (30d)
- 23
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from cloudfour/lighthouse-parade
-
Run the dependency drift checkPossibly taken @spaceninja claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 72/100
cloudfour/lighthouse-parade#449 ·
Maintainers usually reply within 1 day
-
Dependency DashboardOpen
Difficulty 2/5 1-3 hours Newbie friendliness 20/100
cloudfour/lighthouse-parade#166 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
cloudfour/lighthouse-parade#151 ·
Maintainers usually reply within 1 day
-
Will the reports be affected if I'm doing something processor intensive while the report is running?Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
cloudfour/lighthouse-parade#147 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
cloudfour/lighthouse-parade#146 · 12 comments ·
Maintainers usually reply within 1 day
All issues in cloudfour/lighthouse-parade
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
unocss/unocss#5421 · 1 comment ·
Maintainers usually reply within 1 day
-
needs-area
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Azure/typespec-azure#5936 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 Half a day Newbie friendliness 72/100
vercel-labs/scriptc#2910 ·
Maintainers usually reply within 1 day
-
automation missing-model model-sync provider:fireworks-ai
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
anomalyco/models.dev#12458 ·
Maintainers usually reply within 1 day