Ingest: a crawl source for sites without a sitemap

Aberta
  • Algo errado?
#586 1 comentário 0 reações 1 responsável Ver no GitHub

Mantenedores costumam responder em até 1 dia

@bhushan4work já está trabalhando nisso.

Desde 4/11/2026.

  • #588 de @bhushan4work — aberto

Avaliação

Esta issue ainda não foi avaliada.

Descrição

enhancement

F-06 task 12. ingest() finds pages from a sitemap, a list of URLs, or a local folder (IngestSource in packages/ingest/src/types.ts:5-9). Plenty of small sites have no sitemap. For those, the cloud's onboarding (and chatcops ingest) needs to crawl from the homepage instead.

sources: [{ type: 'crawl', url: 'https://example.com', maxDepth: 3, maxPages: 200 }]

What to do

  • Add { type: 'crawl'; url: string; maxDepth?: number; maxPages?: number } to IngestSource, with defaults of 3 levels and 500 pages.
  • Handle it in discover() (packages/ingest/src/discover.ts:25). Starting from url, fetch each page, collect <a href> links, and follow those on the same origin breadth-first, up to maxDepth levels and maxPages pages.
    • Normalize every link with the existing normalizeUrl() and skip ones already seen.
    • Skip mailto:, tel:, fragments-only links and non-HTML responses.
    • Respect robots.txt the way the other sources already do (robots.ts), and apply include/exclude before fetching.
  • Reuse the fetched HTML so a crawled page isn't downloaded twice when it's extracted. The fetch and extract steps are in ingest.ts.
  • The concurrency cap (concurrency) and timeoutMs apply to crawl fetches too.
  • Add --crawl <url> to chatcops ingest in packages/cli/src/main.ts, or accept crawl: sources in the config, whichever fits the existing --source handling better.

Acceptance criteria

  • Tests with a local fixture site served by a test HTTP server (packages/ingest/tests already has one for sitemaps): follows same-origin links, stops at maxDepth and maxPages, ignores other origins, mailto: and fragments, honours robots.txt and exclude, and fetches each page once
  • The source in the ingest options table in website/src/content/docs/core/knowledge-base.mdx (pnpm docs:check passes)
  • A minor changeset for @chatcops/ingest (and @chatcops/cli if the CLI changes); pnpm test:coverage (85/80), pnpm lint and pnpm api (commit the snapshot) pass

Comment here to be assigned.

Linguagem predominante
TypeScript
Estrelas
13
Forks
7
Merge médio
3h 14min
PRs com merge (30d)
332

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de codercops/chatcops

Todas as issues de codercops/chatcops

Issues semelhantes

Mais issues de TypeScript

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.