Ingest: a crawl source for sites without a sitemap

Aperta
  • Qualcosa non va?
#586 1 commento 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@bhushan4work ci sta già lavorando.

Dal 4/11/2026.

  • #588 di @bhushan4work — aperta

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

enhancement

F-06 task 12. ingest() finds pages from a sitemap, a list of URLs, or a local folder (IngestSource in packages/ingest/src/types.ts:5-9). Plenty of small sites have no sitemap. For those, the cloud's onboarding (and chatcops ingest) needs to crawl from the homepage instead.

sources: [{ type: 'crawl', url: 'https://example.com', maxDepth: 3, maxPages: 200 }]

What to do

  • Add { type: 'crawl'; url: string; maxDepth?: number; maxPages?: number } to IngestSource, with defaults of 3 levels and 500 pages.
  • Handle it in discover() (packages/ingest/src/discover.ts:25). Starting from url, fetch each page, collect <a href> links, and follow those on the same origin breadth-first, up to maxDepth levels and maxPages pages.
    • Normalize every link with the existing normalizeUrl() and skip ones already seen.
    • Skip mailto:, tel:, fragments-only links and non-HTML responses.
    • Respect robots.txt the way the other sources already do (robots.ts), and apply include/exclude before fetching.
  • Reuse the fetched HTML so a crawled page isn't downloaded twice when it's extracted. The fetch and extract steps are in ingest.ts.
  • The concurrency cap (concurrency) and timeoutMs apply to crawl fetches too.
  • Add --crawl <url> to chatcops ingest in packages/cli/src/main.ts, or accept crawl: sources in the config, whichever fits the existing --source handling better.

Acceptance criteria

  • Tests with a local fixture site served by a test HTTP server (packages/ingest/tests already has one for sitemaps): follows same-origin links, stops at maxDepth and maxPages, ignores other origins, mailto: and fragments, honours robots.txt and exclude, and fetches each page once
  • The source in the ingest options table in website/src/content/docs/core/knowledge-base.mdx (pnpm docs:check passes)
  • A minor changeset for @chatcops/ingest (and @chatcops/cli if the CLI changes); pnpm test:coverage (85/80), pnpm lint and pnpm api (commit the snapshot) pass

Comment here to be assigned.

Lingua principale
TypeScript
Stelle
13
Fork
7
Merge medio
3h 11m
PR unite (30g)
330

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di codercops/chatcops

Tutte le issue di codercops/chatcops

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.