Ingest: a crawl source for sites without a sitemap

未关闭
  • 哪里不对？
#586 1 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 1 天内回复

@bhushan4work 已经在做这个了。

开始于 2026年11月4日。

  • #588 来自 @bhushan4work —— 未关闭

评估

这个 Issue 还没有评估数据。

描述

enhancement

F-06 task 12. ingest() finds pages from a sitemap, a list of URLs, or a local folder (IngestSource in packages/ingest/src/types.ts:5-9). Plenty of small sites have no sitemap. For those, the cloud's onboarding (and chatcops ingest) needs to crawl from the homepage instead.

sources: [{ type: 'crawl', url: 'https://example.com', maxDepth: 3, maxPages: 200 }]

What to do

  • Add { type: 'crawl'; url: string; maxDepth?: number; maxPages?: number } to IngestSource, with defaults of 3 levels and 500 pages.
  • Handle it in discover() (packages/ingest/src/discover.ts:25). Starting from url, fetch each page, collect <a href> links, and follow those on the same origin breadth-first, up to maxDepth levels and maxPages pages.
    • Normalize every link with the existing normalizeUrl() and skip ones already seen.
    • Skip mailto:, tel:, fragments-only links and non-HTML responses.
    • Respect robots.txt the way the other sources already do (robots.ts), and apply include/exclude before fetching.
  • Reuse the fetched HTML so a crawled page isn't downloaded twice when it's extracted. The fetch and extract steps are in ingest.ts.
  • The concurrency cap (concurrency) and timeoutMs apply to crawl fetches too.
  • Add --crawl <url> to chatcops ingest in packages/cli/src/main.ts, or accept crawl: sources in the config, whichever fits the existing --source handling better.

Acceptance criteria

  • Tests with a local fixture site served by a test HTTP server (packages/ingest/tests already has one for sitemaps): follows same-origin links, stops at maxDepth and maxPages, ignores other origins, mailto: and fragments, honours robots.txt and exclude, and fetches each page once
  • The source in the ingest options table in website/src/content/docs/core/knowledge-base.mdx (pnpm docs:check passes)
  • A minor changeset for @chatcops/ingest (and @chatcops/cli if the CLI changes); pnpm test:coverage (85/80), pnpm lint and pnpm api (commit the snapshot) pass

Comment here to be assigned.

主要语言
TypeScript
星标
13
派生
7
平均合并
3 小时 11 分钟
30 天内合并 PR
330

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

codercops/chatcops 的其他 Issue

查看 codercops/chatcops 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。