Help wanted: re-record the connector cassettes against real services
メンテナーはふだん 1 日以内に返信
関連するプルリクエストがすでにマージされています。
- #639 @github-actions による — マージ済み
評価
- 難易度
- 3/5
- 見積もり時間
- 1〜2日
- 初心者へのやさしさ
- 55/100
調査の方向性
Choose an unrecorded connector from the service list and read its integration test, especially CreateProvider, plus the cassette setup in WireMockServerFixture. Build the integration-test project, run the selected test with WIREMOCK_RECORD=true, scrub personal data from the response, then unset the flag and rerun to verify replay with the placeholder token. Done means a real-service cassette is committed, replay passes, CassetteSecretTests passes, and the PR names the service, endpoints, date, and unverified paths.
索引モデルが issue の本文から書いたものです。
説明
We need people with ordinary accounts on ordinary SaaS products. No .NET expertise beyond running one build command and one test command, and no access to anything private in this project.
The problem, in one paragraph
Rag.NET ships 19 connectors that talk to real services. Their tests replay hand-written WireMock cassettes — fixtures somebody wrote from reading the API docs. That verifies the code against our belief about the API, not against the API.
This project has now been punished for exactly that twice:
- a hand-written cassette agreed with a reranker defect that destroyed 26% of every document, and the tests stayed green;
WebSearch.Tavilyshipped with broken authentication — #625, fixed in #626. It sent the key as anapi_keybody field Tavily had deprecated, with noAuthorizationheader at all. Every test passed, because the cassette matched on path and method and therefore could not observe a credential. A unit test actively pinned the defect, so a correct fix would have gone red and read as a regression.
An audit on 2026-09-15 counted 41 source cassettes, 38 of them hand-written. Only GitHub's three are real recordings. Four have a matcher that examines headers at all — those three plus Tavily's, hardened after the defect above.
The fix is recording one real exchange per service and committing the scrubbed result. The maintainer does not have accounts for most of these. That is the whole blocker.
Updated 2026-09-15. This issue used to say Phase 6.1 would fix it. 6.1 closed without the recordings — it closed by reason, which the definition of done always permitted: a
<VerifiedByReason>in the csproj naming the service and the gap, so the release would not hang on credentials that may never arrive. Seventeen packages carry one. That is honest, and it is not the same as verified. v1.0.0 has shipped. The recordings are still wanted and no longer block anything, which means this work now depends entirely on volunteers.
What one contribution looks like
Pick a service you already have an account on. Total time is usually 20–40 minutes, most of it getting a token.
1. Prerequisites
- .NET 10 SDK
- Docker not required for this work
- An account on the service, and a token/PAT with read access to something boring (one project, one page, one channel)
2. Record
The harness already supports this. WIREMOCK_RECORD=true puts WireMock into proxy-and-record mode: it forwards your test's requests to the real service and writes the real responses into the cassette directory.
# Example: Asana. Substitute your connector.
# Build first — the test runner below does not build.
dotnet build tests/Rag.NET.DataProviders.IntegrationTests -c Release
export WIREMOCK_RECORD=true
tests/Rag.NET.DataProviders.IntegrationTests/bin/Release/net10.0/Rag.NET.DataProviders.IntegrationTests.exe \
-class "*AsanaDataProviderTests"
Updated 2026-09-14 — the old
dotnet test --filterform no longer runs. It now fails with
error RAGNET0001before any test starts. Every test project moved to Microsoft.Testing.Platform,
which ignores a VSTest filter, so that command used to run every test in the assembly while
looking like it ran one; the build now refuses it rather than doing that silently. The invocation
above is the replacement — everything else on this page is unchanged.It is also better for this task: the runner prints each test's skip reason, which
dotnet testsuppresses. These suites are gated on environment variables, so the skip reason is
usually the thing you needed to read — it names the variable to set.
The test's constructor already names the upstream host it proxies to — e.g. _fixture.LoadCassettes("Asana", "https://app.asana.com"). You will need to edit the test's CreateProvider to pass your real token instead of the "fake-pat" placeholder, and your real workspace/project id, for the recording run only. Revert that edit before committing; the committed test keeps the placeholder, because the cassette no longer needs a real one.
Recorded cassettes land in:
tests/Rag.NET.DataProviders.IntegrationTests/Cassettes/<Connector>/__admin/mappings/
WireMock writes them into that __admin/mappings subdirectory. The next replay run lifts them into
Cassettes/<Connector>/ automatically — commit them from wherever they end up; both locations are read.
If you tried this before 2026-08-17, it did not work, and it was not your fault. Record mode was
broken in two ways, both invisible at the time: recordings were written to a directory replay never
read, and every recorded mapping matched onHost: localhost:<ephemeral port>, which cannot match
twice — not even on the machine that recorded it. Because record mode proxies to the real service,
the tests passed while recording and only failed later on replay. Fixed in #290, which also
produced the first working cassette.
3. Scrub — this is the part that matters
The harness no longer records your credential at all. Since #290 the proxy excludes
Authorization, Cookie, Host, User-Agent and other volatile headers, so there is no token in the
file to forget to remove. Not recording it is a better guarantee than remembering to delete it.
What you still have to check is the response body:
- your account id, workspace id, email address, real names
- anything belonging to a real customer or colleague
Set-Cookie, and any token the service returns in the body — some do
And check the parts of the response your connector never reads. This is the easy one to miss.
GitHub's compare API answers with a commits / base_commit / merge_base_commit envelope carrying
commit author and committer email addresses, while the provider reads only .Files. Emptying those
fields removed personal data and cut that cassette from 17 KB to 4 KB, and changed no value the code
consumes.
Deleting a field the connector ignores is scrubbing. Changing a value it reads is fixing the
cassette to match the code, which is the thing not to do.
A guard runs on every push and fails the build if a committed cassette matches a credential shape — GitHub, Slack, Notion, Airtable, Google, Atlassian, AWS, JWTs, private keys, bearer tokens. It reports a redacted match rather than reprinting your secret. Run it locally before pushing:
dotnet build tests/Rag.NET.RepoConventions.Tests -c Release
tests/Rag.NET.RepoConventions.Tests/bin/Release/net10.0/Rag.NET.RepoConventions.Tests.exe \
-class "*CassetteSecretTests"
If you do commit a token by accident, say so and rotate it. It is in git history from that moment; that is not a telling-off, it is just how git works.
4. Verify the replay
Unset the record flag and run again. It must pass with the placeholder token, reading only your committed cassette:
unset WIREMOCK_RECORD
tests/Rag.NET.DataProviders.IntegrationTests/bin/Release/net10.0/Rag.NET.DataProviders.IntegrationTests.exe \
-class "*<Connector>DataProviderTests"
If it fails here, work through these before assuming your recording is bad:
- A
404on every test usually means no mapping matched. Open a recorded file and look at
Request.Headers— anything machine-specific listed there is being used as a match condition. Add
it toVolatileRequestHeadersinWireMockServerFixtureand re-record. - Do not exclude
Accept. Some APIs use it to choose a representation — GitHub returns raw file
content versus JSON metadata on the same path byAcceptalone — so a cassette that stops matching
on it answers the wrong request. That header is semantic, not volatile. - Only then consider a timestamp or a nonce in the body.
Say so in the PR either way: a recording that reveals a harness problem is as useful as one that does
not.
5. In the pull request, please state
- which service and which endpoints the recording covers
- the date and, if visible, the API version — a recording is evidence of one exchange on one day, and staleness cannot be detected from inside it
- what you deliberately did not cover — e.g. "read only, never touched the write path"
- why you picked that account, project or repo, when the choice affects the cassette's size
On size: pick the smallest real thing that still has the shape the tests need. Recording this project's
own repository as the GitHub fixture produced a 904 KB cassette from a single tree call, rewritten
wholesale on every re-record; a 9-file repository gave identical coverage in 6 KB. What the tests
need is usually "more than one item, and at least one to filter out" — not a large account.
Acceptance criteria
- Cassette recorded from the real service, not hand-written
- No credentials, cookies, or personal data —
CassetteSecretTestsgreen - The connector's test passes on replay with the placeholder token
- PR states service, endpoints, date, and what stays unverified
The services we need (18 left of 19)
Comment to claim one, so two people do not record the same thing.
DataProviders.GitHub is done (#290) — the maintainer has an account there, and doing it first is
what surfaced the two harness defects above. Use it as the worked example: full tree, one file's
content, and the delta/compare path, recorded unauthenticated, 17 KB total.
Free account, usually minutes
-
DataProviders.GitHub— GitHub API — done, #290 -
DataProviders.GitLab— GitLab API -
DataProviders.Notion— Notion API -
DataProviders.Airtable— Airtable API -
DataProviders.Linear— Linear API -
DataProviders.Slack— Slack API -
DataProviders.Dropbox— Dropbox API -
DataProviders.Box— Box API -
DataProviders.Asana— Asana API -
WebSearch.Tavily— Tavily API — this is the one that shipped an auth defect (#625). Still unrecorded. -
Reranking.Cohere— Cohere Rerank API
Free tier, a little more setup
-
DataProviders.Jira— Atlassian Cloud -
DataProviders.Confluence— same Atlassian site as Jira; one account covers both -
DataProviders.Zendesk— trial account
Needs a tenant or a cloud subscription
-
DataProviders.Microsoft365— Microsoft Graph; needs a tenant and an app registration -
DataProviders.Gmail— Google Cloud project + OAuth consent -
DataProviders.GoogleDrive— same Google project as Gmail; one setup covers both -
Parsers.Pdf.AzureDocumentIntelligence— needs an Azure subscription. Milestone 4 recorded this live half as never run.
What we are not asking for
Not a passing recording at any cost. A recording that reveals the connector is broken is the most valuable outcome here — that is the entire point of recording instead of writing. If the real API returns a shape the code mishandles, open the issue; do not fix the cassette to match the code.
And if you would rather not create an account for a service you do not use: that is fine and expected. A package with no recording gets a <VerifiedByReason> in its csproj naming the service and what stays unverified. Honest and machine-readable beats absent.
- 主要言語
- C#
- スター
- 5
- フォーク
- 2
- 平均マージ
- 14時間 28分
- マージ済み PR(30日)
- 190
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
MarcelRoozekrans/Rag.NET のほかの issue
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
MarcelRoozekrans/Rag.NET#1030 ·
メンテナーはふだん 1 日以内に返信
-
難易度 5/5 1週間以上 初心者へのやさしさ 35/100
MarcelRoozekrans/Rag.NET#944 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 4/5 3〜5日 初心者へのやさしさ 35/100
MarcelRoozekrans/Rag.NET#937 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 4/5 3〜5日 初心者へのやさしさ 48/100
MarcelRoozekrans/Rag.NET#888 ·
メンテナーはふだん 1 日以内に返信
-
enhancement
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
MarcelRoozekrans/Rag.NET#855 ·
メンテナーはふだん 1 日以内に返信
MarcelRoozekrans/Rag.NET の issue をすべて見る
似ている issue
-
quality 提案 要トリアージ
難易度 2/5 1〜3時間 初心者へのやさしさ 64/100
メンテナーはふだん 1 日以内に返信
-
The Language and Region lists in Options read to a screen reader as "LanguageInfo { Id = ar, ... }"オープンbug localization small ui
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
hexide-io/HexIDE#4176 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
ReapplyLocalizationsToAllComponents aborts (and keeps aborting) on a disposed registered controlオープン
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
-
難易度 2/5 1〜3時間 初心者へのやさしさ 88/100
AvaloniaUI/Avalonia#22841 ·
メンテナーはふだん 1 日以内に返信
-
a11y tech-debt
難易度 1/5 1時間未満 初心者へのやさしさ 78/100
philipphoeninger/procure-access#350 ·
メンテナーはふだん 1 日以内に返信