Help wanted: re-record the connector cassettes against real services

オープン
#283 コメント 5 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

関連するプルリクエストがすでにマージされています。

  • #639 @github-actions による — マージ済み

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
55/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
csharp
領域
backend, testing

調査の方向性

Choose an unrecorded connector from the service list and read its integration test, especially CreateProvider, plus the cassette setup in WireMockServerFixture. Build the integration-test project, run the selected test with WIREMOCK_RECORD=true, scrub personal data from the response, then unset the flag and rerun to verify replay with the placeholder token. Done means a real-service cassette is committed, replay passes, CassetteSecretTests passes, and the PR names the service, endpoints, date, and unverified paths.

索引モデルが issue の本文から書いたものです。

説明

good first issue help wanted

We need people with ordinary accounts on ordinary SaaS products. No .NET expertise beyond running one build command and one test command, and no access to anything private in this project.

The problem, in one paragraph

Rag.NET ships 19 connectors that talk to real services. Their tests replay hand-written WireMock cassettes — fixtures somebody wrote from reading the API docs. That verifies the code against our belief about the API, not against the API.

This project has now been punished for exactly that twice:

  • a hand-written cassette agreed with a reranker defect that destroyed 26% of every document, and the tests stayed green;
  • WebSearch.Tavily shipped with broken authentication — #625, fixed in #626. It sent the key as an api_key body field Tavily had deprecated, with no Authorization header at all. Every test passed, because the cassette matched on path and method and therefore could not observe a credential. A unit test actively pinned the defect, so a correct fix would have gone red and read as a regression.

An audit on 2026-09-15 counted 41 source cassettes, 38 of them hand-written. Only GitHub's three are real recordings. Four have a matcher that examines headers at all — those three plus Tavily's, hardened after the defect above.

The fix is recording one real exchange per service and committing the scrubbed result. The maintainer does not have accounts for most of these. That is the whole blocker.

Updated 2026-09-15. This issue used to say Phase 6.1 would fix it. 6.1 closed without the recordings — it closed by reason, which the definition of done always permitted: a <VerifiedByReason> in the csproj naming the service and the gap, so the release would not hang on credentials that may never arrive. Seventeen packages carry one. That is honest, and it is not the same as verified. v1.0.0 has shipped. The recordings are still wanted and no longer block anything, which means this work now depends entirely on volunteers.

What one contribution looks like

Pick a service you already have an account on. Total time is usually 20–40 minutes, most of it getting a token.

1. Prerequisites
  • .NET 10 SDK
  • Docker not required for this work
  • An account on the service, and a token/PAT with read access to something boring (one project, one page, one channel)
2. Record

The harness already supports this. WIREMOCK_RECORD=true puts WireMock into proxy-and-record mode: it forwards your test's requests to the real service and writes the real responses into the cassette directory.

# Example: Asana. Substitute your connector.
# Build first — the test runner below does not build.
dotnet build tests/Rag.NET.DataProviders.IntegrationTests -c Release

export WIREMOCK_RECORD=true
tests/Rag.NET.DataProviders.IntegrationTests/bin/Release/net10.0/Rag.NET.DataProviders.IntegrationTests.exe \
  -class "*AsanaDataProviderTests"

Updated 2026-09-14 — the old dotnet test --filter form no longer runs. It now fails with
error RAGNET0001 before any test starts. Every test project moved to Microsoft.Testing.Platform,
which ignores a VSTest filter, so that command used to run every test in the assembly while
looking like it ran one; the build now refuses it rather than doing that silently. The invocation
above is the replacement — everything else on this page is unchanged.

It is also better for this task: the runner prints each test's skip reason, which
dotnet test suppresses. These suites are gated on environment variables, so the skip reason is
usually the thing you needed to read — it names the variable to set.

The test's constructor already names the upstream host it proxies to — e.g. _fixture.LoadCassettes("Asana", "https://app.asana.com"). You will need to edit the test's CreateProvider to pass your real token instead of the "fake-pat" placeholder, and your real workspace/project id, for the recording run only. Revert that edit before committing; the committed test keeps the placeholder, because the cassette no longer needs a real one.

Recorded cassettes land in:

tests/Rag.NET.DataProviders.IntegrationTests/Cassettes/<Connector>/__admin/mappings/

WireMock writes them into that __admin/mappings subdirectory. The next replay run lifts them into
Cassettes/<Connector>/ automatically
— commit them from wherever they end up; both locations are read.

If you tried this before 2026-08-17, it did not work, and it was not your fault. Record mode was
broken in two ways, both invisible at the time: recordings were written to a directory replay never
read, and every recorded mapping matched on Host: localhost:<ephemeral port>, which cannot match
twice — not even on the machine that recorded it. Because record mode proxies to the real service,
the tests passed while recording and only failed later on replay. Fixed in #290, which also
produced the first working cassette.

3. Scrub — this is the part that matters

The harness no longer records your credential at all. Since #290 the proxy excludes
Authorization, Cookie, Host, User-Agent and other volatile headers, so there is no token in the
file to forget to remove. Not recording it is a better guarantee than remembering to delete it.

What you still have to check is the response body:

  • your account id, workspace id, email address, real names
  • anything belonging to a real customer or colleague
  • Set-Cookie, and any token the service returns in the body — some do

And check the parts of the response your connector never reads. This is the easy one to miss.
GitHub's compare API answers with a commits / base_commit / merge_base_commit envelope carrying
commit author and committer email addresses, while the provider reads only .Files. Emptying those
fields removed personal data and cut that cassette from 17 KB to 4 KB, and changed no value the code
consumes.

Deleting a field the connector ignores is scrubbing. Changing a value it reads is fixing the
cassette to match the code
, which is the thing not to do.

A guard runs on every push and fails the build if a committed cassette matches a credential shape — GitHub, Slack, Notion, Airtable, Google, Atlassian, AWS, JWTs, private keys, bearer tokens. It reports a redacted match rather than reprinting your secret. Run it locally before pushing:

dotnet build tests/Rag.NET.RepoConventions.Tests -c Release
tests/Rag.NET.RepoConventions.Tests/bin/Release/net10.0/Rag.NET.RepoConventions.Tests.exe \
  -class "*CassetteSecretTests"

If you do commit a token by accident, say so and rotate it. It is in git history from that moment; that is not a telling-off, it is just how git works.

4. Verify the replay

Unset the record flag and run again. It must pass with the placeholder token, reading only your committed cassette:

unset WIREMOCK_RECORD
tests/Rag.NET.DataProviders.IntegrationTests/bin/Release/net10.0/Rag.NET.DataProviders.IntegrationTests.exe \
  -class "*<Connector>DataProviderTests"

If it fails here, work through these before assuming your recording is bad:

  1. A 404 on every test usually means no mapping matched. Open a recorded file and look at
    Request.Headers — anything machine-specific listed there is being used as a match condition. Add
    it to VolatileRequestHeaders in WireMockServerFixture and re-record.
  2. Do not exclude Accept. Some APIs use it to choose a representation — GitHub returns raw file
    content versus JSON metadata on the same path by Accept alone — so a cassette that stops matching
    on it answers the wrong request. That header is semantic, not volatile.
  3. Only then consider a timestamp or a nonce in the body.

Say so in the PR either way: a recording that reveals a harness problem is as useful as one that does
not.

5. In the pull request, please state
  • which service and which endpoints the recording covers
  • the date and, if visible, the API version — a recording is evidence of one exchange on one day, and staleness cannot be detected from inside it
  • what you deliberately did not cover — e.g. "read only, never touched the write path"
  • why you picked that account, project or repo, when the choice affects the cassette's size

On size: pick the smallest real thing that still has the shape the tests need. Recording this project's
own repository as the GitHub fixture produced a 904 KB cassette from a single tree call, rewritten
wholesale on every re-record; a 9-file repository gave identical coverage in 6 KB. What the tests
need is usually "more than one item, and at least one to filter out" — not a large account.

Acceptance criteria

  • Cassette recorded from the real service, not hand-written
  • No credentials, cookies, or personal data — CassetteSecretTests green
  • The connector's test passes on replay with the placeholder token
  • PR states service, endpoints, date, and what stays unverified

The services we need (18 left of 19)

Comment to claim one, so two people do not record the same thing.

DataProviders.GitHub is done (#290) — the maintainer has an account there, and doing it first is
what surfaced the two harness defects above. Use it as the worked example: full tree, one file's
content, and the delta/compare path, recorded unauthenticated, 17 KB total.

Free account, usually minutes

  • DataProviders.GitHub — GitHub API — done, #290
  • DataProviders.GitLab — GitLab API
  • DataProviders.Notion — Notion API
  • DataProviders.Airtable — Airtable API
  • DataProviders.Linear — Linear API
  • DataProviders.Slack — Slack API
  • DataProviders.Dropbox — Dropbox API
  • DataProviders.Box — Box API
  • DataProviders.Asana — Asana API
  • WebSearch.Tavily — Tavily API — this is the one that shipped an auth defect (#625). Still unrecorded.
  • Reranking.Cohere — Cohere Rerank API

Free tier, a little more setup

  • DataProviders.Jira — Atlassian Cloud
  • DataProviders.Confluence — same Atlassian site as Jira; one account covers both
  • DataProviders.Zendesk — trial account

Needs a tenant or a cloud subscription

  • DataProviders.Microsoft365 — Microsoft Graph; needs a tenant and an app registration
  • DataProviders.Gmail — Google Cloud project + OAuth consent
  • DataProviders.GoogleDrive — same Google project as Gmail; one setup covers both
  • Parsers.Pdf.AzureDocumentIntelligence — needs an Azure subscription. Milestone 4 recorded this live half as never run.

What we are not asking for

Not a passing recording at any cost. A recording that reveals the connector is broken is the most valuable outcome here — that is the entire point of recording instead of writing. If the real API returns a shape the code mishandles, open the issue; do not fix the cassette to match the code.

And if you would rather not create an account for a service you do not use: that is fine and expected. A package with no recording gets a <VerifiedByReason> in its csproj naming the service and what stays unverified. Honest and machine-readable beats absent.

主要言語
C#
スター
5
フォーク
2
平均マージ
14時間 28分
マージ済み PR(30日)
190

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

MarcelRoozekrans/Rag.NET のほかの issue

MarcelRoozekrans/Rag.NET の issue をすべて見る

似ている issue

C# の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。