Python: relevance-score gates (min_relevance_score / DEFAULT_RELEVANCE) have no polarity cases in tests: a memory that says the opposite of the query is recalled as highly relevant.
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 55/100
Research direction
Review python/tests/integration/embeddings/test_embedding_service_with_memory.py and the recall paths in text_memory_plugin.py and semantic_text_memory.py. Add behavioral coverage for polarity and low-overlap cases at non-zero thresholds, using the issue's proposed vector provenance requirements; done means the shipped gates are exercised beyond the current min_relevance_score=0.0 integration calls.
Written by the indexing model from the issue text.
Description
Language: Python. The .NET defaults differ and are recorded at the end.
Verified against main at b39d95a on 2026-08-25.
Hi SK team. I just finished a construct-validity audit of embedding-cosine gates across agent
frameworks, and SK's memory-recall path carries the same test blind spot I found in four other
suites. Preprint: https://arxiv.org/abs/2608.10216.
The pattern. TextMemoryPlugin.recall gates on DEFAULT_RELEVANCE = 0.75
(text_memory_plugin.py#L22),
and SemanticTextMemory.search gates on min_relevance_score, defaulting to 0.0
(semantic_text_memory.py#L129),
which it passes through to the store at L153. Both implement one decision, "is this stored text
relevant to, meaning does it answer or agree with, this query?", and both answer it by thresholding
embedding similarity. That score measures shared wording rather than shared meaning, and in the pairs
a recall gate most needs to rank correctly the two run opposite ways: reversing an instruction is a
one-token edit that keeps similarity near maximal, while a faithful restatement in fresh words drops
it substantially. To be precise about framing, nothing here is a bug in SK's code, and the gate does
exactly what it says. The blind spot sits in what the test suite asserts the gate means.
Specimen from my audit corpus (cosine under nomic-embed-text-v1.5, MRL-256, frozen in the paper's
artifact; the mutation class scored 0.83 to 0.9997 across all nine encoders tested, so this is not
one model's quirk):
- Stored memory: "Withhold the study drug from any participant who reports chest tightness."
- Query: "Administer the study drug to any participant who reports chest tightness."
- Cosine 0.9608. It clears 0.75 comfortably, and clears any plausible relevance floor. At the 0.0
default it is not even a speed bump. - Mirror case: faithful low-overlap restatements in the same corpus average about 0.76 under the same
encoder, for example "Retry the request at most three times." against "Give the call up to three
attempts, then stop.". That average lands right at the plugin cut, so faithful restatements sit on
the knife edge while reversals clear it comfortably.
What the suite covers today. recall has no behavioral unit test. Across python/tests,
TextMemoryPlugin appears only in test_serialization.py, and the one test that exercises the gate
is
test_embedding_service_with_memory.py#L155-L169,
an integration test needing live credentials, which calls memory.search(...) four times and passes
min_relevance_score=0.0 at every call site. The threshold is never exercised at a non-zero value
anywhere in the suite, and the fixtures are taxonomic facts, so no pair shares wording while
disagreeing about what it decides.
Proposal, sized to be PR-able, and I am happy to write it:
- Add polarity cases: pairs sharing wording with opposite decisions (negation,
musttomay,
quantity changes, scope inversion) asserted at the shipped defaults, and pairs sharing the
decision with no shared wording. Section 11.3 of the paper gives the two families. The corpus
pairs are released and were authored blind to any encoder. - Record the vectors from a named encoder and check them in with provenance (model id, dimension,
date, generating script) rather than hand-authoring them. A hand-authored vector asserts only the
fixture author's arithmetic, and passes whatever the sentences happen to say. - Optionally, one line in the
relevanceandmin_relevance_scoredocstrings noting that the score
is surface-form similarity, so "relevant" should not be read as "in agreement".
.NET, and a correction to my original filing. @journaltraces checked the C# side and found two things
I had wrong.
TextMemoryPlugin.cs#L47
has private const double DefaultRelevance = 0.0;, so the .NET plugin ships an open gate where the
Python one ships 0.75. And VectorStoreTextSearch carries no minRelevanceScore at all:
ExecuteVectorSearchCoreAsync builds a VectorSearchOptions<TRecord> with Filter only and passes
top. I originally wrote that the deprecation migration ports the blind spot forward. It does not
port it, it drops the gate: a caller who migrates moves from a threshold that can at least be raised
to a top-k that always returns its k, with no floor left to argue about. Thanks to them for both.
The audit found the identical gap in LlamaIndex, LangChain, and GPTCache-style caches. SK is in good
company, which is rather the point: the failure mode ships because no framework's tests can see it.
- Dominant language
- C#
- Stars
- 28.8k
- Forks
- 4.9k
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 48
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/semantic-kernel
-
Python: Bug: ChatMessageContent.from_element drops AudioContent and BinaryContent into raw TextContentPossibly taken @halcyon-past claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
microsoft/semantic-kernel#14824 · 1 comment ·
Maintainers usually reply within 3 days
-
Python: Bug: Redis search returns None for every field that declares a storage_name, because RETURN lists the declared namePossibly taken @wasim-builds claimed this 9 days ago. Openpython triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
microsoft/semantic-kernel#14753 · 1 comment ·
Maintainers usually reply within 3 days
-
.Net: gpt-image-1 is the default image model in the .NET OpenAI connector, and OpenAI shuts it down on October 23Possibly taken @nightcityblade claimed this 46 days ago. Open.NET triage
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
microsoft/semantic-kernel#14526 · 1 comment ·
Maintainers usually reply within 3 days
-
Python: VolatileMemoryStore.get_batch and get_nearest_matches ignore with_embeddings=False (deepcopy result is discarded)Possibly taken @VANDRANKI claimed this 48 days ago. Openpython triage
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
microsoft/semantic-kernel#14522 ·
Maintainers usually reply within 3 days
-
.Net: Bug: BinaryContent does not decode the %xx escapes of a non-base64 data URIPossibly taken @Laurianti claimed this 48 days ago. Open.NET triage
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
microsoft/semantic-kernel#14518 ·
Maintainers usually reply within 3 days
All issues in microsoft/semantic-kernel
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Support Flux ignore rules (drift detection ignore paths) in Microsoft.KubernetesConfiguration schemaOpen
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Azure/azure-resource-manager-schemas#3921 ·
Maintainers usually reply within 1 day
-
text-align: justify is ignored in HTML to PDF conversionPossibly taken @PrzemyslawKlys claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
asyncapi/modelina#2709 · 1 comment ·
Maintainers usually reply within 1 day