[LogosDB] Add memory consolidation for duplicate detection
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Start in tools/cli/cli.cpp at the /teach implementation and read SPECS.md for the documented behavior. Review issue #8 and issue #10 first, then resolve which deduplication approach the project should use; done means the selected approach, CLI options, statistics, accuracy tests, and documentation satisfy the acceptance criteria.
Written by the indexing model from the issue text.
Description
Implement memory consolidation to detect and merge duplicate or similar memories.
Problem
Teaching the same fact multiple times creates duplicate entries:
id=1: "My commute is 42 minutes"
id=2: "My commute takes 42 minutes"
id=3: "It takes 42 minutes to commute"
This wastes space and can confuse the compressor.
Proposed Solution
Option 1: Pre-insert duplicate detection
- Before teaching, search for similar embeddings
- If similarity > threshold, skip or update existing
Option 2: Periodic consolidation
- Background task or manual /consolidate command
- Find clusters of similar embeddings
- Merge or keep only the best representative
Option 3: Update-on-duplicate
- If similar memory exists, update it instead of creating new
- Preserves ID, updates text/timestamp/embedding
Implementation Sketch
// Pre-insert check
auto existing = logosdb_search(mem_db, embedding, dim, 1, &err);
if (existing && logosdb_result_count(existing) > 0) {
float score = logosdb_result_score(existing, 0);
if (score > 0.95) {
// Update instead of insert
uint64_t id = logosdb_result_id(existing, 0);
logosdb_update(mem_db, id, ...);
}
}
CLI Additions
- /consolidate - Run manual deduplication
- --semantic-memory-dedup-threshold <0-1> - Similarity threshold
- --semantic-memory-auto-dedup - Enable automatic deduplication
Acceptance Criteria
- Implement similarity check before insert
- Add /consolidate command for manual cleanup
- Add deduplication threshold option
- Add auto-dedup flag option
- Report deduplication stats in /stats
- Test duplicate detection accuracy
- Document in SPECS.md
Dependencies
- Requires LogosDB v0.2.0 (issue #8)
- Benefits from memory ID tracking (issue #10)
Related Files
- tools/cli/cli.cpp - /teach implementation
- SPECS.md - documentation
- Dominant language
- C++
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from jose-compu/funes.cpp
-
stale
Difficulty 4/5 3-5 days Newbie friendliness 52/100
jose-compu/funes.cpp#11 ·
-
stale
Difficulty 4/5 3-5 days Newbie friendliness 48/100
jose-compu/funes.cpp#10 ·
-
stale
Difficulty 4/5 3-5 days Newbie friendliness 55/100
jose-compu/funes.cpp#9 ·
-
stale
Difficulty 4/5 3-5 days Newbie friendliness 45/100
jose-compu/funes.cpp#8 ·
-
stale
Difficulty 4/5 3-5 days Newbie friendliness 48/100
jose-compu/funes.cpp#7 ·
All issues in jose-compu/funes.cpp
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
hyprwm/aquamarine#426 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
amnezia-vpn/amnezia-client#3222 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
sdatkinson/NeuralAmpModelerCore#342 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
valkey-io/valkey-search#1465 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
KhronosGroup/Vulkan-Tutorial#524 ·
Maintainers usually reply within 1 day