reworkd/AgentGPT

✨ Investigate the best similarity score threshold to remove duplicate tasks

Open

#729 opened on Jun 6, 2023

View on GitHub
 (0 comments) (0 reactions) (0 assignees)TypeScript (34,594 stars) (9,446 forks)batch import
enhancementhelp wanted

Description

When we generate tasks, we filter tasks that have a similarity score that is too close to existing tasks in the vector database

similar_tasks = memory.get_similar_tasks(
    task, score_threshold=0.95  # TODO: Once we use ReAct, revisit
)

This is done with the help of the code above. Arbitrarily, we use 0.95. Even with this, the task may not be related.

On the other hand, there may be very related / duplicated tasks that have a score that is less than this.

This ticket is tasked with investigating what the best value for this threshold is, or to use some other means of calculating similarity for this given case.

Contributor guide

Tech stack
typescript
Domain
aidatabackend
Issue type
research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
similarity metricsvector databasesAgentGPT codebase
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Investigate the function memory.get similar tasks in the codebase to understand current implementation. Experiment with different score threshold values (e.g., 0.9, 0.8, 0.7) and evaluate the impact on task deduplication using a test dataset. Analyze the trade offs between false positives and false negatives and propose a recommended threshold or alternative similarity approach.