Increase mitigation eval success rate by providing localization info in the init prompts
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 72/100
Research direction
Start at aiopslab/orchestrator/tasks/mitigation.py#L27 and inspect how mitigation task initialization builds the agent's prompts. Trace where localization information is available and determine how it should be included at the beginning of those prompts. Done means mitigation evaluations receive the localization context and failed traces no longer require the agent to locate the faulty service first.
Written by the indexing model from the issue text.
Description
Currently, the mitigation tasks have the lowest success rate(18%) when evaluated with GPT-4o-mini, ReAct.
Failed traces revealed that sometimes the agent could not even locate the faulty service correctly.
To isolate and evaluate mitigation itself, the agent should be given the localization info at the beginning.
Even if the agent manages to locate the faulty service, the localization process(reading long logs etc) takes up the context window and degrades agent performance.
- Dominant language
- Python
- Stars
- 989
- Forks
- 176
- Avg merge
- 7d 15h
- Merged PRs (30d)
- 3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/AIOpsLab
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
All issues in microsoft/AIOpsLab
Similar issues
-
triage/confirmed
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
agentscope-ai/agentscope#2775 ·
-
comp/desktop P3 type/bug
Difficulty 1/5 Under an hour Newbie friendliness 92/100
NousResearch/hermes-agent#118866 ·
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 90/100
apache/cloudstack#14222 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100