router eval: fixed 60 s embedding request timeout fails full runs on slower machines
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 67/100
Research direction
Start with systems/superintelligence/scripts/meta_select.py and inspect _embed_at, then trace how evals/routing/score.py --mode auto calls it. The issue offers two possible fixes, so confirm the intended approach with the maintainers before changing the timeout or retry behavior. Done when full router evaluations tolerate slower embedding requests while preserving the refusal to write a keyword score as the router score.
Written by the indexing model from the issue text.
Description
What happens
systems/superintelligence/scripts/meta_select.py sends every embedding request with a fixed 60 second timeout (_embed_at). A full router eval (evals/routing/score.py --mode auto) sends one request per prompt with the prompt plus the team profiles. On a slower or busy machine one of those requests can take longer than 60 seconds, and the whole run stops with:
embedding server failed during scoring at <url>: TimeoutError: timed out
refusing to write a keyword score as the router top-1
Re-running the same command on a quieter machine passes, so the result depends on machine load rather than the router.
Suggested fix
Either of these would do:
- read the timeout from an environment variable (for example
COCO_EMBED_TIMEOUT, default 60) so slower machines can raise it; - add a solo/serial mode to
score.pythat retries a timed-out request once before failing the run.
The refusal to write a keyword score as the router score should stay as it is.
- Dominant language
- HTML
- Stars
- 347
- Forks
- 12
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 95
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from coco-research/coco
-
bug P2
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
coco-research/coco#539 ·
Maintainers usually reply within 1 day
-
bug P2
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
coco-research/coco#538 ·
Maintainers usually reply within 1 day
-
bug P2
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
coco-research/coco#537 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
coco-research/coco#530 ·
Maintainers usually reply within 1 day
-
check-asset-counts: bare 149 tripwire collides with a real active-assets count of 149Possibly taken @aryankalantri claimed this 1 day ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
coco-research/coco#473 ·
Maintainers usually reply within 1 day
All issues in coco-research/coco
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Validate the Ollama AI provider with a real serverPossibly taken @visy-ani claimed this today. Opengood first issue help wanted
Difficulty 2/5 Half a day Newbie friendliness 70/100
Rullst/Rullst#812 · 1 comment ·
Maintainers usually reply within 1 day
-
Expose session state keys on the Go todo and agent-mode providersPossibly taken @gabisonia claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
microsoft/agent-framework-go#1412 ·
Maintainers usually reply within 1 day
-
[Bug]: hf_model_if_cache retry decorator lacks reraise=True, surfaces RetryError instead of the real exceptionPossibly taken @Shizoqua claimed this today. Openbug triage
Difficulty 1/5 Under an hour Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
area: agents-mcp priority: normal
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
Maintainers usually reply within 1 day