Projects: a restart can deliver a worker's update to the coordinator again

Open
#6,817 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
backend

Research direction

Start in reconcile_coordinator_updates in project_wake.py, where the delivered-update check reads only _TURN_LOOKBACK (50) user-source messages from the coordinator. The reproduction test in tests/agent_server/test_projects_router.py, test_reconcile_after_fifty_newer_messages, is the target: it currently gets 1 where it should get 0. Choosing between a durable per-turn delivery marker and paging with next_page_id is a design decision, so confirm the direction with maintainers first. Done means that test passes and the existing reconcile tests still pass. This work depends on the unmerged PR #6602.

Written by the indexing model from the issue text.

Description

Problem

This is in the Projects code on #6602, which hasn't merged yet. When the agent server starts, reconcile_coordinator_updates (in project_wake.py) finds updates that never reached a coordinator. Each project member's latest turn was started by the coordinator, so it checks every member whose turn has ended. An update counts as delivered when the coordinator holds a message from that member that was saved after the member's last output. That check reads only the coordinator's 50 newest user-source messages (_TURN_LOOKBACK) and does not page further.

Take a worker that finished and then sat idle while the coordinator received 50 or more other messages, such as other workers' updates, the user's own messages, or copies of the user's messages in workers. That worker's update has already been delivered, but it now falls outside the page. Every restart therefore delivers the update again and wakes the coordinator.

This came up in a review of #6602. It was decided to leave it out of scope for the current iteration of Projects.

Reproduction

Here is a test for tests/agent_server/test_projects_router.py on the #6602 branch. It fails there: the reconcile delivers 1 update instead of 0, and the coordinator's message count goes from 51 to 52.

@pytest.mark.asyncio
async def test_reconcile_after_fifty_newer_messages(
    client, service, tmp_path, monkeypatch
):
    cat = await _start(service, tmp_path)
    worker = await _start(service, tmp_path, title="worker")
    project = (
        await client.post(
            "/api/projects", json={"name": "P", "coordinator_conversation_id": cat.hex}
        )
    ).json()
    await client.put(f"/api/projects/{project['id']}/conversations/{worker.hex}")
    _record_runs(monkeypatch)
    worker_service = await service.get_event_service(worker)
    cat_service = await service.get_event_service(cat)
    from openhands.agent_server.project_board import ConversationOutcomeSubscriber

    async def unseen(self, event) -> None:
        return None

    monkeypatch.setattr(ConversationOutcomeSubscriber, "__call__", unseen)
    await worker_service.send_message(
        Message(role="user", content=[TextContent(text="Pick a database.")]),
        sender=f"conversation:{cat.hex}",
    )
    await _agent_says(service, worker, "Postgres.", ConversationExecutionStatus.FINISHED)
    assert await reconcile_coordinator_updates(service) == 1
    for i in range(50):
        await cat_service.send_message(
            Message(role="user", content=[TextContent(text=f"user note {i}")])
        )
    assert await reconcile_coordinator_updates(service) == 0  # fails: 1

Direction

The preferred fix is a durable delivery marker: record which turn of a member has been delivered to its coordinator. Restart reconciliation can then compare against that marker instead of searching the coordinator's history.

Paging through the coordinator's messages with next_page_id until a match is found also makes the test pass. It is a smaller interim fix, but every restart pays for it in proportion to the coordinator's history.

Dominant language
Python
Stars
1.4k
Forks
797
Avg merge
1d 17h
Merged PRs (30d)
354

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenHands/software-agent-sdk

All issues in OpenHands/software-agent-sdk

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.