Running BlueAPI with NumTracker not running causes error difficult to diagnose

Bắt đầu với blueapi/utils/numtracker.py, đặc biệt là create_scan, và theo dõi việc sử dụng nó từ blueapi/core/context.py trong traceback. Tái hiện quá trình scan khi NumTracker không khả dụng và kiểm tra cách xử lý lỗi hiện có; công việc được xem là hoàn tất khi lỗi xác định rõ NumTracker và vấn đề kết nối liên quan, đồng thời vẫn giữ nguyên hành vi hiện có khi dịch vụ khả dụng.

When trying to run blueapi locally using the following configuration

api:
  url: http://0.0.0.0:8000
env:
  metadata:
    instrument: i06
  sources:
    - kind: deviceManager
      module: dodal.beamlines.i06_1
      mock: True
    - kind: planFunctions
      module: dodal.plans
    - kind: planFunctions
      module: dodal.plan_stubs.wrapped
stomp:
  enabled: true
  url: tcp://localhost:61613/
 numtracker:
   url: http://localhost:8406/graphql

uv run blueapi -c /path/to/blueapi_source.yaml serve

And then running this with blueapi python client, when trying to run a plan such as moving a device it runs okay. However, when trying to run a scan, an error occurs that is not obvious what is causing it.

Client side

>>> pl.step_grid_scan([], params=[(scmc.cart.x, [0, 1, 0.1])])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/scratch/bluesky_development/blueapi/src/blueapi/client/client.py", line 159, in __call__
    raise PlanFailedError(typ, msg)
blueapi.client.client.PlanFailedError: All connection attempts failed

Server side

2026-08-04 15:43:43,360     INFO blueapi.worker.task_worker Submitting: task_id='ef0c21cc-bdb1-49e4-8851-caad0170ffeb' task=Task(name='step_grid_scan', params={'detectors': [], 'params': [['scmc.cart.x', [0, 1, 0.1]]]}, metadata={'user': 'Unknown', 'instrument_session': 'cm44150-1'}) request_id=None is_complete=False is_pending=True errors=[] outcome=None
2026-08-04 15:43:43,360     INFO blueapi.worker.task_worker Got new task: task_id='ef0c21cc-bdb1-49e4-8851-caad0170ffeb' task=Task(name='step_grid_scan', params={'detectors': [], 'params': [['scmc.cart.x', [0, 1, 0.1]]]}, metadata={'user': 'Unknown', 'instrument_session': 'cm44150-1'}) request_id=None is_complete=False is_pending=True errors=[] outcome=None
2026-08-04 15:43:43,360     INFO blueapi.worker.task Asked to run plan step_grid_scan with {'detectors': [], 'params': [['scmc.cart.x', [0, 1, 0.1]]]} and metadata {'user': 'Unknown', 'instrument_session': 'cm44150-1'} for all runs
2026-08-04 15:43:43,360     INFO bluesky Executing plan <generator object step_grid_scan at 0x7fb259eecf20>
2026-08-04 15:43:43,360     INFO bluesky.RE.state Change state on <bluesky.run_engine.RunEngine object at 0x7fb29bd2fc90> from 'idle' -> 'running'
2026-08-04 15:43:43,362     INFO blueapi.service.main 127.0.0.1:35674 PUT /worker/task 200
2026-08-04 15:43:43,368    ERROR bluesky Run aborted
Traceback (most recent call last):
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
    raise exc
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_async/connection.py", line 78, in handle_async_request
    stream = await self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_async/connection.py", line 124, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp
    return await self._backend.connect_tcp(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_backends/anyio.py", line 113, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/lib64/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/run_engine.py", line 1616, in _run
    msg = self._plan_stack[-1].send(resp)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/dodal/plans/wrapped.py", line 509, in step_grid_scan
    yield from bp.list_grid_scan(
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/plans.py", line 356, in list_grid_scan
    return (yield from scan_nd(detectors, full_cycler, per_step=per_step, md=_md))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/plans.py", line 1168, in scan_nd
    return (yield from inner_scan_nd())
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/utils/__init__.py", line 1351, in dec_inner
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/preprocessors.py", line 1013, in stage_wrapper
    return (yield from finalize_wrapper(inner(), unstage_devices()))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/preprocessors.py", line 548, in finalize_wrapper
    ret = yield from plan
          ^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/preprocessors.py", line 1011, in inner
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/utils/__init__.py", line 1351, in dec_inner
    return (yield from plan)
            ^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/preprocessors.py", line 362, in run_wrapper
    rs_uid = yield from open_run(md)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/utils/__init__.py", line 2047, in __iter__
    return (yield from self._iter)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/plan_stubs.py", line 1366, in open_run
    return (yield Msg("open_run", **(md or {})))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/run_engine.py", line 1683, in _run
    new_response = await coro(msg)
                   ^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/run_engine.py", line 1855, in _open_run
    self.md["scan_id"] = await maybe_await(self.scan_id_source(self.md))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/utils/__init__.py", line 2029, in maybe_await
    return await ret
           ^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/src/blueapi/core/context.py", line 159, in _update_scan_num
    scan = await numtracker.create_scan(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/src/blueapi/utils/numtracker.py", line 98, in create_scan
    response = await client.post(
               ^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1859, in post
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1540, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1629, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1730, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/lib64/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed

@tpoliaw managed to help me fix it by commenting out numtracker configuration and then the scan would work again, but this traceback is not obvious at all what is causing the issue. Is it possible we could make the traceback user friendly mentioning num tracker?