vaexio/vaex

[BUG-REPORT] Vaex dataframe access crashes asyncio event loop tasks

Open

#1,701 opened on 2021年11月14日

GitHub で見る
 (3 comments) (0 reactions) (0 assignees)Python (591 forks)batch import
bughelp wanted

Repository metrics

Stars
 (8,117 stars)
PR merge metrics
 (30d に merged PR はありません)

説明

Description I'm experiencing issues when using vaex in the context of an asyncio event loop. The repeated interaction with the data frame seems to kill auxiliary tasks in the same event loop. The following code reproduces my issue:

import asyncio
import vaex
import numpy as np

async def vaex_task(df):
    while True:
        await asyncio.sleep(0.1)
        print(df.count())

async def aux_task():
    while True:
        await asyncio.sleep(0.3)
        print("Aux task alive")

df = vaex.from_dict(dict(x=np.linspace(0, 1., 100)))

loop = asyncio.get_event_loop()
loop.create_task(vaex_task(df))
loop.create_task(aux_task())
loop.run_forever()

Example output is:

100
100
Aux task alive
100
100
100
Aux task alive
100
100
Aux task alive
100
100
100
Aux task alive
100
100
Exception in callback <TaskWakeupMethWrapper object at 0x00000156BC40C4F8>(<Future finished result=None>)
handle: <Handle <TaskWakeupMethWrapper object at 0x00000156BC40C4F8>(<Future finished result=None>)>
Traceback (most recent call last):
  File "C:\Python37\lib\asyncio\events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
RuntimeError: Cannot enter into task <Task pending coro=<aux_task() running at D:\minimal_vaex_problem.py:20> wait_for=<Future finished result=None>> while another task <Task pending coro=<vaex_task() running at D:\minimal_vaex_problem.py:16>> is being executed.
Task was destroyed but it is pending!
task: <Task pending coro=<aux_task() running at D:\minimal_vaex_problem.py:20> wait_for=<Future finished result=None>>
100
100
100
100
100
100
100
100
100
100

The time after which the exception is thrown varies from run to run, but usually happens within the first 10 iterations of aux_task

Does that look like a bug in vaex or am I doing something wrong here?

Thanks in advance for looking into this!

Software information

  • Vaex version (import vaex; vaex.__version__): {'vaex': '4.5.0', 'vaex-core': '4.5.1', 'vaex-viz': '0.5.0', 'vaex-hdf5': '0.10.0', 'vaex-server': '0.6.1', 'vaex-astro': '0.9.0', 'vaex-jupyter': '0.6.0', 'vaex-ml': '0.14.0'}
  • Vaex was installed via: pip
  • OS: Win10
  • Python: 3.7.7

コントリビューターガイド