vaexio/vaex

[BUG-REPORT] Vaex dataframe access crashes asyncio event loop tasks

Open

#1 701 ouverte le 14 nov. 2021

Voir sur GitHub
 (3 commentaires) (0 réactions) (0 assignés)Python (591 forks)batch import
bughelp wanted

Métriques du dépôt

Stars
 (8 117 stars)
Métriques de merge PR
 (Aucune PR mergée en 30 j)

Description

Description I'm experiencing issues when using vaex in the context of an asyncio event loop. The repeated interaction with the data frame seems to kill auxiliary tasks in the same event loop. The following code reproduces my issue:

import asyncio
import vaex
import numpy as np

async def vaex_task(df):
    while True:
        await asyncio.sleep(0.1)
        print(df.count())

async def aux_task():
    while True:
        await asyncio.sleep(0.3)
        print("Aux task alive")

df = vaex.from_dict(dict(x=np.linspace(0, 1., 100)))

loop = asyncio.get_event_loop()
loop.create_task(vaex_task(df))
loop.create_task(aux_task())
loop.run_forever()

Example output is:

100
100
Aux task alive
100
100
100
Aux task alive
100
100
Aux task alive
100
100
100
Aux task alive
100
100
Exception in callback <TaskWakeupMethWrapper object at 0x00000156BC40C4F8>(<Future finished result=None>)
handle: <Handle <TaskWakeupMethWrapper object at 0x00000156BC40C4F8>(<Future finished result=None>)>
Traceback (most recent call last):
  File "C:\Python37\lib\asyncio\events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
RuntimeError: Cannot enter into task <Task pending coro=<aux_task() running at D:\minimal_vaex_problem.py:20> wait_for=<Future finished result=None>> while another task <Task pending coro=<vaex_task() running at D:\minimal_vaex_problem.py:16>> is being executed.
Task was destroyed but it is pending!
task: <Task pending coro=<aux_task() running at D:\minimal_vaex_problem.py:20> wait_for=<Future finished result=None>>
100
100
100
100
100
100
100
100
100
100

The time after which the exception is thrown varies from run to run, but usually happens within the first 10 iterations of aux_task

Does that look like a bug in vaex or am I doing something wrong here?

Thanks in advance for looking into this!

Software information

  • Vaex version (import vaex; vaex.__version__): {'vaex': '4.5.0', 'vaex-core': '4.5.1', 'vaex-viz': '0.5.0', 'vaex-hdf5': '0.10.0', 'vaex-server': '0.6.1', 'vaex-astro': '0.9.0', 'vaex-jupyter': '0.6.0', 'vaex-ml': '0.14.0'}
  • Vaex was installed via: pip
  • OS: Win10
  • Python: 3.7.7

Guide contributeur