vaexio/vaex

[BUG-REPORT] Export to HDF5 writes an empty (and unreadable) file to disk upon export fail

Open

#1,947 opened on Feb 23, 2022

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Python (591 forks)batch import
enhancementhelp wantedpriority: medium

Repository metrics

Stars
 (8,117 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

Thank you for reaching out and helping us improve Vaex!

Before you submit a new Issue, please read through the documentation. Also, make sure you search through the Open and Closed Issues - your problem may already be discussed or addressed.

Description When exporting a DataFrame to hdf5 format, if the export fails, a small (empty?) and unreadable hdf5 file is written to disk despite the call failing.

Software information

  • Vaex version (import vaex; vaex.__version__):
  • {'vaex': '4.8.0', 'vaex-core': '4.8.0', 'vaex-viz': '0.5.1', 'vaex-hdf5': '0.12.0', 'vaex-server': '0.8.1', 'vaex-astro': '0.9.0', 'vaex-jupyter': '0.7.0', 'vaex-ml': '0.17.0'}
  • Vaex was installed via: pip / conda-forge / from source
    • pip
  • OS:
    • Ubuntu 20.04

Additional information To reproduce:

  1. Create an empty Pandas DataFrame:
import Pandas as pd
empty_df = pd.DataFrame([], columns = ["A", "B"])
  1. Create vaex DataFrame
import vaex
empty_vaex = vaex.from_pandas(empty_df)
  1. Export vaex DataFrame to hdf5
empty_vaex.export("empty_hdf5.hdf5") # NOTE: also occurs with export_hdf5

Observe expected error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-42-4873481ec44b> in <module>
----> 1 vaex.from_pandas(empty_df).export("empty.hdf5")

~/.local/lib/python3.8/site-packages/vaex/dataframe.py in export(self, path, progress, chunk_size, parallel, fs_options, fs)
   6593             self.export_feather(path, parallel=parallel, fs_options=fs_options)
   6594         elif naked_path.endswith('.hdf5'):
-> 6595             self.export_hdf5(path, progress=progress, parallel=parallel)
   6596         elif naked_path.endswith('.fits'):
   6597             self.export_fits(path, progress=progress)

~/.local/lib/python3.8/site-packages/vaex/dataframe.py in export_hdf5(self, path, byteorder, progress, chunk_size, parallel, column_count, writer_threads, group, mode)
   6806             progressbar_write = progressbar.add("write data")
   6807             with Writer(path=path, group=group, mode=mode, byteorder=byteorder) as writer:
-> 6808                 writer.layout(self, progress=progressbar_layout)
   6809                 writer.write(
   6810                     self,

~/.local/lib/python3.8/site-packages/vaex/hdf5/writer.py in layout(self, df, progress)
     46         N = len(df)
     47         if N == 0:
---> 48             raise ValueError("Cannot layout empty table")
     49         column_names = df.get_column_names()
     50 

ValueError: Cannot layout empty table
  1. Observe that a file empty_hdf5.hdf5 is created. Attempting to load the file with varex.open("empty_hdf5.hdf5") produces the following error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/.local/lib/python3.8/site-packages/IPython/core/formatters.py in __call__(self, obj)
    343             method = get_real_method(obj, self.print_method)
    344             if method is not None:
--> 345                 return method()
    346             return None
    347         else:

~/.local/lib/python3.8/site-packages/vaex/dataframe.py in _repr_html_(self)
   4116         """Representation for Jupyter."""
   4117         self._output_css()
-> 4118         return self._head_and_tail_table()
   4119 
   4120     def __str__(self):

~/.local/lib/python3.8/site-packages/vaex/dataframe.py in _head_and_tail_table(self, n, format)
   3863         n = n or vaex.settings.display.max_rows
   3864         N = _len(self)
-> 3865         if N <= n:
   3866             return self._as_table(0, N, format=format)
   3867         else:

TypeError: '<=' not supported between instances of 'NoneType' and 'int'

Contributor guide