daphne-project/daphne

Read from parquet does not work

Aberta

#587 aberto em 9 de ago. de 2023

 (7 comentários) (0 reação) (0 responsável)C++ (84 forks)auto 404
good first issue

Métricas do repositório

Stars
 (80 estrelas)
Métricas de merge de PR
 (Nenhuma PRs mesclada em 30d)

Description

Parquet file is existing and can be read with https://parquet-viewer-online.com

Execution of DSL script results in printing a frame with all (10) values nan

// read_from_parquet.daph
group_data = readFrame("/path/to/data.parquet");

print(group_data);

OUTPUT:

Frame(5x2, [Id:double, Vds:double])
nan nan
nan nan
nan nan
nan nan
nan nan
// /path/to/data.parquet.meta
{
  "numRows": 5,
  "numCols": 2,
  "schema": [
    {
      "label": "Id",
      "valueType": "f64"
    },
    {
      "label": "Vds",
      "valueType": "f64"
    }
  ]
}

This is how I created the parquet file:

import pyarrow.parquet as pq
from pathlib import Path
import pyarrow.csv as csv

def main(path: str):

    p = Path(path)
    ro = csv.ReadOptions(column_names = ["Id", "Vds"])
    table = csv.read_csv(p, read_options=ro)    

    print("Arrow table from csv ----------------------------------------------------------------------------")
    print(f"Num cols in table: {table.num_columns}")
    print(f"Num rows in table: {table.num_rows}")
    print(table)
    print("Writing arrow table -----------------------------------------------------------------------------")
    destpth = p.with_suffix(".parquet")
    print(f"writing to {destpth.resolve()}")
    pq.write_table(table, destpth, use_dictionary=False)
    print("Reading back again from parquet file-------------------------------------------------------------")
    rtable=pq.read_table(p.with_suffix(".parquet"))
    print(rtable)
    print("pandas repr.:")
    print(rtable.to_pandas())

from a csv file which looks like this:

0.1184805,4.2727
0.026556,4.2356
-0.0653686,4.248
-0.0347271,4.248
-0.0040855,4.6257

Sample parquet files can be acquired here (data is much more complex here): https://github.com/kaysush/sample-parquet-files/tree/main

Guia do colaborador