feathr-ai/feathr

[BUG] get_offline_feature ignores `parquet` output file option

Open

Aperta il 29 set 2022

Vedi su GitHub
 (2 commenti) (0 reazioni) (0 assegnatari)Scala (1929 star) (244 fork)batch import
buggood first issue

Descrizione

Willingness to contribute

No. I cannot contribute a bug fix at this time.

Feathr version

0.8.0

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 20.0): both on Linux Ubuntu 20 and Databricks
  • Python version: 3.10
  • Spark version, if reporting runtime issue:

Describe the problem

get_offline_feature always write into avro regardless of the execution config.

Tracking information

No response

Code to reproduce bug

Run:

get_offline_feature(
    execution_configurations=SparkExecutionConfiguration({
        "spark.feathr.inputFormat": "parquet",
        "spark.feathr.outputFormat": "parquet",
    }),
    ....
)

still write file as avro

What component(s) does this bug affect?

  • Python Client: This is the client users use to interact with most of our API. Mostly written in Python.
  • Computation Engine: The computation engine that execute the actual feature join and generation work. Mostly in Scala and Spark.
  • Feature Registry API: The frontend API layer supports SQL, Purview(Atlas) as storage. The API layer is in Python(FAST API)
  • Feature Registry Web UI: The Web UI for feature registry. Written in React

Guida contributor

Tech stack
pythonscala
Dominio
backenddata
Tipo issue
bug
DifficoltàQuanto dovrebbe essere impegnativa per un nuovo contributor.
3
Tempo stimatoTempo stimato per completare e verificare uno scope piccolo.
3-5 days
Stato attivitàQuanto è probabile che la issue sia ancora attiva e reviewable.
fresh
ChiarezzaQuanto chiaramente la issue descrive problema, scope e risultato atteso.
clear
Prerequisiti
basic Spark knowledgefamiliarity with Feathr offline feature generation
Adatta ai principiantiQuanto la issue è adatta a contributor alla prima esperienza.
30
Direzione di ricerca
Investigate the Python client's `get offline feature` function to see how it passes `execution configurations` to the Spark engine. Check the Spark job builder in the Scala computation engine to verify if the output format configuration is being read and applied. The bug likely lies in the translation between Python config and Spark settings. Look at relevant files such as `feathr.py` in the Python client and the Spark job builder in Scala.