vespa-engine/vespa

Support feeding from Spark

オープン

#9,158 opened on 2019/04/23

 (6 件のコメント) (0 件のリアクション) (0 人の担当者)Java (561 件のフォーク)batch import
enhancementgood first issue

Repository metrics

Stars
 (4,948 個のスター)
PR merge metrics
 (平均マージ 1d 22h) (30d で 209 merged PRs)

説明

Today, the Hadoop integration tools for Vespa support Hadoop and Pig for feeding and querying Vespa. The Pig feeder is a thin wrapper around the Vespa HTTP client.

We should support feeding directly from Spark as well, to avoid Spark pipelines having to write to HDFS and run another Pig job for the actual feeding. Similarly to the Pig feeder, this could be implemented as a thin wrapper around the HTTP client.

コントリビューターガイド