ohs-foundation/fhir-data-pipes

Evaluate Presto/Trino as a query engine on the generated Parquet files

Open

#257 opened on Apr 13, 2022

View on GitHubContributor guide
 (6 comments) (0 reactions) (0 assignees)Jupyter Notebook (124 forks)auto 404
P3:TBEenhancementgood first issue

Repository metrics

Stars
 (222 stars)
PR merge metrics
 (PR metrics pending)

Description

Presto is a query engine to run interactive SQL queries in a distributed environment. It is open-source and seems to resemble some of the features of BigQuery (I have not used Presto myself).

This issue is to track experimentation with Presto on top of Parquet files that we generate for FHIR resources. Here is an incomplete list of things we like to try:

  1. Is it possible to use Preston to query Parquet files locally? If yes, document the setup.
  2. How does the distributed environment with many nodes need to be set up?
  3. How does it perform compared to Spark, e.g., run same queries on the same Parquet files through Presto and Spark to compare.
  4. How transferable is the Presto SQL queries to other distributed SQL engines like BigQuery or Spark SQL; particular attention needs to be paid to queries involving nested and repeated fields of FHIR.
  5. How well the integration of Presto and Spark works.

Update May 2024: In 2019, Trino was forked from Presto and it seems it has a faster pace of development. We should evaluate Trino too.

Contributor guide

Research direction
Set up Presto or Trino locally, run sample queries on generated Parquet files, and compare with Spark. Document the setup and query compatibility.
Tech stack
sql
Domain
backenddata
Issue type
Research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
SQL
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40