delta-io/delta

Unable to write metadata to delta table using scala api

Open

#857 opened on 2021年12月7日

GitHub で見る
 (14 comments) (9 reactions) (2 assignees)Scala (2,100 forks)batch import
enhancementgood first issue

Repository metrics

Stars
 (8,807 stars)
PR merge metrics
 (平均マージ 7d 1h) (30d で 142 merged PRs)

説明

Extension to #321

When using an api like the one below, the only supported method to write metadata is through the use of the sparkSession which works until a single spark session is performing multiple writes simultaneously. In this case, manipulating the spark session still doesn't provide the ability to pass accurate metadata to the table. Could we find a way to ensure metadata writes are possible at the writer level even when using the following syntax? Thanks

cc @zsxwing

DeltaTable.forPath(target.tableLocation).alias("target")
  .merge(updatesDF, mergeCondition)
  .whenMatched
  .updateAll()
  .whenNotMatched
  .insertAll()
  .execute()

Thank you

コントリビューターガイド