delta-io/delta

Unable to write metadata to delta table using scala api

Open

#857 建立於 2021年12月7日

在 GitHub 查看
 (14 留言) (9 反應) (2 負責人)Scala (2,100 fork)batch import
enhancementgood first issue

倉庫指標

Star
 (8,807 star)
PR 合併指標
 (平均合併 7天 1小時) (30 天內合併 142 個 PR)

描述

Extension to #321

When using an api like the one below, the only supported method to write metadata is through the use of the sparkSession which works until a single spark session is performing multiple writes simultaneously. In this case, manipulating the spark session still doesn't provide the ability to pass accurate metadata to the table. Could we find a way to ensure metadata writes are possible at the writer level even when using the following syntax? Thanks

cc @zsxwing

DeltaTable.forPath(target.tableLocation).alias("target")
  .merge(updatesDF, mergeCondition)
  .whenMatched
  .updateAll()
  .whenNotMatched
  .insertAll()
  .execute()

Thank you

貢獻者指南