delta-io/delta

Unable to write metadata to delta table using scala api

Open

#857 创建于 2021年12月7日

在 GitHub 查看
 (14 评论) (9 反应) (2 负责人)Scala (2,100 fork)batch import
enhancementgood first issue

仓库指标

Star
 (8,807 star)
PR 合并指标
 (平均合并 7天 1小时) (30 天内合并 142 个 PR)

描述

Extension to #321

When using an api like the one below, the only supported method to write metadata is through the use of the sparkSession which works until a single spark session is performing multiple writes simultaneously. In this case, manipulating the spark session still doesn't provide the ability to pass accurate metadata to the table. Could we find a way to ensure metadata writes are possible at the writer level even when using the following syntax? Thanks

cc @zsxwing

DeltaTable.forPath(target.tableLocation).alias("target")
  .merge(updatesDF, mergeCondition)
  .whenMatched
  .updateAll()
  .whenNotMatched
  .insertAll()
  .execute()

Thank you

贡献者指南