apache/beam

Impossible to submit Google Dataflow job from Windows machine

Open

#18,734 opened on 2022年6月3日

GitHub で見る
 (0 comments) (0 reactions) (0 assignees)Java (4,097 forks)batch import
P4bugdataflowgood first issuerunnerswindows

Repository metrics

Stars
 (7,313 stars)
PR merge metrics
 (平均マージ 6d) (30d で 243 merged PRs)

説明

Got exception when trying to submit Dataflow job from java application running on Windows machine.

java.lang.IllegalArgumentException: Location must be local or on Cloud Storage, got C:\Users\myuser\AppData\Local\Temp\dataflow-job341260414234234.json.  Here:

https://github.com/apache/beam/blob/67f6cf2c7786e339758854efb1cf1230909e4f1d/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L746

following check is performed:  fileLocation.startsWith("/") || fileLocation.startsWith("gs://"),

which prohibits usage of any Windows-style file names.

Workaround is to provide explicit file location on GCS:

dataflowPipelineOptions.setDataflowJobFile("gs://mybucket/dataflowjob.pb");

 

Imported from Jira BEAM-4380. Original Jira may contain additional context. Reported by: kardashov.

コントリビューターガイド