demo does not work in pyCharm

Open
  • Is something wrong?
#58 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
python, spark, tensorflow
Domain
data, distributed-systems, machine-learning

Research direction

Start with the demo in tensor_frame_test.py and trace tfs.block through tensorframes/core.py, especially _auto_placeholder and _java_api. Check how the supplied tensorframes-0.2.3-s_2.10.jar is loaded and investigate the reported missing org.tensorframes.impl.DebugRowOps class. Done means the demo reaches df2.collect() without the ClassNotFoundException.

Written by the indexing model from the issue text.

Description

help wanted

i'm currently trying to run the demo in pyCharm my file looks like this:
the attached pyFile is the location of my tensorframes jar

import tensorflow as tf
from pyspark.sql import Row
from pyspark.sql import SparkSession

spark = SparkSession
.builder
.appName("PythonSQL")
.getOrCreate()

spark.sparkContext.addPyFile('/home/hdiuser/PycharmProjects/TensorFramesTest/tensorframes-0.2.3-s_2.10.jar')
import tensorframes as tfs

data = [Row(x=float(x)) for x in range(10)]
df = spark.createDataFrame(data)
print df.show()
with tf.Graph().as_default() as g:
#The TensorFlow placeholder that corresponds to column 'x'.
#The shape of the placeholder is automatically inferred from the DataFrame.
x = tfs.block(df, "x")
# The output that adds 3 to x
z = tf.add(x, 3, name='z')
# The resulting dataframe
df2 = tfs.map_blocks(z, df)

The transform is lazy as for most DataFrame operations. This will trigger it:

df2.collect()

the output is as follows:

| x|
+---+
|0.0|
|1.0|
|2.0|
|3.0|
|4.0|
|5.0|
|6.0|
|7.0|
|8.0|
|9.0|
+---+

None
Traceback (most recent call last):
File "/home/hdiuser/PycharmProjects/TensorFramesTest/tensor_frame_test.py", line 40, in
x = tfs.block(df, "x")
File "/tmp/spark-8785f696-946f-4b70-8c4f-11d5b1e1ecbc/userFiles-bb752893-c175-4844-968e-5ee426ef7e03/tensorframes-0.2.3-s_2.10.jar/tensorframes/core.py", line 315, in block
File "/tmp/spark-8785f696-946f-4b70-8c4f-11d5b1e1ecbc/userFiles-bb752893-c175-4844-968e-5ee426ef7e03/tensorframes-0.2.3-s_2.10.jar/tensorframes/core.py", line 333, in _auto_placeholder
File "/tmp/spark-8785f696-946f-4b70-8c4f-11d5b1e1ecbc/userFiles-bb752893-c175-4844-968e-5ee426ef7e03/tensorframes-0.2.3-s_2.10.jar/tensorframes/core.py", line 30, in _java_api
File "/home/hdiuser/anaconda2/envs/tensorflow/lib/python2.7/site-packages/py4j/java_gateway.py", line 1133, in call
answer, self.gateway_client, self.target_id, self.name)
File "/usr/hdp/current/spark-client/python/pyspark/sql/utils.py", line 63, in deco
return f(_a, *_kw)
File "/home/hdiuser/anaconda2/envs/tensorflow/lib/python2.7/site-packages/py4j/protocol.py", line 319, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o67.loadClass.
: java.lang.ClassNotFoundException: org.tensorframes.impl.DebugRowOps
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:211)
at java.lang.Thread.run(Thread.java:745)

Dominant language
Scala
Stars
743
Forks
158
No merged PRs in 30d

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/tensorframes

All issues in databricks/tensorframes

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.