Issue building from source

build/sbt と assembly の出力から始め、次に tensorframes/core.py 内のエントリーポイント tfs.block() と _auto_placeholder を追跡します。ソースからビルドした assembly と、あらかじめコンパイルされた --packages バージョンを比較し、Spark クラス RelationalGroupedDataset が利用できない理由を特定します。報告されたクラス読み込みエラーと Py4J エラーなしでデモコマンドが実行できれば完了です。

I was trying to work around the GPU requirement in the currently published package (My environment does not have GPUs) and build the package from source with a different build of tensorflow. The import works, but once I try to run a command from the tensorframes package (tfs.block() from the demo) I get a gigantic stack trace, that concludes with
raise Py4JError("Answer from Java side is empty")
Py4JError: Answer from Java side is empty
Exception in thread "Thread-90" java.lang.NoClassDefFoundError: org/apache/spark/sql/RelationalGroupedDataset
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetPublicMethods(Class.java:2902)
at java.lang.Class.getMethods(Class.java:1615)
at py4j.reflection.ReflectionEngine.getMethodsByNameAndLength(ReflectionEngine.java:367)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:319)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344)
at py4j.Gateway.invoke(Gateway.java:252)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.RelationalGroupedDataset
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 12 more
ERROR:py4j.java_gateway:Error while sending or receiving.
Traceback (most recent call last):
File "/usr/hdp/2.5.3.0-37/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 746, in send_command
raise Py4JError("Answer from Java side is empty")
Py4JError: Answer from Java side is empty
Exception in thread "Thread-91" java.lang.NoClassDefFoundError: org/apache/spark/sql/RelationalGroupedDataset
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetPublicMethods(Class.java:2902)
at java.lang.Class.getMethods(Class.java:1615)
at py4j.reflection.ReflectionEngine.getMethodsByNameAndLength(ReflectionEngine.java:367)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:319)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344)
at py4j.Gateway.invoke(Gateway.java:252)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.RelationalGroupedDataset
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 12 more
Exception in thread "Thread-92" ERROR:py4j.java_gateway:Error while sending or receiving.
Traceback (most recent call last):
File "/usr/hdp/2.5.3.0-37/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 746, in send_command
raise Py4JError("Answer from Java side is empty")
Py4JError: Answer from Java side is empty
java.lang.NoClassDefFoundError: org/apache/spark/sql/RelationalGroupedDataset
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetPublicMethods(Class.java:2902)
at java.lang.Class.getMethods(Class.java:1615)
at py4j.reflection.ReflectionEngine.getMethodsByNameAndLength(ReflectionEngine.java:367)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:319)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344)
at py4j.Gateway.invoke(Gateway.java:252)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.RelationalGroupedDataset
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 12 more
Exception in thread "Thread-93" ERROR:py4j.java_gateway:Error while sending or receiving.
Traceback (most recent call last):
File "/usr/hdp/2.5.3.0-37/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 746, in send_command
raise Py4JError("Answer from Java side is empty")
Py4JError: Answer from Java side is empty
^Cjava.lang.NoClassDefFoundError: org/apache/spark/sql/RelationalGroupedDataset
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetPublicMethods(Class.java:2902)
at java.lang.Class.getMethods(Class.java:1615)
at py4j.reflection.ReflectionEngine.getMethodsByNameAndLength(ReflectionEngine.java:367)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:319)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344)
at py4j.Gateway.invoke(Gateway.java:252)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.RelationalGroupedDataset
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 12 more
Exception in thread "Thread-94" java.lang.NoClassDefFoundError: org/apache/spark/sql/RelationalGroupedDataset
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetPublicMethods(Class.java:2902)
at java.lang.Class.getMethods(Class.java:1615)
at py4j.reflection.ReflectionEngine.getMethodsByNameAndLength(ReflectionEngine.java:367)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:319)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344)
at py4j.Gateway.invoke(Gateway.java:252)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.RelationalGroupedDataset
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 12 more
Traceback (most recent call last):
File "", line 1, in
File "/app/home/nzshiffm/tensorframes/target/scala-2.11/tensorframes-assembly-0.2.4.jar/tensorframes/core.py", line 315, in block
File "/app/home/nzshiffm/tensorframes/target/scala-2.11/tensorframes-assembly-0.2.4.jar/tensorframes/core.py", line 333, in _auto_placeholder
File "/usr/hdp/2.5.3.0-37/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 811, in call
File "/usr/hdp/2.5.3.0-37/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 631, in send_command
File "/usr/hdp/2.5.3.0-37/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 749, in send_command
File "/usr/lib64/python2.7/logging/init.py", line 1182, in exception
self.error(msg, *args, **kwargs)
File "/usr/lib64/python2.7/logging/init.py", line 1175, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib64/python2.7/logging/init.py", line 1268, in _log
self.handle(record)
File "/usr/lib64/python2.7/logging/init.py", line 1278, in handle
self.callHandlers(record)
File "/usr/lib64/python2.7/logging/init.py", line 1318, in callHandlers
hdlr.handle(record)
File "/usr/lib64/python2.7/logging/init.py", line 749, in handle
self.emit(record)
File "/usr/lib64/python2.7/logging/init.py", line 879, in emit
self.handleError(record)
File "/usr/lib64/python2.7/logging/init.py", line 802, in handleError
None, sys.stderr)
File "/usr/lib64/python2.7/traceback.py", line 125, in print_exception
print_tb(tb, limit, file)
File "/usr/lib64/python2.7/traceback.py", line 69, in print_tb
line = linecache.getline(filename, lineno, f.f_globals)
File "/usr/lib64/python2.7/linecache.py", line 14, in getline
lines = getlines(filename, module_globals)
File "/usr/lib64/python2.7/linecache.py", line 40, in getlines
return updatecache(filename, module_globals)
File "/usr/lib64/python2.7/linecache.py", line 128, in updatecache
lines = fp.readlines()
RuntimeError: maximum recursion depth exceeded while calling a Python object

The noclassdef found errors don't occur when I run with the precompiled version included with --packages. I couldn't get build/sbt tfPackage to work, there was a 'no key found tfPackage' message when I tried to run that. Have I not completed the build if I just ran build/sbt assembly?