I'm trying to capture the string representation generated by the show()
function as suggested here, but it seems like the showString
method is no longer public.
logger.info('
{}'.format(raw_data._jdf.showString(20, False, False)))
is returning the following error trace
Traceback (most recent call last):
File "xxxxxxxx/SparkTest/main.py", line 63, in <module>
logger.info('
{}
'.format(raw_data._jdf.showString(20, False, False)))
File "xxxxxxxx/envs/SparkTest/lib/python3.8/site-packages/py4j/java_gateway.py", line 1304, in __call__
return_value = get_return_value(
File "xxxxxxxx/envs/SparkTest/lib/python3.8/site-packages/pyspark/sql/utils.py", line 128, in deco
return f(*a, **kw)
File "xxxxxxxx/envs/SparkTest/lib/python3.8/site-packages/py4j/protocol.py", line 330, in get_return_value
raise Py4JError(
py4j.protocol.Py4JError: An error occurred while calling o45.showString. Trace:
py4j.Py4JException: Method showString([class java.lang.Integer, class java.lang.Boolean, class java.lang.Boolean]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
at py4j.Gateway.invoke(Gateway.java:274)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:832)
is there another alternative to this approach?
question from:
https://stackoverflow.com/questions/65836215/dataframe-show-string-representation-fails-with-showstringinteger-boolean-boo 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…