Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.1k views
in Technique[技术] by (71.8m points)

pyspark - DataFrame show string representation fails with showString(Integer, Boolean, Boolean) does not exist

I'm trying to capture the string representation generated by the show() function as suggested here, but it seems like the showString method is no longer public.

logger.info('
{}'.format(raw_data._jdf.showString(20, False, False)))

is returning the following error trace

Traceback (most recent call last):
  File "xxxxxxxx/SparkTest/main.py", line 63, in <module>
    logger.info('
{}
'.format(raw_data._jdf.showString(20, False, False)))
  File "xxxxxxxx/envs/SparkTest/lib/python3.8/site-packages/py4j/java_gateway.py", line 1304, in __call__
    return_value = get_return_value(
  File "xxxxxxxx/envs/SparkTest/lib/python3.8/site-packages/pyspark/sql/utils.py", line 128, in deco
    return f(*a, **kw)
  File "xxxxxxxx/envs/SparkTest/lib/python3.8/site-packages/py4j/protocol.py", line 330, in get_return_value
    raise Py4JError(
py4j.protocol.Py4JError: An error occurred while calling o45.showString. Trace:
py4j.Py4JException: Method showString([class java.lang.Integer, class java.lang.Boolean, class java.lang.Boolean]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
    at py4j.Gateway.invoke(Gateway.java:274)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.base/java.lang.Thread.run(Thread.java:832)

is there another alternative to this approach?

question from:https://stackoverflow.com/questions/65836215/dataframe-show-string-representation-fails-with-showstringinteger-boolean-boo

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

The error says that the method showString(Integer, Boolean, Boolean) does not exist. If you look at showString source code, it takes Integer as second argument but you're passing a boolean.

You want to set truncate=False so simply pass 0 in the second argument:

logger.info('
{}'.format(raw_data._jdf.showString(20, 0, False)))

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...