You can't use Python format strings in Spark. Use concat
instead:
import pyspark.sql.functions as F
df_2 = df.withColumn("countryAndState", F.concat(F.col('country'), F.lit('_'), F.col('state')))
or concat_ws
, if you need to chain many columns together with a given separator:
import pyspark.sql.functions as F
df_2 = df.withColumn("countryAndState", F.concat_ws('_', F.col('country'), F.col('state')))
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…