scala - Replace a value in a row with a value in another row in Spark dataframe

Question

I have a dataframe like so:

id	startValue	endValue
1	null	11a
1	554	22b
2	null	33c
2	6743	44d

question from:https://stackoverflow.com/questions/65905074/replace-a-value-in-a-row-with-a-value-in-another-row-in-spark-dataframe

Answer 1

You can coalesce the nulls with the other startValue found in the same partition of id, minus 10:

import org.apache.spark.sql.expressions.Window

val df2 = df.withColumn(
    "startValue",
    coalesce($"startValue", max($"startValue").over(Window.partitionBy("id")) - 10)
)

df2.show
+---+----------+--------+
| id|startValue|endValue|
+---+----------+--------+
|  1|       544|     11a|
|  1|       554|     22b|
|  2|      6733|     33c|
|  2|      6743|     44d|
+---+----------+--------+