how to suppress empty list in spark dataframe json output

Question

Here is my pyspark code:

mydf = df.groupby(key1,key2)
           .agg(collect_list(struct(key3)).alias("list2"))
             .groupby(key1)
               .agg(collect_list(struct(key2,list2)).alias("list1"))
           .selectExpr(key1,list1)

Here is the output of the above dataframe written as json:

{
  key1: val1,
  list1: [
    {
        key2: val2,
        list2: [
            {
                key3: val3
            }
        ]
    }
  ]
}

in some cases, there may not be any corresponding list2 data for key2, in which case I don't want list2 to appear in the json at all, but currently it shows empty list:

key2: val2,
list2[
  {}
]

how can I suppress empty list2? plz note: i can't filter for empty list2 records because I still want to show key2

question from:https://stackoverflow.com/questions/66054735/how-to-suppress-empty-list-in-spark-dataframe-json-output

Answer 1

Waitting for answers