json - Is there a way to get columns names of dataframe in pyspark without reading the whole dataset?

Question

I have huges datasets in my HDFS environnement, say 500+ datasets and all of them are around 100M+ rows. I want to get only the column names of each dataset without reading the whole datasets because it will take too long time to do that. My data are json formatted and I'm reading them using the classic spark json reader : spark.read.json('path'). So what's the best way to get columns names without wasting my time and memory ?

Thanks...

Answer 1

from the official doc :

If the schema parameter is not specified, this function goes through the input once to determine the input schema.

Therefore, you cannot get the column names with only the first line.
Still, you can do an extra step first, that will extract one line and create a dataframe from it, then extract the column names.