apache spark - Convert RDD to DataFrame using pyspark

Question

I have a file in spark with following data

Property ID|Location|Price|Bedrooms|Bathrooms|Size|Price SQ Ft|Status

i have read this file as rdd using a=sc.textFile("/FileStore/tables/realestate.txt")

Now I need to convert this rdd into dataframe. I am using the below mentioned command

d=spark.createDataFrame(a).toDF("Property ID","Location","Price","Bedrooms","Bathrooms","Size","Price SQ Ft","Status")

But i am getting an error as TypeError: Can not infer schema for type: <class 'str'>

Answer 1

You can split the column first:

d = spark.createDataFrame(a.map(lambda x: x.split('|'))).toDF("Property ID","Location","Price","Bedrooms","Bathrooms","Size","Price SQ Ft","Status")

Or equivalently, calling toDF on the RDD directly

d = a.map(lambda x: x.split('|')).toDF(["Property ID","Location","Price","Bedrooms","Bathrooms","Size","Price SQ Ft","Status"])

In fact, I'd recommend using the Spark CSV reader for this purpose, which could handle the header appropriately too:

df = spark.read.csv('/FileStore/tables/realestate.txt', header=True, inferSchema=True, sep='|')