apache spark - Caching dataframes while keeping partitions

Question

I'm on Spark 2.2.0, running on EMR.

I have a big dataframe df (40G or so in compressed snappy files) which is partitioned by keys k1 and k2.

When I query by k1 === v1 or (k1 === v1 && k2 ===v2`), I can see that it's only querying the files in the partition (about 2% of the files).

However if I cache or persist df, suddenly those queries are hitting all the partitions and either blows up memory or is much less performant.

This is a big surprise - is there any way to do caching which preserves the partitoning information

Answer 1

This is to be expected. Spark internal columnar format used for caching is input format agnostic. Once you loaded data there there is no connection to the original input is gone.

The exception here is new data source API [SPARK-22389][SQL] data source v2 partitioning reporting interface, which allows for persisting partitioning information, but it is new in 2.3 and still experimental.