Recent questions tagged rdd

0 votes

351 views

1 answer

rdd - Spark Mlib FPGrowth job fails with Memory Error

I have a fairly simple use case, but potentially very large result set. My code does the following (on ... support threshold are valid. -Raj See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

399 views

1 answer

rdd - Spill to disk and shuffle write spark

I'm getting confused about spill to disk and shuffle write. Using the default Sort shuffle manager, we use an ... that phase. Thanks. Giorgio See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

318 views

1 answer

rdd - Difference between sc.textFile and spark.read.text in Spark

I am trying to read a simple text file into a Spark RDD and I see that there are two ways of doing ... circumstances should we use which method? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

484 views

1 answer

rdd - How spark read a large file (petabyte) when file can not be fit in spark's main memory

What will happen for large files in these cases? 1) Spark gets a location from NameNode for data . Will Spark ... load other data from disc. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

382 views

1 answer

rdd - Why does sortBy transformation trigger a Spark job?

As per Spark documentation only RDD actions can trigger a Spark job and the transformations are lazily evaluated when ... in the SparkUI. Why? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags