Recent questions tagged Spark

0 votes

635 views

1 answer

spark streaming - Custom partiotioning of JavaDStreamPairRDD

In Spark streaming, what's the recommended way to implement a custom partiotioner on DStreams? I've ... equivalent on JavaDStreamPairRDD. Thanks See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

822 views

1 answer

spark streaming: select record with max timestamp for each id in dataframe (pyspark)

I have a dataframe with schema - |-- record_id: integer (nullable = true) |-- Data1: string (nullable ... is not supported in streaming. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

721 views

1 answer

spark reading data from mysql in parallel

Im trying to read data from mysql and write it back to parquet file in s3 with specific partitions as follows ... write partial data to parquet? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

spark + sbt-assembly: "deduplicate: different file contents found in the following"

I ran spark application and wanna pack the test classes into the fat jar. What is weird is I ran ... ) Project.inConfig(Test)(assemblySettings) See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

935 views

1 answer

spark dataframe - combine text from multiple rows in pyspark

I created a PySpark dataframe using the following code testlist = [ {"category":"A","name":"A1"}, {"category" ... best way to make this happen ? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

754 views

1 answer

spark dataframe drop duplicates and keep first

Question: in pandas when dropping duplicates you can specify which columns to keep. Is there an equivalent in Spark ... about the last row ? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

905 views

1 answer

spark createOrReplaceTempView vs createGlobalTempView

Spark Dataset 2.0 provides two functions createOrReplaceTempView and createGlobalTempView. I am not able to understand ... sparkSession.close()? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

708 views

1 answer

spark dataframe - Applying a Window function to calculate differences in pySpark

I am using pySpark, and have set up my dataframe with two columns representing a daily asset price as follows: ind = ... I am unable to see how. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags