Recent questions tagged PySpark

0 votes

616 views

1 answer

pyspark - How to get a value from the Row object in Spark Dataframe?

for averageCount = (wordCountsDF .groupBy().mean()).head() I get Row(avg(count)=1.6666666666666667) but when ... What am I doing wrong? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

569 views

1 answer

pyspark - Spark: Merge 2 dataframes by adding row index/number on both dataframes

Q: Is there is any way to merge two dataframes or copy a column of a dataframe to another in PySpark? For ... I hope this example was clear. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

569 views

1 answer

pyspark - Apache spark dealing with case statements

I am dealing with transforming SQL code to PySpark code and came across some SQL statements. I don't know how ... Then 'callitB' else 'CallitC' See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

454 views

1 answer

pyspark mysql jdbc load An error occurred while calling o23.load No suitable driver

I use docker image sequenceiq/spark on my Mac to study these spark examples, during the study process, I upgrade ... give some advice, thanks! See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

433 views

1 answer

pyspark - Why is the fold action necessary in Spark?

I've a silly question involving fold and reduce in PySpark. I understand the difference between these two methods, ... to me) Cheers Tomas See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

pyspark - Spark load data and add filename as dataframe column

I am loading some data into Spark with a wrapper function: def load_data( filename ): df = sqlContext.read.format("com ... ). How can I do this? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

506 views

1 answer

pyspark - How do I set the driver's python version in spark?

I'm using spark 1.4.0-rc2 so I can use python 3 with spark. If I add export PYSPARK_PYTHON= ... PYSPARK_DRIVER_PYTHON=python3 didn't work. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

679 views

1 answer

pyspark - Spark gives a StackOverflowError when training using ALS

When attempting to train a machine learning model using ALS in Spark's MLLib, I kept on receiving a ... memory available did not help. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

602 views

1 answer

pyspark - Flatten Nested Spark Dataframe

Is there a way to flatten an arbitrarily nested Spark Dataframe? Most of the work I'm seeing is written for ... of Spark are also appreciated. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

699 views

1 answer

pyspark - Python Spark Cumulative Sum by Group Using DataFrame

How do I compute the cumulative sum per group specifically using the DataFrame abstraction; and in PySpark? With ... (ordered) time variable. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags