Recent questions tagged pyspark

0 votes

464 views

1 answer

pyspark - Google Dataproc to SQL Server(based on centos 7) connection error?

I got stuck into an issue which already has wasted 3 days of mine. I have a dataproc cluster 1.5 and ... ").load() Connection Error Snapshot See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

3.6k views

1 answer

pyspark - How to divide a column by its sum in a Spark DataFrame

How can I divide a column by its own sum in a Spark DataFrame, efficiently and without immediately triggering ... solutions based on pyspark. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

984 views

1 answer

pyspark - Write each row of a spark dataframe as a separate file

I have Spark Dataframe with a single column, where each row is a long string (actually an xml file). I want to go ... can't find how to do this. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

855 views

1 answer

pyspark - How to import referenced files in ETL scripts?

I have a script which I'd like to pass a configuration file into. On the Glue jobs page, I see ... ImportError: No module named configuration). See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

871 views

1 answer

pyspark - How to list all tables in database using Spark SQL?

I have a SparkSQL connection to an external database: from pyspark.sql import SparkSession spark = SparkSession . ... that makes any difference. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

732 views

1 answer

pyspark - Write each row of a spark dataframe as a separate file

I have Spark Dataframe with a single column, where each row is a long string (actually an xml file). I want to go ... can't find how to do this. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

pyspark - Spark incremental loading overwrite old record

I have a requirement to do the incremental loading to a table by using Spark (PySpark) Here's the example: ... tool, e.g. Presto? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

pyspark - How can set the default spark logging level?

I launch pyspark applications from pycharm on my own workstation, to a 8 node cluster. This cluster also has ... level that spark starts with? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

497 views

1 answer

pyspark - Apache Spark Codegen Stage grows beyond 64 KB

I'm getting an error when I'm feature engineering on 30+ columns to create about 200+ columns. ... " grows beyond 64 KB See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

506 views

1 answer

pyspark - Apache Spark Codegen Stage grows beyond 64 KB

I'm getting an error when I'm feature engineering on 30+ columns to create about 200+ columns. ... " grows beyond 64 KB See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

pyspark - Spark cosine distance between rows using Dataframe

I have to compute a cosine distance between each rows but I have no idea how to do it using Spark API ... in Advance for all the help See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

667 views

1 answer

pyspark - Count number of duplicate rows in SPARKSQL

I have requirement where i need to count number of duplicate rows in SparkSQL for Hive tables. from pyspark import ... are 4. (for example) See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

962 views

1 answer

pyspark - Total size of serialized results of tasks is bigger than spark.driver.maxResultSize

Good day. I am running a development code for parsing some log files. My code will run smoothly if I tried ... to resolve this issue? Thanks. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

2.5k views

1 answer

pyspark - to_date fails to parse date in Spark 3.0

I am trying to parse date using to_date() but I get the following exception. SparkUpgradeException: You may get a different ... |12/1/2010 8:26| See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

862 views

1 answer

pyspark - Low JDBC write speed from Spark to MySQL

I need write about 1 million rows from Spark a DataFrame to MySQL but the insert is too slow. How can I ... table='xx', mode='overwrite') See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

727 views

1 answer

pyspark - Spark standalone configuration having multiple executors

I'm trying to setup a standalone Spark 2.0 server to process an analytics function in parallel. To do this I ... executor with 8 cores to it. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

654 views

1 answer

pyspark - Spark 2.0: Redefining SparkSession params through GetOrCreate and NOT seeing changes in WebUI

I'm using Spark 2.0 with PySpark. I am redefining SparkSession parameters through a GetOrCreate method that was ... wrong? Thanks in advance! See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

622 views

1 answer

pyspark - Using spark-submit with python main

Reading at this and this makes me think it is possible to have a python file be executed by spark-submit however I ... ? What am I doing wrong? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

435 views

1 answer

pyspark - Spark SQL security considerations

What are the security considerations when accepting and executing arbitrary spark SQL queries? Imagine the following ... "EXPLAIN" prefix in See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

669 views

1 answer

pyspark - How to TRUNCATE and / or use wildcards with Databrick

I'm trying to write a script in databricks that will select a file based on certain characters in the name of ... code to select on the file. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.4k views

1 answer

pyspark - get all the dates between two dates in Spark DataFrame

I have a DF in which I have bookingDt and arrivalDt columns. I need to find all the dates between these two dates. Sample ... ---+----------+ See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

845 views

1 answer

pyspark - get all the dates between two dates in Spark DataFrame

I have a DF in which I have bookingDt and arrivalDt columns. I need to find all the dates between these two dates. Sample ... ---+----------+ See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

814 views

1 answer

pyspark - Spark Matrix multiplication with python

I am trying to do matrix multiplication using Apache Spark and Python. Here is my data from pyspark.mllib.linalg. ... will be helpful for me. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

553 views

1 answer

pyspark - How to set up a local development environment for Scala Spark ETL to run in AWS Glue?

I'd like to be able to write Scala in my local IDE and then deploy it to AWS Glue as part of a ... since the Glue python library uses Py4J. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

458 views

1 answer

pyspark - Join two data frames, select all columns from one and some columns from the other

Let's say I have a spark data frame df1, with several columns (among which the column id) and data frame ... as a function parameter. Thanks! See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

639 views

1 answer

pyspark - Enable case sensitivity for spark.sql globally

The option spark.sql.caseSensitive controls whether column names etc should be case sensitive or not. It can ... rationale behind that advice? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

557 views

1 answer

pyspark - Spark-submit can't locate local file

I've written a very simple python script for testing my spark streaming idea, and plan to run it ... 'notebook' export SPARK_LOCAL_IP=localhost See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

pyspark - How to run arbitrary / DDL SQL statements or stored procedures using AWS Glue

Is it possible to execute arbitrary SQL commands like ALTER TABLE from AWS Glue python job? I know I can ... some ALTER commands right after. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags