Recent questions tagged Scala

0 votes

732 views

1 answer

scala - Joining Spark dataframes on the key

I have constructed two dataframes. How can we join multiple Spark dataframes ? For Example : PersonDf, ... combining PersonDf and ProfileDf? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

605 views

1 answer

scala - Derive multiple columns from a single column in a Spark DataFrame

I have a DF with a huge parseable metadata as a single string column in a Dataframe, lets call it DFA, ... working or pseudo code. Thanks Sanjay See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

549 views

1 answer

scala - private[this] vs private

In Scala I see such feature as object-private variable. From my not very rich Java background I learnt to close ... it for some special case. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

521 views

1 answer

scala - I want to get the type of a variable at runtime

I want to get the type of a variable at runtime. How do I do this? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

691 views

1 answer

scala - Reduce, fold or scan (Left/Right)?

When should I use reduceLeft, reduceRight, foldLeft, foldRight, scanLeft or scanRight? I want an intuition/ ... with some simple examples. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - Spark Dataframe :How to add a index Column : Aka Distributed Data Index

I read data from a csv file ,but don't have index. I want to add a column from 1 to row's number. What should I do,Thanks (scala) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

915 views

1 answer

scala - Encode an ADT / sealed trait hierarchy into Spark DataSet column

If I want to store an Algebraic Data Type (ADT) (ie a Scala sealed trait hierarchy) within a Spark ... .apache.spark.sql.DataSet[Occupation] See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

531 views

1 answer

scala - Why is it possible to declare variable with same name in the REPL?

scala> val hi = "Hello "e" hi: String = Hello "e scala> val hi = "go" hi: String = go Within ... This error i understood we cannot reassign val See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

601 views

1 answer

scala - Operate on neighbor elements in RDD in Spark

As I have a collection: List(1, 3,-1, 0, 2, -4, 6) It's easy to make it sorted as: ... partition and collect results on each partition together? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

713 views

1 answer

scala - Why does partition parameter of SparkContext.textFile not take effect?

scala> val p=sc.textFile("file:///c:/_home/so-posts.xml", 8) //i've 8 cores p: org.apache.spark.rdd ... UI even though the spark-shell prints 8. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

670 views

1 answer

scala - Spark: Transpose DataFrame Without Aggregating

I have looked at a number of questions online, but they don't seem to do what I'm trying to achieve. I'm ... want to aggregate data? Thanks :) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

758 views

1 answer

scala - How to process multi line input records in Spark

I have each record spread across multiple lines in the input file(Very huge file). Ex: Id: 2 ASIN: ... each multi line record in spark? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

868 views

1 answer

scala - Replace null values in Spark DataFrame

I saw a solution here but when I tried it doesn't work for me. First I import a cars.csv file : val df = ... didn't change. Can anyone help me ? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

724 views

1 answer

scala - Apache Spark, add an "CASE WHEN ... ELSE ..." calculated column to an existing DataFrame

I'm trying to add an "CASE WHEN ... ELSE ..." calculated column to an existing DataFrame, using Scala ... How should I implement this logic? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

629 views

1 answer

scala - No configuration setting found for key 'akka.version'

I am learning akka-remoting and this is how my project looks The project structure looks like project/pom.xml project ... }</scalaVersion> </configuration> </plugin> </plugins>...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

631 views

1 answer

scala - Forward References - why does this code compile?

Consider this snippet: object A { val b = c val c = "foo" } println( A.b ) // prints "null" As part ... to the (later) declaration of 'c'. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

597 views

1 answer

scala - Java 8 Stream, getting head and tail

Java 8 introduced a Stream class that resembles Scala's Stream, a powerful lazy construct using which it is ... to achieve this? Thanks. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

602 views

1 answer

scala - Left Anti join in Spark?

I've defined two tables like this: val tableName = "table1" val tableName2 = "table2" val format = new ... Like using subtract, except etc. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

706 views

1 answer

scala - Accessing the underlying ActorRef of an akka stream Source created by Source.actorRef

I'm trying to use the Source.actorRef method to create an akka.stream.scaladsl.Source object. Something of the ... for your review and response. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

588 views

1 answer

scala - Explicit self-references with no type / difference with ''this''

I understand the use for explicitly typed self-references: trait T { self : T2 => ... } In the body, self is ... is not just an alias for this? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

553 views

1 answer

scala - What is the difference between `##` and `hashCode`?

What is the difference between methods ## and hashCode? They seem to be outputting the same values no matter which ... it cannot find symbol ##. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Escape a dollar sign in string interpolation

How do I escape a dollar sign in string interpolation? def getCompanion(name: String) = Class.forName(s"my. ... "error: unclosed string literal" See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

716 views

1 answer

scala string.split does not work

Following is my REPL output. I am not sure why string.split does not work here. val s = "Pedro|groceries|apple|1.42" s: ... e, |, 1, ., 4, 2) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

705 views

1 answer

scala - Spark UDF with varargs

Is it an only option to list all the arguments up to 22 as shown in documentation? https://spark.apache.org/ ... 2 arguments at the time) Thanks See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

702 views

1 answer

scala - Apache Spark: Get number of records per partition

I want to check how can we get information about each partition such as total no. of records in each partition ... log or print on the console. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

445 views

1 answer

scala - Why is the Aux technique required for type-level computations?

I'm pretty sure I'm missing something here, since I'm pretty new to Shapeless and I'm learning, but ... something in my simple example. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

656 views

1 answer

scala - Doing something before or after all Scalatest tests

I have a suite of scalatest tests that test different endpoints of a RESTful API. I really want them separated ... -- shut down http server See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

628 views

1 answer

scala - How to apply a function to a tuple?

This should be an easy one. How do I apply a function to a tuple in Scala? Viz: scala> def f (i : Int, ... ^ scala> grr! Many thanks in advance. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags