Recent questions tagged Scala

0 votes

1.2k views

1 answer

scala - Function literal with multiple implicit arguments

How to define function literal with multiple implicit arguments in Scala? I've tried this way: def create = ... but it throws compilation error. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - How to test a Try[T] with ScalaTest correctly?

I have a method that returns a Try object: def doSomething(p: SomeParam): Try[Something] = { // code } I ... my approach really the way to go? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - Error initializing SparkContext: A master URL must be set in your configuration

I used this code My error is: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 17 ... finished with exit code 1 See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - Using JUnit @Rule with ScalaTest (e.g. TemporaryFolder)

I would like to be able to use JUnit rules such as TemporaryFolder or other TestRules we have already developed in- ... not yet been created } } See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Inconsistent behaviour for xs.sliding(n) if n is less than size?

According to scaladoc, sliding() returns... "An iterator producing iterable collections of size size, except ... implemented this way instead? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.4k views

1 answer

scala - How to escape column names with hyphen in Spark SQL

I have imported a json file in Spark and convertd it into a table as myDF.registerTempTable("myDF") I then want ... I resolve this is Spark SQL? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Apache Flink: ProcessWindowFunction KeyBy() multiple values

I'm trying to use WindowFunction with DataStream, my goal is to have a Query like the following SELECT *, count( ... c_city: Int, c_addr: Int ) See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Spark Shell Add Multiple Drivers/Jars to Classpath using spark-defaults.conf

We are using Spark-Shell REPL Mode to test various use-cases and connecting to multiple sources/sinks We ... provide details for correct syntax See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

scala - Unnecessary recompilations by SBT

I have a question about seemingly unnecessary recompilations by SBT. I have the following scenario: I'm running SBT inside ... 0_77, SBT 0.13.11 See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - How to replace empty values in a column of DataFrame?

How can I replace empty values in a column Field1 of DataFrame df? Field1 Field2 AA 12 BB This command does ... Field1 Field2 Anonymous AA 12 BB See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Using ReduceByKey to group list of values

I want to group list of values per key and was doing something like this: sc.parallelize(Array(("red", "zero") ... ) (red,zerotwo) (yellow,one) See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

scala - Why does RDD.foreach fail with "SparkException: This RDD lacks a SparkContext"?

I have a dataset (as an RDD) that I divide into 4 RDDs by using different filter operators. val RSet = ... more information, See SPARK-13758. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

scala - problem with implicit ambiguity between my method and conforms in Predef

The following code, which is taken from Apocalisp's excellent blog series: Type level programming in scala , and modified ... ,HNil](HNil) } } See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - How to order execution of tests in sbt?

Please suggest best approach how to control order of test/spec execution in sbt? Is there any option like runOrder in maven-sirefire-plugin See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - spark apply function to columns in parallel

Spark will process the data in parallel, but not the operations. In my DAG I want to call a function per column ... ]]](colName)).toMap }.toMap See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

scala - How to create a wrapper of List with a specific type

I am trying to create a wrapper of List with a specific type (e.g. List[Int]) such that ... MyList extends ListWrapperFactory[Int, MyList] See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - Spark sample is too slow

I'm trying to execute a simple random sample with Scala from an existing table, containing around 100e6 records. ... make it faster? Best, See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Type inference with existential type

I have a generic trait SomeTrait defined as so: trait SomeTrait[T] { def foo(t: T): String } And methods ... however I don't think it applies. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Spark: broadcasting jackson ObjectMapper

I have a spark application which reads lines from a files and tries to deserialize them using jackson. To get ... I missing here? Thanks, Aliza See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

862 views

1 answer

scala - How to eval code that uses InterfaceStability annotation (that fails with "illegal cyclic reference involving class InterfaceStability")?

I want to dynamically generate some kafka stream code when the program is running， but I get an exception when I compile ... version : "1.2.7" See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Call Solr asynchronous from Play Framework

I have created a Play 2.1 Scala application. I am uncertain what's the best way to call Solr from a Play ... /select?q=id%3A123&wt=json").get() See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - How to create a schema from CSV file and persist/save that schema to a file?

I have CSV file with 10 columns. Half String and half are Integers. What is the Scala code to: Create (infer) ... creating it on a fly. Thanks. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - Why Apache Spark take function not parallel?

Reading Apache Spark guide at http://spark.apache.org/docs/latest/programming-guide.html it states : Why is take ... to traverse entire RDD ? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - String interpolation, escaping quotation mark

I'm somewhat baffled by how difficult this turns out to be. I've already looked around stackoverflow, but no ... or before the $file-variable? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

884 views

1 answer

scala - Mockito's mock throw ClassNotFoundException in Spark application

I found that mock object in Mockito would throw ClassNotFoundException when used in Spark. Here is a minimal ... deserialized class on executor? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

scala - RDD to LabeledPoint conversion

If I have a RDD with about 500 columns and 200 million rows, and RDD.columns.indexOf("target", 0) shows ... the goal with DataFrame than RDD. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - Spark broadcast error: exceeds spark.akka.frameSize Consider using broadcast

I have a large data called "edges" org.apache.spark.rdd.RDD[org.apache.spark.graphx.Edge[(String, Int)]] = ... filters on it work just fine. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.4k views

1 answer

scala - Using Tuples in map, flatmap,... partial functions

If I do: val l = Seq(("un", ""), ("deux", "hehe"), ("trois", "lol")) l map { t => t._1 ... Why should I use "case" keyword to use named tuples? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags