Recent questions tagged Scala

0 votes

1.2k views

1 answer

scala - How to read a file from HDFS in map() quickly with Spark

I need to read a different file in every map() ,the file is in HDFS val rdd=sc.parallelize(1 to 10000) ... runs on multiple machines. Thank you! See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - Load Dataset from Dynamically generated Case Class

What is Needed: number of tables in source database are changing rapidly and thus I don't want to edit case ... parameter bounds [T <: Product] See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Why does implement abstract method using val and call from superclass in val expression return NullPointerException

I have an abstract class with an unimplemented method numbers that returns a list of numbers, and this method ... allow this unsafe operation? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - Is there a native grouping function that works like the sortWith function?

There are a few libraries such as Spark and other Scala extensions that have the "groupWith" function available. This ... List(Item(16,red))) See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.5k views

1 answer

scala - udf No TypeTag available for type string

I don't understand a behavior of spark. I create an udf wich return an Integer like below import org.apache.spark. ... .. Do you have any idea? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - filter and report multiple predicates

This is a kind of cosmetic Scala question. A list with objects needs to be filtered on the objects' ... to implement the desired behaviour? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.5k views

1 answer

scala - How to let Spark parse a JSON-escaped String field as a JSON Object to infer the proper structure in DataFrames?

I have as input a set of files formatted as a single JSON object per line. The problem, however, is ... assumptions about what fields exist. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - How to filter duplicate records having multiple key in Spark Dataframe?

I have two dataframes. I want to delete some records in Data Frame-A based on some common column values in Data ... 9 6 Any solution for this. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.6k views

1 answer

scala - Need to convert a Seq[Option[A]] to Option[Seq[A]]

USE CASE I have a list of files that can might have a valid mime type or not. In my code, I represent ... liftOptionItem[A] does not fit in. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

985 views

1 answer

scala - Deriving nested shapeless lenses using only a type

I'm trying to come up with something similar to Classy Lenses to use with cats-mtl. For this, I want to be ... , but none of these have worked. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - Spark MLib Word2Vec Error: The vocabulary size should be > 0

I am trying to implement word vectorization using Spark's MLLib. I am following the example given here. I have ... not take sentences as input? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.8k views

1 answer

scala - How to transform a Seq[Try] to a Try[Seq]

With Futures there is an easy way to transform Seq[Future] to a Future[Seq]: Future.sequence(seqOfFutures) I ... for the 'compound' Failure? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

scala - How risky is it to call Await.result on db calls

When using phantom how dangerous is it to be following this pattern in db calls: Await.result(dbOperationFut, 30. ... akka, akka http) Thoughts? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - "./sbt/sbt assembly" errors "Not a valid command: assembly" for Apache Spark project

I'm having trouble with installing Apache Spark on Ubuntu 13.04. Im using spark-0.8.1-incubating, and ... guidance would be much appreciated. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - SBT before/after hooks for a task

(reposted from here) I'm trying to measure/log the running time of a task. I've looked into "wrapping" ... What should this "somehowInvoke" be? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.7k views

1 answer

scala - How to access Spark RDD Array of elements based on index

I have an RDD with Array of elements like below, each element can be treated as tuple, Now question is i ... new to spark so still learning. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - What is return type of `if` statement

From Programming Scala book I read that in following code configFilePath constant will be type of Unit: scala> val ... I use Scala 2.10. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Disambiguate constructor parameter with same name as class field of superclass

When playing around with Scala I had a code like this: class Superclass(var i : Int){} class Subclass(i : ... access the field i of Superclass? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

987 views

1 answer

scala - Read data from Cassandra for processing in Flink

I have to process data streams from Kafka using Flink as the streaming engine. To do the analysis on the ... am looking for Scala examples. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.4k views

1 answer

scala - Spark not working with pureconfig

I'm trying to use pureConfig and configFactory for my spark application configuration. here is my code: import ... what may work? Thanks See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

scala - Akka: "Trying to deserialize a serialized ActorRef without an ActorSystem in scope" error

I am integrating the use of Akka actors and Spark in the following way: when a task is distributed among ... really useful in the general case See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Apache Spark's RDD splitting according to the particular size

I am trying to read strings from a text file, but I want to limit each line according to a particular size. ... best performance way to do that? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - Proper way to make a Spark Fat Jar using SBT

I need a Fat Jar with Spark because I'm creating a custom node for Knime. Basically it's a self-contained jar ... 't find the source right now. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - publish jar files with sbt (3rd party)

Hello I have 5 jar files and tried to publish them to a local repository with sbt. But when I place them ... /guide-3rd-party-jars-local.html See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - How to create a Schema file in Spark

I am trying to read a Schema file (which is a text file) and apply it to my CSV file without a header. ... convert my input CSV file to ORC. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Practical use of futures? Ie, how to kill them?

Futures are very convenient, but in practice, you may need some guarantees on their execution. For example, ... futures given these risks? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - SBT Assembly - Deduplicate error & Exclude error

I am trying to build a JAR with dependencies using sbt-assembly. But I am running into this error again and ... [error] Type error in expression See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - Play Slick: How to inject DbConfigProvider in tests

I am using Play 2.5.10, Play-slick 2.0.2, and my activator-generated project comes with scalatest and ... access the default database in tests. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags