Recent questions tagged Scala

0 votes

664 views

1 answer

scala - Why does Spark fail with java.lang.OutOfMemoryError: GC overhead limit exceeded?

I'm trying to implement a Hadoop Map/Reduce job that worked fine before in Spark. The Spark app definition is ... very long. Thanks in advance. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

511 views

1 answer

scala - Sets, Functors and Eq confusion

A discussion came up at work recently about Sets, which in Scala support the zip method and how this can ... the Set instance in particular? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

523 views

1 answer

scala - Spark final task takes 100x times longer than first 199, how to improve

I am seeing some performance issues while running queries using dataframes. I have seen in my research, that long ... I have empty executors? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

571 views

1 answer

scala - Play 2.x : Reactive file upload with Iteratees

I will start with the question: How to use Scala API's Iteratee to upload a file to the cloud storage (Azure Blob ... for me to get the idea. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

521 views

1 answer

scala - How to create correct data frame for classification in Spark ML

I am trying to run random forest classification by using Spark ML api but I am having issues with creating right data ... ) } } Thanks for help! See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

533 views

1 answer

scala - Testing an assertion that something must not compile

The problem When I'm working with libraries that support type-level programming, I often find myself writing ... 'll settle for anything. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

806 views

1 answer

scala - reduceByKey: How does it work internally?

I am new to Spark and Scala. I was confused about the way reduceByKey function works in Spark. Suppose we have the ... + b? how are they filled? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

631 views

1 answer

scala - Spark : how to run spark file from spark shell

I am using CDH 5.2. I am able to use spark-shell to run the commands. How can I run the file(file.spark ... 5.2 without sbt? Thanks in advance See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

695 views

1 answer

scala - Implicit conversion vs. type class

In Scala, we can use at least two methods to retrofit existing or new types. Suppose we want to express that ... be nice, even without example. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

707 views

1 answer

scala - ScalaTest in sbt: is there a way to run a single test without tags?

I know that a single test can be ran by running, in sbt, testOnly *class -- -n Tag Is there a way of ... a single test without it having a tag? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

437 views

1 answer

scala - using prepared statement multiple times, giving a warning of Cassandra Querying Reducing Performance

I am getting data from somewhere and inserting it into cassandra daily basis then I need to retrieve the data from cassandra ... ............ See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

579 views

1 answer

scala - How to access sub-entities in JSON file?

I have a json file look like this: { "employeeDetails":{ "name": "xxxx", "num":"415" }, "work":[ ... [_corrupt_record#0] json I am new to Spark. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

650 views

1 answer

scala - Futures / Success race

I'm learning futures, and I'm trying to create a method that, take two futures as parameter (f and g) and ... I know which one completed first ? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

652 views

1 answer

scala - dynamically parse json in flink map

I'm using flink to dynamically analyze json type data,to keyby and sum with the given column,in my mapFunction,I ... to case class or tuple? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

794 views

1 answer

scala - Get a TypeTag from a Type?

I can get a Type from a TypeTag[A] by using the tpe method. But can I also recover the type-tag from a ... have the type and dropped the tag. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

826 views

1 answer

scala - NoClassDefFoundError: org/apache/hadoop/fs/StreamCapabilities while reading s3 Data with spark

I would like to run a simple spark job on my local dev machine (through Intellij) reading data from Amazon ... Thanks in advance for your help See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

558 views

1 answer

scala - Broadcast not happening while joining dataframes in Spark 1.6

Below is the sample code that I am running. when this spark job runs, Dataframe joins are happening using ... datasize of countriesDf is 5mb. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

651 views

1 answer

scala - Scalaz Bind[Seq] typeclass

I'm currently porting some code from traditional Scala to Scalaz style. It's fairly common through most of my ... my own Bind[Seq] typeclass? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

703 views

1 answer

scala - Futures do not run before program termination

I was trying to reproduce the example on new Scala 2.10 futures feature. The code I've used is: import scala. ... compiler is 2.10.0-20120507. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

693 views

1 answer

scala - Spark column string replace when present in other column (row)

I would like to remove strings from col1 that are present in col2: val df = spark.createDataFrame(Seq( ("Hi I ... so that res looks as below: See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

617 views

1 answer

scala - How to create a Dataset of Maps?

I'm using Spark 2.2 and am running into troubles when attempting to call spark.createDataset on a Seq of ... not covered by current encoders? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

734 views

1 answer

scala - Concrete classes with abstract type members

Given the following traits and class. Why does this compile? Can this be actually used for something? trait ... motivation for it: #1753. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

518 views

1 answer

scala - Separating application logs in Logback from Spark Logs in log4j

I have a Scala Maven project using that uses Spark, and I am trying implement logging using Logback. I am ... still be shown at STDOUT. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

632 views

1 answer

scala - What is a Singleton Type exactly?

What is a singleton type? what are the applications, the implications ? Examples are more than welcome and layman terms are even more welcome ! See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

580 views

1 answer

scala - "Parameter type in structural refinement may not refer to an abstract type defined outside that refinement"

When I compile: object Test extends App { implicit def pimp[V](xs: Seq[V]) = new { def dummy(x: V) ... refinement" doesn't really answer this.) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - How to slice and sum elements of array column?

I would like to sum (or perform other aggregate functions too) on the array column using SparkSQL. I have a table ... apart from the UDF one ? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

756 views

1 answer

scala - Is there a simple example of how to generate verilog from Chisel3 module?

I'm looking for a simple howto to convert a simple Chisel3 module in Verilog. I take Gcd source code given ... for converting it in Verilog. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

671 views

1 answer

scala - Multiple parameter closure argument type not inferred

I have a piece of code that I can't get to behave in the way I'd like. I have a class defined in the following way ( ... a, b : Int) => a + b) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags