Recent questions tagged apache-spark

0 votes

681 views

1 answer

apache-spark - 如何用消费者的热情衡量卡夫卡话题的滞后？(how to measure the lag of a kafka topic with consumer being a spark?)

We have a spark batch application that uses kafka as output. (我们有一个使用kafka作为输出的spark batch应用程序 )Is there a way to ... do it. (我们找不到办法 ) ask by user3308496 translate from so...

asked Mar 6, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

687 views

1 answer

apache-spark - 如何用消费者的热情衡量卡夫卡话题的滞后？(how to measure the lag of a kafka topic with consumer being a spark?)

We have a spark batch application that uses kafka as output. (我们有一个使用kafka作为输出的spark batch应用程序 )Is there a way to ... do it. (我们找不到办法 ) ask by user3308496 translate from so...

asked Mar 6, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

634 views

1 answer

apache-spark - 在循环中使用mapreduce进行火花(Spark with mapreduce in loop)

I've been testing spark with the word count example and it works fine. (我一直在用单词计数示例测试spark，它工作正常 ) However, I need to use the ... 何完成此操作的建议以及这种方法的利弊 ) ask by CFLS translate from so...

asked Mar 6, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

665 views

1 answer

apache-spark - 为什么读取Parquet文件比在SPARK中写入要占用更多的内存(Why reading Parquet file take much more memory than writing in SPARK)

I have a CSV file 40G, and use a SPARK job to convert it into parquet file with snappy compression and get a 1.8G parquet ... 对此有想法？)Thanks! (谢谢！) ask by Danny translate from so...

asked Mar 6, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

685 views

1 answer

apache-spark - 为什么读取Parquet文件比在SPARK中写入要占用更多的内存(Why reading Parquet file take much more memory than writing in SPARK)

I have a CSV file 40G, and use a SPARK job to convert it into parquet file with snappy compression and get a 1.8G parquet ... 对此有想法？)Thanks! (谢谢！) ask by Danny translate from so...

asked Mar 6, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

683 views

1 answer

apache-spark - 使用结构化流聚合流静态内部联接的输出(Aggregation on output of Stream-Static Inner Joins using Structured Streaming)

This problem pertains to Spark 2.4.4. (此问题与Spark 2.4.4有关 ) I am doing a Stream-static inner join and getting ... Please help!!! (请帮忙！！！) ask by Sarfaraz Hussain translate from so...

asked Mar 6, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

632 views

1 answer

apache-spark - 为什么当其中一个数据帧在scala spark中为unpersist（）时，会从存储中删除多个数据帧(why multiple dataframes are removed from storage when one of them are unpersist() in scala spark)

For the following scala-spark code, x and y are persisted. (对于以下scala-spark代码，x和y被保留 ) Why was y removed from ... () y.show() x.unpersist() ask by wainhuang translate from so...

asked Mar 6, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

692 views

1 answer

apache-spark - 为什么当其中一个数据帧在scala spark中为unpersist（）时，会从存储中删除多个数据帧(why multiple dataframes are removed from storage when one of them are unpersist() in scala spark)

For the following scala-spark code, x and y are persisted. (对于以下scala-spark代码，x和y被保留 ) Why was y removed from ... () y.show() x.unpersist() ask by wainhuang translate from so...