apache-spark - 在循环中使用mapreduce进行火花(Spark with mapreduce in loop)

Question

I've been testing spark with the word count example and it works fine.

(我一直在用单词计数示例测试spark，它工作正常。)

However, I need to use the output as new input for mapreduce and I dont know the better way to do that.

(但是，我需要将输出用作mapreduce的新输入，但我不知道这样做的更好方法。)

This operations is needed because each mapreduce operation will be seen as an iteration and the algorithm needs a lot of iterations (>5000).

(需要执行此操作，因为每个mapreduce操作将被视为一个迭代，并且该算法需要大量迭代（> 5000）。)

If you are curious: the algorithm desired is PSO.

(如果您很好奇：所需的算法是PSO。)

So, I need suggestions of how this can be done and pros and cons of this approach.

(因此，我需要有关如何完成此操作的建议以及这种方法的利弊。)

  ask by CFLS translate from so

Answer 1

等待大神答复