python - Using multiple mapper inputs in one streaming job on hadoop?

Question

In java I would use:

MultipleInputs.addInputPath(conf, path, inputFormatClass, mapperClass)

to add multiple inputs with a different mapper for each.

Now I am using python to write a streaming job in hadoop, can a similiar job be done?

Answer 1

You can use multiple -input options to specify multiple input paths:

hadoop jar hadoop-streaming.jar -input foo.txt -input bar.txt ...