machine learning - Weka: Results of each fold in 10-fold CV

Question

For Weka Explorer (GUI), when we do a 10-fold CV for any given ARFF file, then what Weka Explorer provides (as far as I can see) is the average result for all the 10 folds.

Q. Is there any way to get the results of each fold? For instance, I need the error rates (incorrectly identified instances) for each fold.

Help appreciated.

Answer 1

I think this is possible using Weka's GUI. You need to use the Experimenter though instead of the Explorer. Here are the steps:

1. Open the Experimenter from the GUI Chooser
   • Create a new experiment (New button @ top-right)
   • [optional] Enter a filename and location in the Results Destination to save the results to
   • Set the Number of (cross-validation) folds to your liking (start experimenting with 2 folds for easy results)
   • Add your dataset (if your dataset needs preprocessing then you should do this in the Explorer first and then save the preprocessed dataset)
   • Set the Number of repetitions (I recommend 1 to start of with)
   • Add the algorithm(s) you want to test (again start easy, start with one algorithm)
2. Go to the Run tab and Start the experiment and wait till it finishes
3. Go to the Analyse tab and import the experiment results by clicking Experiment (top-right)
   • For Row select: Fold
   • For Column select: Percent_incorrect or Number_incorrect (or any other measure you want to see)
   • You now see the specified results for each fold