scikit learn - The dimension of dual_coef_ in sklearn. SVC

Question

In SVC() for multi-classification, the one-vs-one classifiers are trained. So there are supposed to be n_class * (n_class - 1)/2 classifiers in total. But why clf.dual_coef_ returns me only (n_class - 1) * n_SV? What does each row represent then?

Answer 1

The dual coefficients of a sklearn.svm.SVC in the multiclass setting are tricky to interpret. There is an explanation in the scikit-learn documentation. The sklearn.svm.SVC uses libsvm for the calculations and adopts the same data structure for the dual coefficients. Another explanation of the organization of these coefficients is in the FAQ. In the case of the coefficients you find in the fitted SVC classifier, interpretation goes as follows:

The support vectors identified by the SVC each belong to a certain class. In the dual coefficients, they are ordered according to the class they belong to. Given a fitted SVC estimator, e.g.

from sklearn.svm import SVC
svc = SVC()
svc.fit(X, y)

you will find

svc.classes_   # represents the unique classes
svc.n_support_ # represents the number of support vectors per class

The support vectors are organized according to these two variables. Each support vector being clearly identified with one class, it becomes evident that it can be implied in at most n_classes-1 one-vs-one problems, viz every comparison with all the other classes. But it is entirely possible that a given support vector will not be implied in all one-vs-one problems.

Taking a look at

support_indices = np.cumsum(svc.n_support_)
svc.dual_coef_[0:support_indices[0]]  # < ---
                                      # weights on support vectors of class 0
                                      # for problems 0v1, 0v2, ..., 0v(n-1)
                                      # so n-1 columns for each of the 
                                      # svc.n_support_[0] support vectors
svc.dual_coef_[support_indices[1]:support_indices[2]]  
                                      #  ^^^
                                      # weights on support vectors of class 1
                                      # for problems 0v1, 1v2, ..., 1v(n-1)
                                      # so n-1 columns for each of the 
                                      # svc.n_support_[1] support vectors
...
svc.dual_coef_[support_indices[n_classes - 2]:support_indices[n_classes - 1]]
                                      #  ^^^
                                      # weights on support vectors of class n-1
                                      # for problems 0vs(n-1), 1vs(n-1), ..., (n-2)v(n-1)
                                      # so n-1 columns for each of the 
                                      # svc.n_support_[-1] support vectors

gives you the weights of the support vectors for the classes 0, 1, ..., n-1 in their respective one-vs-one problems. Comparisons to all other classes except its own are made, resulting in n_classes - 1 columns. The order in which this happens follows the order of the unique classes exposed above. There are as many rows in each group as there are support vectors.

Possibly what you are looking for are the primal weights, which live in feature space, in order to inspect them as to their "importance" for classification. This is only possible with a linear kernel. Try this

from sklearn.svm import SVC
svc = SVC(kernel="linear")
svc.fit(X, y)  # X is your data, y your labels

Then take a look at

svc.coef_

This is an array of shape ((n_class * (n_class -1) / 2), n_features) and represents the aforementioned weights.

According to the doc the weights are ordered as:

class 0 vs class 1
class 0 vs class 2
...
class 0 vs class n-1
class 1 vs class 2
class 1 vs class 3
...
...
class n-2 vs class n-1