How it works...

In comparing the SVC and logistic regression classifier, you might wonder (and be a little suspicious) about a lot of scores being very different. The final test on SVC scored lower than logistic regression. To help with this situation, we can do cross-validation in scikit-learn.

Cross-validation involves splitting the training set into parts, as we did before. To match the preceding example, we split the training set into four parts, or folds. We are going to design a cross-validation iteration by taking turns with one of the four folds for testing and the other three for training. It is the same split as done before four times over with the same set, thereby rotating, in a sense, the test set:

With scikit-learn, this is relatively easy to accomplish:

1. We start with an import:

from sklearn.model_selection import cross_val_score

1. Then we produce an accuracy score on four folds:

svc_scores = cross_val_score(svc_clf, X_train, y_train, cv=4)
svc_scores

array([ 0.82758621,  0.85714286,  0.92857143,  0.77777778])

1. We can find the mean for average performance and standard deviation for a measure of spread of all scores relative to the mean:

print "Average SVC scores: ", svc_scores.mean()
print "Standard Deviation of SVC scores: ", svc_scores.std()

Average SVC scores:  0.847769567597
Standard Deviation of SVC scores:  0.0545962864696

1. Similarly, with the logistic regression instance, we compute four scores:

lr_scores = cross_val_score(lr_clf, X_train, y_train, cv=4)
print "Average SVC scores: ", lr_scores.mean()
print "Standard Deviation of SVC scores: ", lr_scores.std()

Average SVC scores:  0.748893906221
Standard Deviation of SVC scores:  0.0485633168699

Now we have many scores, which confirms our selection of SVC over logistic regression. Thanks to cross-validation, we used the training multiple times and had four small test sets within it to score our model.

Note that our model is a bigger model that consists of:

• Training an SVM through cross-validation
• Training a logistic regression through cross-validation
• Choosing between SVM and logistic regression