Making a scorer

To make a scorer, you need:

• A scoring function that compares y_test, the ground truth, with y_pred, the predictions
• To determine whether a high score is good or bad

Before passing the SVR regressor to the cross-validation, make a scorer by supplying two elements:

1. In practice, begin by importing the make_scorer function:

from sklearn.metrics import make_scorer

1. Use this sample scoring function:

#Only works for this iris example with targets 0 and 1
def for_scorer(y_test, orig_y_pred):
    y_pred = np.rint(orig_y_pred).astype(np.int)   #rounds prediction to the nearest integer
    return accuracy_score(y_test, y_pred)

The np.rint function rounds off the prediction to the nearest integer, hopefully one of the targets, 0 or 1. The astype method changes the type of the prediction to integer type, as the original target is in integer type and consistency is preferred with regard to types. After the rounding occurs, the scoring function uses the old accuracy_score function, which you are familiar with.

1. Now, determine whether a higher score is better. Higher accuracy is better, so for this situation, a higher score is better. In scikit code:

svr_to_class_scorer = make_scorer(for_scorer, greater_is_better=True) 

1. Finally, run the cross-validation with a new parameter, the scoring parameter:

svr_scores = cross_val_score(svr_clf, X_train, y_train, cv=4, scoring = svr_to_class_scorer)

1. Find the mean:

svr_scores.mean()

0.94663742690058483

The accuracy scores are similar for the SVR regressor-based classifier and the traditional SVC classifier.