K-Nearest Neighbors and Support Vector Machines

In Chapter 3, Logistic Regression, we discussed using generalized linear models to determine the probability that a predicted observation belongs to a categorical response what we refer to as a classification problem. That was just the beginning of classification methods, with many techniques that we can use to try and improve our predictions.

In this chapter, we'll delve into two nonlinear techniques: K-Nearest Neighbors (KNN) and Support Vector Machines (SVMs). These techniques are more sophisticated than those we discussed earlier because the assumptions on linearity can be relaxed, which means a linear combination of the features to define the decision boundary isn't needed. Be forewarned, though, that this doesn't always equal superior predictive ability. Additionally, these models can be a bit problematic to interpret for business partners, and they can be computationally inefficient. When used wisely, they provide a powerful complement to the other tools and techniques discussed in this book. They can be used for continuous outcomes in addition to classification problems; however, for this chapter, we'll focus only on the latter.

After a high-level background on the techniques, we'll put both of them to the test, starting with KNN.

Following are the topics that we'll be covering in this chapter:

• K-nearest neighbors
• Support vector machines
• Manipulating data
• Modeling and evaluation