The support vector machine (SVM) is a powerful and widely used classificationalgorithm. This paper uses the Karush-Kuhn-Tucker conditions to providerigorous mathematical proof for new insights into the behavior of SVM. Theseinsights provide perhaps unexpected relationships between SVM and two otherlinear classifiers: the mean difference and the maximal data piling direction.For example, we show that in many cases SVM can be viewed as a cropped versionof these classifiers. By carefully exploring these connections we show how SVMtuning behavior is affected by characteristics including: balanced vs.unbalanced classes, low vs. high dimension, separable vs. non-separable data.These results provide further insights into tuning SVM via cross-validation byexplaining observed pathological behavior and motivating improvedcross-validation methodology. Finally, we also provide new results on thegeometry of complete data piling directions in high dimensional space.