In previous chapters we discussed various ways to implement and control LR. In particular, Chapter 5 chooses a set of LR controls and enhancements by analyzing performance on a collection of real-world datasets. The result was three versions of LR: IRLS with cgeps, IRLS with cgdeveps, and MLE with CG. These will be called LR-CGEPS, LR-CGDEVEPS, and CG-MLE in this chapter. In this chapter we will examine how these LR implementations compare to each other and to other classifiers.
Section 6.1 discusses several classifiers we will use in our comparisons. These are support vector machines (SVM), k-nearest-neighbor (KNN) and Bayes' Classifier (BC). We are not testing decision trees on these datasets because we expect very poor performance on the linearly-separable synthetic datasets, and we have observed poor performance on our real-world datasets. It is possible that other varieties of tree-based classifiers, such as bagged or boosted decision trees, would perform better. However, we do not have an implementation of decision trees other than C4.5 [38].
Section 6.2 contains experimental results on the synthetic datasets described in Section 5.1.3.3. These datasets help us measure classifier score and speed as a function of the dataset's number of rows, number of attributes, sparsity and attribute coupling. In Section 6.3 we have a showdown between all the classifiers on the six real-world datasets described in Sections 5.1.3.1 and 5.1.3.2.
To compare the classifiers we will use AUC scores and times. The AUC score is defined in Section 5.1.4. In particular we will include a 95% confidence interval for all AUC scores. Our method of timing is described in Section 5.1.5. Efforts have been made to find reasonable parameter choices for these classifiers. Where possible we have used the implementer's default settings unless performance was unreasonable and a better value was found. Non-default parameter choices will be described in Section 6.1. Please note that the timings in this chapter are from computers with the new BIOS, as explained in Section 5.1.5. For this reason, these timings are generally faster than the timings in Chapter 5. The one exception is KNN, which appears to run between five and fifteen percent slower on the new BIOS. The larger the dataset, the smaller the KNN slowdown. However, this slowdown is immaterial in the comparisons to other classifiers.