Classification Accuracy evaluation
•Holdout method
–Given data is randomly partitioned into two independent sets
•Training set (e.g., 2/3) for model construction
•Test set (e.g., 1/3) for accuracy estimation
–Random sampling: a variation of holdout
•Repeat holdout k times, accuracy = avg. of the accuracies obtained
•Cross-validation (k-fold, where k = 10 is most popular)
–Randomly partition the data into k mutually exclusive subsets, each approximately equal size
–At i-th iteration, use Di as test set and others as training set
–Leave-one-out: k folds where k = # of tuples, for small sized data
–Stratified cross-validation: folds are stratified so that class dist. in each fold is approx. the same as that in the initial data
•Bootstrap
–Works well with small data sets
–Samples the given training tuples uniformly with replacement
•i.e., each time a tuple is selected, it is equally likely to be selected again and re-added to the training set
•Several boostrap methods, and a common one is .632 boostrap
–Suppose we are given a data set of d tuples. The data set is sampled d times, with replacement, resulting in a training set of d samples. The data tuples that did not make it into the training set end up forming the test set. About 63.2% of the original data will end up in the bootstrap, and the remaining 36.8% will form the test set (since (1 – 1/d)d ≈ e-1 = 0.368)
–Repeat the sampling procedue k times, overall accuracy of the model:
Predictor Error Measures
•Measure predictor accuracy: measure how far off the predicted value is from the actual known value
•Loss function: measures the error betw. yi and the predicted value yi’
–Absolute error: | yi – yi’|
–Squared error: (yi – yi’)2
•Test error (generalization error): the average loss over the test set
–Mean absolute error: Mean squared error:
–Relative absolute error: Relative squared error:
The mean squared-error exaggerates the presence of outliers
Popularly use (square) root mean-square error, similarly, root relative squared error