Stability and Bias-Variance of LOO vs. 3Way

Stability across datasets: For each evaluation method, over 19 datasets, see how many times solos appear in top 16. The numbers and plots for all methods are here.

Bias & Variance of LOO and 3-Way: Let L be the squared loss function, f be a predictor and y be the predicted value for a particular x. Also let y* be the optimal prediction (actual response for x), and ym be the main prediction. Under squared loss function, main function becomes mean of all predictions: ym=mean(all y's). Then the following definitions follow:

1. Bias(x) = L(y*, ym)
2. Var(x) = ED(L(ym, y)) where D is the occurrence of x in all training sets.
   

Above definitions are valid for single instances, however they can be average over all instances:

1. Biasavg = Ex(B(x))
2. Varavg = Ex(Var(x))

When we follow these definitions we get bias(x-axis) vs. variance(y-axis) figures here.