Tutorial on Practical Prediction Theory for Classification
John Langford; 6(Mar):273--306, 2005.
AbstractWe discuss basic prediction theory and its impact on classification success evaluation, implications for learning algorithm design, and uses in learning algorithm execution. This tutorial is meant to be a comprehensive compilation of results which are both theoretically rigorous and quantitatively useful.
There are two important implications of the results presented
here. The first is that common practices for reporting results in
classification should change to use the test set bound. The second is
that train set bounds can sometimes be used to directly motivate