Relationship between bias, variance, and test set MSE
In all three cases, the variance increases and the bias decreases as the method’s flexibility increases. However, the flexibility level corresponding to the optimal test MSE differs considerably among the three data sets, because the squared bias and variance change at different rates in each of the data sets.
In the left-hand panel of Figure, the bias initially decreases rapidly, resulting in an initial sharp decrease in the expected test MSE.
On the other hand, in the center panel of Figure, the true f is close to linear, so there is only a small decrease in bias as flexibility increases, and the test MSE only declines slightly before increasing rapidly as the variance increases.
Finally, in the right-hand panel of Figure, as flexibility increases, there is a dramatic decline in bias because the true f is very non-linear. There is also very little increase in variance as flexibility increases.
Consequently, the test MSE declines substantially before experiencing a small increase as model flexibility increases.
The relationship between bias, variance, and test set MSE is referred to as the bias-variance trade-off
Good test set performance of a statistical learning method re- quires low variance as well as low squared bias. This is referred to as a trade-off because it is easy to obtain a method with extremely low bias but high variance (for instance, by drawing a curve that passes through every single training observation) or a method with very low variance but high bias (by fitting a horizontal line to the data).
The challenge lies in finding a method for which both the variance and the squared bias are low.
