Bias

$Bias(\widehat{f(x0)})=\mathbb{E}[\widehat{f(x0)}]-f(x0)$

• Bias is the expectation of the difference between the estimate and the true value.
• Bias values of complex models tend to be lesser than simpler models.

Variance

$Variance=\mathbb{E}[(\hat{f}(x0)-\mathbb{E}[\hat{f}(x0)]{)}^2]$

• Variance is the inconsistency or variability of the model's performance when applied to different data sets.
• Variance values of complex models tend to be higher than simpler models.

Trade-Off

• Reducible error is composed of variance and squared bias of the model.
• Flexibility of a model impacts both its bias and variance.
• The expectation of the squared error is given by

$E{(y_0-\hat{f}(x_0))}^2$.

• This can be decomposed as

$E(Y-\hat{Y}{)}^2=E[f(X)+ϵ-\hat{f}(X){]}^2$.

• Further simplifying this:

$$\begin{gathered} E[f(X)+ϵ-\hat{f}(X){]}^2=\underset{\text{Reducible}}{\underbrace{[f(X)-\hat{f}(X){]}^2}}+\underset{ \\ textIrreducible}{\underbrace{\text{Var}(ϵ)}} \end{gathered}$$

• In order to minimize the expected test error rate, we need to select a Machine Learning method that simultaneously achieves low variance and low bias.
  • Negative correlation between variance and bias of model.
  • ML model's flexibility has direct impact on its Variance & Bias.

View Next Topics

Overfitting and Regularization

Linear Regression Model