Loss Functions Modulate the Optimal Bias-Variance Trade-off
- Adam Bear, Department of Psychology, Harvard University, Cambridge, Massachusetts, United States
- Fiery Cushman, Department of Psychology, Harvard University, Cambridge, Massachusetts, United States
AbstractPrediction problems vary in the extent to which accuracy is rewarded and inaccuracy is penalized—i.e., in their loss functions. Here, we focus on a particular feature of loss functions that controls how much large errors are penalized relative to how much precise correctness is rewarded: convexity. We show that prediction problems with convex loss functions (i.e., those in which large errors are particularly harmful) favor simpler models that tend to be biased, but exhibit low variability. Conversely, problems with concave loss functions (in which precise correctness is particularly rewarded) favor more complex models that are less biased, but exhibit higher variability. We discuss how this relationship between the bias-variance trade-off and the shape of the loss function may help explain features of human psychology, such as dual-process psychology and fast versus slow learning strategies, and inform statistical inference.