The process appears to be at least partially empirical, and over time, different functions have risen to the fore. ReLU turned out to work pretty well in early models, and it’s still popular now. It’s ...