Non-linear Activation Functions for Neural Networks Simplified

Activation function is a what forms an output of a neuron. This is what adds non-linearlity to your prediction and makes a Neural Network based predictor so much better than linear models.

The question we usually ask ourselves is which activation function should I use? 

The answer is there is no one-works-all answer to this question. It depends.

Let me walk you through the most commonly used activation functions and their pros and cons to help you make a better decision.

We can define our own activation functions to best fit our need, the most commonly used ones are:

1. Sigmoid Activation 

2. Tan hyperbolic Activation

3. ReLU (Rectified Linear Unit)

4. Leaky ReLU

This is how each of them looks like:

Photo Source: DeepLearning.ai Specialization

1. Sigmoid Activation

The sigmoid activation ranges between 0 and 1. It looks like the common "S shaped" curve we see in different fields of studies. 

Pros:

Simple - logic and arithmetic-wise

Offers good non-linearity

Natural probablity output - between 0 and 1 for classification problems.

Cons:

Network stops learning when you shoot values towards the extremes of the sigmoid - This is called the problem of vanishing gradients.

2. Tan Hyperbolic

This is pretty much a sigmoid over an extended range (-1 to 1). 

Pros:

This increases the steady non-linear range in the middle of the sigmoid before the slope/gradient flattens out. This increased range helps the network learn faster.

Cons:

This activation does limit the problem of vanishing gradients on the tail ends of the sigmoid to a certain grade but we have better options to learn faster.

3. ReLU

Rectified - MAX(0,value)

Linear - for z >0 (positive values)

ReLU is a fancy name for a postive-only linear function. For negative predictions the unit has a slope of 0. However, for the positive activations the network can surely learn a lot faster with a linear slope.

Pros:
Learns faster.

Slope is 1 as long as z is positive.

Cons:

Yet to find any :)

4. Leaky ReLU

This provides a slight slope for negative values of the prediction from the neuron. It's an improvement over ReLU.

These non-linear activations help improve and lay the foundation for Neural Networks. 

Until next time!