There are several common activation functions used in neural networks. These activation functions introduce non-linear properties into the network, allowing it to learn complex patterns and make nonlinear decisions. Some commonly used activation functions include:

  1. Sigmoid function: The sigmoid function is a smooth, S-shaped function that maps the input values to probabilities between 0 and 1. It is given by the formula: f(x) = 1 / (1 + e^(-x)). It is often used in the output layer of a binary classification problem.

  2. Rectified Linear Unit (ReLU): ReLU is a simple activation function that returns the input if it is positive, and zero otherwise. It is given by the formula: f(x) = max(0, x). ReLU helps in training deep neural networks efficiently as it avoids the vanishing gradient problem.

  3. Leaky ReLU: Leaky ReLU is similar to ReLU, but it allows a small negative slope for negative input values instead of setting them to zero. It is given by the formula: f(x) = max(ax, x), where a is a small constant, typically 0.01. Leaky ReLU also helps in addressing the vanishing gradient problem.

  4. Hyperbolic tangent (tanh): The tanh activation function is similar to the sigmoid function but maps values between -1 and 1. It is given by the formula: f(x) = (e^x - e^(-x)) / (e^x + e^(-x)). Tanh is often used in the hidden layers of a neural network.

  5. Softmax: Softmax is used in the output layer of a multiclass classification problem. It converts the input values into a probability distribution across multiple classes. It is given by the formula: f(x_i) = e^(x_i) / (∑(e^(x_j))), where x_i represents the input to the softmax function and (∑(e^(x_j))) represents the sum of the exponential of all the inputs.

  These are just a few examples of the commonly used activation functions in neural networks. The choice of activation function depends on the problem at hand and the network architecture being used.