Sigmoid derivative

Posted on

The sigmoid function has found extensive use as a non-linear activation function for neurons in artificial neural networks. I’m currently following a course on machine learning and I found interesting to redo the maths for it’s derivative.

The sigmoid function is given by the equation \eqref{eq:sigmoid-derivative-first-result}. The advantage of this

\begin{equation} \sigma (a) = \frac{1}{1 + e^{-a}} \label{eq:sigmoid-derivative-first-result} \end{equation}

Sigmoid function

We want to get the derivative of \(\sigma (a)\). In order to do that, we can use one of the forms known for the derivatives saying that \((\frac{1}{f})’ = -\frac{f’}{f^2}\). This gives us:

\begin{align} \frac{\partial}{\partial a} \frac{1}{1 + e^{-a}} & = - \frac{\frac{\partial}{\partial a} 1 + \frac{\partial}{\partial a} e^{-t}}{(1 + e^{-t})^2} \end{align}

\begin{align} \frac{\partial}{\partial a} \frac{1}{1 + e^{-a}} & = - \frac{0 + - e^{-t}}{(1 + e^{-t})^2} \end{align}

\begin{align} \frac{\partial}{\partial a} \frac{1}{1 + e^{-a}} & = \frac{e^{-t}}{(1 + e^{-t})^2} \end{align}

Detailed steps of \eqref{eq:sigmoid-derivative-first-result}:

$$ \frac{e^{-t}}{1 + e^{-t}} = \frac{1 + e^{-t} - 1}{1 + e^{-t}} $$

$$ \frac{e^{-t}}{1 + e^{-t}} = \frac{1 + e^{-t}}{1 + e^{-t}} - \frac{1}{1 + e^{-t}} $$

$$ \frac{e^{-t}}{1 + e^{-t}} = 1 - \frac{1}{1 + e^{-t}} $$