Smooth maximum

In mathematics, a smooth maximum of an indexed family x₁, ..., x_n of numbers is a differentiable approximation to the maximum function

$\{x_{1},\ldots ,x_{n}\}\mapsto \max\{x_{1},\ldots ,x_{n}\},\,$

and the concept of smooth minimum is similarly defined.

For large positive values of the parameter $\alpha >0$, the following formulation is one smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.

${\mathcal {S}}_{\alpha }(\{x_{i}\}_{i=1}^{n})={\frac {\sum _{i=1}^{n}x_{i}e^{\alpha x_{i}}}{\sum _{i=1}^{n}e^{\alpha x_{i}}}}$

${\mathcal {S}}_{\alpha }$ has the following properties:

1. ${\mathcal {S}}_{\alpha }\to \max$ as $\alpha \to \infty$
2. ${\mathcal {S}}_{0}$ is the average of its inputs
3. ${\mathcal {S}}_{\alpha }\to \min$ as $\alpha \to -\infty$

The gradient of ${\mathcal {S}}_{\alpha }$ is closely related to softmax and is given by

$\nabla _{x_{i}}{\mathcal {S}}_{\alpha }(\{x_{i}\}_{i=1}^{n})={\frac {e^{\alpha x_{i}}}{\sum _{j=1}^{n}e^{\alpha x_{j}}}}[1+\alpha (x_{i}-{\mathcal {S}}_{\alpha }(\{x_{i}\}_{i=1}^{n}))].$

This makes the softmax function useful for optimization techniques that use gradient descent.

Another formulation is:

$g(x_{1},\ldots ,x_{n})=\log(\exp(x_{1})+\ldots +\exp(x_{n}))$