In mathematics , a smooth maximum of an indexed family x 1 , ..., x n of numbers is a differentiable approximation to the maximum function
x
1
,
…
,
x
n
↦
max
(
x
1
,
…
,
x
n
)
,
{\displaystyle x_{1},\ldots ,x_{n}\mapsto \max(x_{1},\ldots ,x_{n}),}
and the concept of smooth minimum is similarly defined.
Examples
For large positive values of the parameter
α
>
0
{\displaystyle \alpha >0}
, the following formulation is one smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.
S
α
(
x
1
,
…
,
x
n
)
=
∑
i
=
1
n
x
i
e
α
x
i
∑
i
=
1
n
e
α
x
i
{\displaystyle {\mathcal {S}}_{\alpha }(x_{1},\ldots ,x_{n})={\frac {\sum _{i=1}^{n}x_{i}e^{\alpha x_{i}}}{\sum _{i=1}^{n}e^{\alpha x_{i}}}}}
S
α
{\displaystyle {\mathcal {S}}_{\alpha }}
has the following properties:
S
α
→
max
{\displaystyle {\mathcal {S}}_{\alpha }\to \max }
as
α
→
∞
{\displaystyle \alpha \to \infty }
S
0
{\displaystyle {\mathcal {S}}_{0}}
is the arithmetic mean of its inputs
S
α
→
min
{\displaystyle {\mathcal {S}}_{\alpha }\to \min }
as
α
→
−
∞
{\displaystyle \alpha \to -\infty }
The gradient of
S
α
{\displaystyle {\mathcal {S}}_{\alpha }}
is closely related to softmax and is given by
∇
x
i
S
α
(
x
1
,
…
,
x
n
)
=
e
α
x
i
∑
j
=
1
n
e
α
x
j
[
1
+
α
(
x
i
−
S
α
(
x
1
,
…
,
x
n
)
)
]
.
{\displaystyle \nabla _{x_{i}}{\mathcal {S}}_{\alpha }(x_{1},\ldots ,x_{n})={\frac {e^{\alpha x_{i}}}{\sum _{j=1}^{n}e^{\alpha x_{j}}}}[1+\alpha (x_{i}-{\mathcal {S}}_{\alpha }(x_{1},\ldots ,x_{n}))].}
This makes the softmax function useful for optimization techniques that use gradient descent .
LogSumExp
Another smooth maximum is LogSumExp :
L
S
E
(
x
1
,
…
,
x
n
)
=
log
(
exp
(
x
1
)
+
…
+
exp
(
x
n
)
)
{\displaystyle \mathrm {LSE} (x_{1},\ldots ,x_{n})=\log(\exp(x_{1})+\ldots +\exp(x_{n}))}
This can also be normalized if the
x
i
{\displaystyle x_{i}}
are all non-negative, yielding a function with domain
[
0
,
∞
)
n
{\displaystyle [0,\infty )^{n}}
and range
[
0
,
∞
)
{\displaystyle [0,\infty )}
:
g
(
x
1
,
…
,
x
n
)
=
log
(
exp
(
x
1
)
+
…
+
exp
(
x
n
)
−
(
n
−
1
)
)
{\displaystyle g(x_{1},\ldots ,x_{n})=\log(\exp(x_{1})+\ldots +\exp(x_{n})-(n-1))}
The
(
n
−
1
)
{\displaystyle (n-1)}
term corrects for the fact that
exp
(
0
)
=
1
{\displaystyle \exp(0)=1}
by canceling out all but one zero exponential, and
log
1
=
0
{\displaystyle \log 1=0}
if all
x
i
{\displaystyle x_{i}}
are zero.
Use in numerical methods
This section is empty. You can help by
adding to it .
(February 2015 )
Other choices of smoothing function
This section is empty. You can help by
adding to it .
(March 2015 )
See also
References
M. Lange, D. Zühlke, O. Holz, and T. Villmann, "Applications of lp-norms and their smooth approximations for gradient based learning vector quantization," in Proc. ESANN , Apr. 2014, pp. 271-276.
(https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2014-153.pdf )