Laplace's method

For the optimization method called "steepest descent" see gradient descent.

In mathematics, the steepest descent method or saddle-point approximation is a method used to approximate integrals of the form

\int _{a}^{b}\!e^{Mf(x)}\,dx\,

where f(x) is some twice-differentiable function, M is a large number, and the integral endpoints a and b could possibly be infinite. The technique is also often referred to as Laplace's method, which in fact concerns the special case of real-valued functions f admitting a maximum at a real point.

The idea of Laplace's method

Assume that the function f(x) has a unique global maximum x₀. Then, the value f(x₀) will be larger than other values f(x). If we multiply this function by a large number M, the gap between Mf(x₀) and Mf(x) will only increase, and then it will grow "exponentially" for the function

e^{Mf(x)}.\,

As such, significant contributions to the integral of this function will come only from points x in a neighborhood of x₀, which can then be estimated.

General theory of Laplace's method

To state and prove the method, we need several assumptions. We will assume that x₀ is not an endpoint of the interval of integration, that the values f(x) cannot be very close to f(x₀) unless x is close to x₀, and that $f''(x_{0})<0$ .

We can expand f(x) around x₀ by Taylor's theorem,

f(x)=f(x_{0})+f'(x_{0})(x-x_{0})+{\frac {1}{2}}f''(x_{0})(x-x_{0})^{2}+O\left((x-x_{0})^{3}\right).

Since f has a global maximum at x₀, and since x₀ is not an endpoint, it is a stationary point, the derivative of f vanishes at x₀. Therefore, the function f(x) may be approximated to quadratic order

f(x)\approx f(x_{0})-{\frac {1}{2}}|f''(x_{0})|(x-x_{0})^{2}

for x close to x₀ (recall that the second derivative is negative at the global maximum f(x₀)). The assumptions made ensure the accuracy of the approximation

\int _{a}^{b}\!e^{Mf(x)}\,dx\approx e^{Mf(x_{0})}\int e^{-M|f''(x_{0})|(x-x_{0})^{2}/2}dx

where the integral is taken in a neighborhood of x₀. This latter integral is a Gaussian integral if the limits of integration go from −∞ to +∞ (which can be assumed so because the exponential decays very fast away from x₀), and thus it can be calculated. We find

\int _{a}^{b}\!e^{Mf(x)}\,dx\approx {\sqrt {\frac {2\pi }{M|f''(x_{0})|}}}e^{Mf(x_{0})}{\mbox{ as }}M\to \infty .\,

Steepest descent

In extensions of Laplace's method, complex analysis, and in particular Cauchy's integral formula, is used to find a contour of steepest descent for an (asymptotically with large M) equivalent integral, expressed as a line integral. In particular, if no point x₀ where the derivative of f vanishes exists on the real line, it may be necessary to deform the integration contour to an optimal one, where the above analysis will be possible. Again the main idea is to reduce, at least asymptotically, the calculation of the given integral to that of a simpler integral that can be explicitly evaluated. See the book of Erdelyi (1956) for a simple discussion.

Further generalizations

An extension of the steepest descent method is the so-called nonlinear stationary phase/steepest descent method. Here, instead of integrals, one needs to evaluate asymptotically solutions of Riemann-Hilbert factorization problems.

Given a contour C in the complex sphere, a function f defined on that contour and a special point, say infinity, one seeks a function M holomorphic away from the contour C, with prescribed jump across C, and with a given normalization at infinity. If f and hence M are matrices rather than scalars this is a problem that in general does not admit an explicit solution.

An asymptotic evaluation is then possible along the lines of the linear stationary phase/steepest descent method. The idea is to reduce asymptotically the solution of the given Riemann-Hilbert problem to that of a simpler, explicitly solvable, Riemann-Hilbert problem. Cauchy's theorem is used to justify deformations of the jump contour.

The nonlinear stationary phase was introduced by Deift and Zhou in 1993, based on earlier work of Its. A (properly speaking) nonlinear steepest descent method was introduced by Kamvissis, K. McLaughlin and P. Miller in 2003, based on previous work of Lax, Levermore, Deift, Venakides and Zhou.

The nonlinear stationary phase/steepest descent method has applications to the theory of soliton equations, random matrices and combinatorics.

Complex integrals

For complex integrals in the form:

{\frac {1}{2\pi i}}\int _{c-i\infty }^{c+i\infty }g(s)e^{st}\,ds

with t >> 1, we make the substitution t = iu and the change of variable s = c + ix to get the Laplace bilateral transform:

{\frac {1}{2\pi }}\int _{-\infty }^{\infty }g(c+ix)e^{-ux}e^{icu}\,dx.

We then split g(c+ix) in its real and complex part, after which we recover u = t / i. This is useful for inverse Laplace transforms, the Perron formula and complex integration.

Example: Stirling's approximation

Laplace's method can be used to derive Stirling's approximation

N!\approx {\sqrt {2\pi N}}N^{N}e^{-N}\,

for a large integer N.

From the definition of the Gamma function, we have

N!=\Gamma (N+1)=\int _{0}^{\infty }e^{-x}x^{N}dx.\,

Now we change variables, letting

x=Nz\,

so that

dx=Ndz.\,

Plug these values back in to obtain

$N!\,$	$=\int _{0}^{\infty }e^{-Nz}\left(Nz\right)^{N}Ndz\,$
	$=N^{N+1}\int _{0}^{\infty }e^{-Nz}z^{N}dz\,$
	$=N^{N+1}\int _{0}^{\infty }e^{-Nz}e^{N\ln z}dz\,$
	$=N^{N+1}\int _{0}^{\infty }e^{N(\ln z-z)}dz.\,$

This integral has the form necessary for Laplace's method with

f\left(z\right)=\ln {z}-z

which is twice-differentiable:

f'(z)={\frac {1}{z}}-1\,,

f''(z)=-{\frac {1}{z^{2}}}.\,

The maximum of f(z) lies at z₀=1, and the second derivative of f(z) has at this point the value -1. Therefore, we obtain

N!\approx N^{N+1}{\sqrt {\frac {2\pi }{N}}}e^{-N}={\sqrt {2\pi N}}N^{N}e^{-N}.\,

References

P. Deift, X. Zhou, A steepest descent method for oscillatory Riemann-Hilbert problems. Asymptotics for the MKdV equation, Ann. of Math. (2), v.137 (1993), no. 2, 295–368.
A. Erdelyi, Asymptotic Expansions, Dover, 1956.
S. Kamvissis, K. T.-R. McLaughlin, P. Miller, Semiclassical Soliton Ensembles for the Focusing Nonlinear Schrödinger Equation, Annals of Mathematics Studies, v.154, Princeton University Press, 2003.