Min-max theorem

"Variational theorem" redirects here. The term is also sometimes applied to the variational principle.

In linear algebra and functional analysis, the min-max theorem, or variational theorem, or Courant-Fischer-Weyl min-max principle, is a result that gives a variational characterization of eigenvalues of compact Hermitian operators on Hilbert spaces. It can be viewed as the starting point of many results of similar nature.

This article first discusses the finite dimensional case and its applications before considering compact operators on infinite dimensional Hilbert spaces. We will see that for compact operators, the proof of the main theorem uses essentially the same idea from the finite dimensional argument.

The min-max theorem can be extended to self adjoint operators that are bounded below.

Matrices

Let A be a n × n Hermitian matrix. As with many other variational results on eigenvalues, one considers the Rayleigh-Ritz quotient R: Cⁿ → R defined by

$R(x)={\frac {(Ax,x)}{\|x\|^{2}}}$

where (·, ·) denotes the Euclidean inner product on Cⁿ. Equivalently, the Rayleigh-Ritz quotient can be replaced by

$f(x)=(Ax,x),\;\|x\|=1.$

For Hermitian matrices, the range of the continuous function R(x), or f(x), is a compact subset [a, b] of the real line. The maximum b and the minimum a are the largest and smallest eigenvalue of A, respectively. The min-max theorem is a refinement of this fact.

Lemma Let S_k be a k dimensional subspace.

1. If the eigenvalues of A are listed in increasing order λ₁ ≤ ... ≤ λ_k ≤ ... ≤ λ_n, then there exists x ∈ S_k, ||x|| = 1 such that (Ax, x) ≥ λ_k.
2. Similarly, if the eigenvalues of A are listed in decreasing order λ₁ ≥ ... ≥ λ_k ≥ ... ≥ λ_n, then there exists y ∈ S_k, ||y|| = 1 such that (Ay, y) ≤ λ_k.

Proof:

1. Let u_i be the eigenvector corresponding to λ_i. Consider the subspace S' = span{u_k...u_n}. Simply by counting dimensions, we see that the subspace S' ∩ S_k is nonempty. So there exists x ∈ S' ∩ S_k with ||x|| = 1. But for all x ∈ S' , (Ax, x) ≥ λ_k. So the claim holds.
2. Exactly the same as 1. above.

From the above lemma follows readily the min-max theorem. If the eigenvalues of A are listed in increasing order λ₁ ≤ ... ≤ λ_k ≤ ... ≤ λ_n, by first part of lemma, we have that for all k dimensional subspace S_k,

$\max _{x\in S_{k},\|x\|=1}(Ax,x)\geq \lambda _{k}.$

This implies

$\inf _{S_{k}}\max _{x\in S_{k},\|x\|=1}(Ax,x)\geq \lambda _{k}.$

But choose S_k to be span{u₁...u_k} and we see that equality is achieved. Therefore

$\lambda _{k}^{\uparrow }=\min _{S_{k}}\max _{x\in S_{k},\|x\|=1}(Ax,x).$

, where the ↑ indicates it is the k-th eigenvalue in the increasing order. Similarly, the second part of lemma gives

$\lambda _{k}^{\downarrow }=\max _{S_{k}}\min _{x\in S_{k},\|x\|=1}(Ax,x).$

The min-max theorem consists of the above two equalities.

Applications

Min-max principle for singular values

The singular values {σ_k} of a square matrix M are the square roots of eigenvalues of M*M (equivalently MM*). An immediate consequence of the first equality from min-max theorem is

$\sigma _{k}^{\uparrow }=\min _{S_{k}}\max _{x\in S_{k},\|x\|=1}(M^{*}Mx,x)^{\frac {1}{2}}=\min _{S_{k}}\max _{x\in S_{k},\|x\|=1}\|Mx\|.$

Similarly,

$\sigma _{k}^{\downarrow }=\max _{S_{k}}\min _{x\in S_{k},\|x\|=1}\|Mx\|.$

Cauchy interlacing theorem

Let A be a n × n matrix. A m × m matrix B, where m ≤ n, is called a compression of A if there exists an orthogonal projection P onto a subspace of dimension m such that PAP = B. The Cauchy interlacing theorem states:

Theorem If the eigenvalues of A are α₁ ≤ ... ≤ α_n, and those of B are β₁ ≤ ... β_j ... ≤ β_m, then for all j,

$\alpha _{j}\leq \beta _{j}\leq \alpha _{n-m+j}.$

This can be proven using the min-max principle. Let β_i have corresponding eigenvector b_i and S_j be the j dimensional subspace S_j = span{b₁...b_j}, then

$\beta _{j}=\max _{x\in S_{j},\|x\|=1}(Bx,x)=\max _{x\in S_{j},\|x\|=1}(PAPx,x)=\max _{x\in S_{j},\|x\|=1}(Ax,x).$

According to first part of min-max,

$\alpha _{j}\leq \beta _{j}.$

On the other hand, if we define S_m-j+1 = span{b_j...b_m}, then

$\beta _{j}=\min _{x\in S_{m-j+1},\|x\|=1}(Bx,x)=\min _{x\in S_{m-j+1},\|x\|=1}(PAPx,x)=\min _{x\in S_{m-j+1},\|x\|=1}(Ax,x)\leq \alpha _{n-m+j},$

where the last equality is given by the second part of min-max.

Notice that, when n - m = 1, we have

$\alpha _{j}\leq \beta _{j}\leq \alpha _{j+1}.$

Hence the name interlacing theorem.

Compact operators

Let A be a compact, Hermitian operator on a Hilbert space H. Recall that the spectrum of such an operator form a sequence of real numbers whose only possible cluster point is zero. Every nonzero number in the spectrum is an eigenvalue. It no longer makes sense here to list the positive eigenvalues in increasing order. Let the positive eigenvalues of A be

$\cdots \leq \lambda _{k}\leq \cdots \leq \lambda _{1}$

,where multiplicity is taken into account as in the matrix case. When H is infinite dimensional, the above sequence of eigenvalues is necessarily infinite. We now apply the same reasoning as in the matrix case. Let S_k ⊂ H be a k dimensional subspace, and S' be the closure of the linear span S' = span{u_k, u_{k + 1}...}. The subspace S' has codimension k - 1. By the same dimension count argument as in the matrix case, S' ∩ S_k is non empty. So there exists x ∈ S' ∩ S_k with ||x|| = 1. Since it is an element of S' , such an x necessarily satisfy

$(Ax,x)\leq \lambda _{k}.$

Therefore, for all S_k

$\inf _{x\in S_{k},\|x\|=1}(Ax,x)\leq \lambda _{k}$

But A is compact, therefore the function f(x) = (Ax, x) is weakly continuous. Furthermore, any bounded set in H is weakly compact. This lets us replace the infimum by minimum:

$\min _{x\in S_{k},\|x\|=1}(Ax,x)\leq \lambda _{k}.$

So

$\sup _{S_{k}}\min _{x\in S_{k},\|x\|=1}(Ax,x)\leq \lambda _{k}.$

Because equality is achieved when S_k = span{λ₁...λ_n},

$\max _{S_{k}}\min _{x\in S_{k},\|x\|=1}(Ax,x)=\lambda _{k}.$

This is the first part of min-max theorem for compact self-adjoint operators.

Analogouly, consider now a k - 1 dimensional subspace S_{k - 1}, whose the orthogonal compliment is denoted by S_{k - 1}^⊥. If S' = span{u₁...u_k},

$S'\cap S_{k-1}^{\perp }\neq {0}.$

So

$\exists x\in S_{k-1}^{\perp }\;s.t.\;\|x\|=1,(Ax,x)\geq \lambda _{k}.$

This implies

$\max _{x\in S_{k-1}^{\perp },\|x\|=1}(Ax,x)\geq \lambda _{k}$

where the compactness of A was applied. Index the above by the collection of k - 1 dimensional subspaces gives

$\inf _{S_{k-1}}\max _{x\in S_{k-1}^{\perp },\|x\|=1}(Ax,x)\geq \lambda _{k}.$

Pick S_{k - 1} = span{u₁...u_k-1} and we deduce

$\min _{S_{k-1}}\max _{x\in S_{k-1}^{\perp },\|x\|=1}(Ax,x)=\lambda _{k}.$

In summary,

Theorem (Min-Max) Let A be a compact, self-adjoint operator on a Hilbert space H, whose positive eigenvalues are listed in decreasing order:

$\cdots \leq \lambda _{k}\leq \cdots \leq \lambda _{1}.$

Then

$\max _{S_{k}}\min _{x\in S_{k},\|x\|=1}(Ax,x)=\lambda _{k}^{\downarrow },$

$\min _{S_{k-1}}\max _{x\in S_{k-1}^{\perp },\|x\|=1}(Ax,x)=\lambda _{k}^{\downarrow }.$

A similar pair of equalities hold for negative eigenvalues.

References

• M. Reed and B. Simon, Methods of Modern Mathematical Physics IV: Analysis of Operators, Academic Press, 1978.