Matrix norm

For the general concept, see Norm (mathematics).

In mathematics, a matrix norm is a vector norm in a vector space whose elements (vectors) are matrices (of given dimensions).

Definition

Given a field $K$ of either real or complex numbers, and the vector space $K^{m\times n}$ of all matrices of size $m\times n$ (with $m$ rows and $n$ columns) with entries in the field $K$, a matrix norm is a norm on the vector space $K^{m\times n}$ (with individual norms denoted using double vertical bars such as $\|A\|$). Thus, the matrix norm is a function $\|\cdot \|:K^{m\times n}\to \mathbb {R}$ that must satisfy the following properties:

For all scalars $\alpha \in K$ and for all matrices $A,B\in K^{m\times n}$,

• $\|A\|\geq 0$ (being positive-valued)
• $\|A\|=0\iff A=0_{m,n}$ (being definite)
• $\|\alpha A\|=|\alpha |\|A\|$ (being absolutely homogeneous)
• $\|A+B\|\leq \|A\|+\|B\|$ (being sub-additive or satisfying the triangle inequality)

Additionally, in the case of square matrices (matrices with m = n), some (but not all) matrix norms satisfy the following condition, which is related to the fact that matrices are more than just vectors:

• $\|AB\|\leq \|A\|\|B\|$ for all matrices $A$ and $B$ in $K^{n\times n}$ (being sub-multiplicative)

A matrix norm that satisfies this additional property is called a sub-multiplicative norm (in some books, the terminology matrix norm is used only for those norms which are sub-multiplicative). The set of all $n\times n$ matrices, together with such a submultiplicative norm, is an example of a Banach algebra.

The definition of sub-multiplicativity is sometimes extended to non-square matrices, as in the case of the induced p-norm, where for $A\in {K}^{m\times n}$ and $B\in {K}^{n\times k}$ holds that $\|AB\|_{q}\leq \|A\|_{p}\|B\|_{q}$. Here, $\|\cdot \|_{p}$ and $\|\cdot \|_{q}$ are the norms induced from $K^{m}$ and $K^{n}$, respectively, where p,q ≥ 1.

There are three types of matrix norms which will be discussed below:

• Matrix norms induced by vector norms,
• Entry-wise matrix norms, and
• Schatten norms.

Matrix norms induced by vector norms

Main article: Operator norm

Suppose a vector norm $\|\cdot \|$ on $K^{m}$ is given. Any $m\times n$ matrix A induces a linear operator from $K^{n}$ to $K^{m}$ with respect to the standard basis, and one defines the corresponding induced norm or operator norm on the space $K^{m\times n}$ of all $m\times n$ matrices as follows:

${\begin{aligned}\|A\|&=\sup\{\|Ax\|:x\in K^{n}{\text{ with }}\|x\|=1\}\\&=\sup \left\{{\frac {\|Ax\|}{\|x\|}}:x\in K^{n}{\text{ with }}x\neq 0\right\}.\end{aligned}}$

In particular, if the p-norm for vectors (1 ≤ p ≤ ∞) is used for both spaces $K^{n}$ and $K^{m}$, then the corresponding induced operator norm is:

$\|A\|_{p}=\sup _{x\neq 0}{\frac {\|Ax\|_{p}}{\|x\|_{p}}}.$

These induced norms are different from the "entry-wise" p-norms and the Schatten p-norms for matrices treated below, which are also usually denoted by $\|A\|_{p}.$

Note: The above description pertains to the induced operator norm when the same vector norm was used in the "departure space" $K^{n}$ and the "arrival space" $K^{m}$ of the operator $A\in K^{m\times n}$. This is not a necessary restriction. More generally, given a norm $\|\cdot \|_{\alpha }$ on $K^{n}$ and a norm $\|\cdot \|_{\beta }$ on $K^{m}$, one can define a matrix norm on $K^{m\times n}$ induced by these norms:

$\|A\|_{\alpha ,\beta }=\max _{x\neq 0}{\frac {\|Ax\|_{\beta }}{\|x\|_{\alpha }}}.$

The matrix norm $\|A\|_{\alpha ,\beta }$ is sometimes called a subordinate norm. Subordinate norms are consistent with the norms that induce them, giving

$\|Ax\|_{\beta }\leq \|A\|_{\alpha ,\beta }\|x\|_{\alpha }.$

Any induced operator norm is a sub-multiplicative matrix norm: $\|AB\|\leq \|A\|\|B\|;$ this follows from

$\|ABx\|\leq \|A\|\|Bx\|\leq \|A\|\|B\|\|x\|$

and

$\max _{\|x\|=1}\|ABx\|=\|AB\|.$

Moreover, any induced norm satisfies the inequality

$\|A^{r}\|^{1/r}\geq \rho (A)\quad$ (1)

for all positive integers r, where ρ(A) is the spectral radius of A. For symmetric or hermitian A, we have equality in (1) for the 2-norm, since in this case the 2-norm is precisely the spectral radius of A. For an arbitrary matrix, we may not have equality for any norm; a counterexample would be

$A={\begin{bmatrix}0&1\\0&0\end{bmatrix}},$

which has vanishing spectral radius. In any case, for square matrices we have the spectral radius formula:

$\lim _{r\to \infty }\|A^{r}\|^{1/r}=\rho (A).$

Compatible and consistent norms

A matrix norm $\|\cdot \|$ on $K^{m\times n}$ is called consistent with a vector norm $\|\cdot \|_{a}$ on $K^{n}$ and a vector norm $\|\cdot \|_{b}$ on $K^{m}$, if:

$\|Ax\|_{b}\leq \|A\|\|x\|_{a}$

for all $A\in K^{m\times n},x\in K^{n}$. In the special case of m = n and a = b, $\|\cdot \|$ might also be called compatible with $\|\cdot \|_{a}$. All induced norms are consistent by definition. Also, any sub-multiplicative matrix norm on $K^{n\times n}$ (considered as $(K^{n})^{n}$) induces a compatible vector norm on $K^{n}$ by defining $\|v\|:=\|\left(v,v,...v\right)\|$.

Special cases

In the special cases of $p=1,2,\infty ,$ the induced matrix norms can be computed or estimated by

$\|A\|_{1}=\max _{1\leq j\leq n}\sum _{i=1}^{m}|a_{ij}|,$

which is simply the maximum absolute column sum of the matrix;

$\|A\|_{\infty }=\max _{1\leq i\leq m}\sum _{j=1}^{n}|a_{ij}|,$

which is simply the maximum absolute row sum of the matrix.

In the special case of $p=2$ (the Euclidean norm or $\ell _{2}$-norm for vectors), the induced matrix norm is the spectral norm. The spectral norm of a matrix $A$ is the largest singular value of $A$ (i.e., the square root of the largest eigenvalue of the matrix $A^{*}A$, where $A^{*}$ denotes the conjugate transpose of $A$):

$\|A\|_{2}={\sqrt {\lambda _{\max }\left(A^{*}A\right)}}=\sigma _{\max }(A).$

where $\sigma _{\max }(A)$ represents the largest singular value of matrix $A$. Also,

$\|A^{*}A\|_{2}=\|AA^{*}\|_{2}=\|A\|_{2}^{2}$

since $\|A^{*}A\|_{2}=\sigma _{\max }(A^{*}A)=\sigma _{\max }(A)^{2}=\|A\|_{2}^{2}$ and similarly $\|AA^{*}\|_{2}=\|A\|_{2}^{2}$ by singular value decomposition (SVD). There is another important inequality:

$\|A\|_{2}=\sigma _{\max }(A)\leq \|A\|_{\rm {F}}=\left(\sum _{i=1}^{m}\sum _{j=1}^{n}|a_{ij}|^{2}\right)^{\frac {1}{2}}=\left(\sum _{k=1}^{\min(m,n)}\sigma _{k}^{2}\right)^{\frac {1}{2}},$

where $\|A\|_{\rm {F}}$ is the Frobenius norm. Equality holds if and only if the matrix $A$ is a rank-one matrix or a zero matrix. This inequality can be derived from the fact that the trace of a matrix is equal to the sum of its eigenvalues.

When $p=2$ we have an equivalent definition for $\|A\|_{2}$ as $\sup\{x^{T}Ay:x,y\in K^{n}{\text{ with }}\|x\|_{2}=\|y\|_{2}=1\}$. It can be shown to be equivalent to the above definitions using the Cauchy–Schwarz inequality.

For example, for

$A={\begin{bmatrix}-3&5&7\\2&6&4\\0&2&8\\\end{bmatrix}},$

we have that

$\|A\|_{1}=\max(|{-3}|+2+0;5+6+2;7+4+8)=\max(5,13,19)=19,$

$\|A\|_{\infty }=\max(|{-3}|+5+7;2+6+4;0+2+8)=\max(15,12,10)=15.$

"Entry-wise" matrix norms

These norms treat an $m\times n$ matrix as a vector of size $m\cdot n$, and use one of the familiar vector norms. For example, using the p-norm for vectors, p ≥ 1, we get:

$\|A\|_{p,p}=\|\mathrm {vec} (A)\|_{p}=\left(\sum _{i=1}^{m}\sum _{j=1}^{n}|a_{ij}|^{p}\right)^{1/p}$

This is a different norm from the induced p-norm (see above) and the Schatten p-norm (see below), but the notation is the same.

The special case p = 2 is the Frobenius norm, and p = ∞ yields the maximum norm.

L_2,1 and L_p,q norms

Let $(a_{1},\ldots ,a_{n})$ be the columns of matrix $A$. The $L_{2,1}$ norm is the sum of the Euclidean norms of the columns of the matrix:

$\|A\|_{2,1}=\sum _{j=1}^{n}\|a_{j}\|_{2}=\sum _{j=1}^{n}\left(\sum _{i=1}^{m}|a_{ij}|^{2}\right)^{\frac {1}{2}}$

The $L_{2,1}$ norm as an error function is more robust, since the error for each data point (a column) is not squared. It is used in robust data analysis and sparse coding.

For p, q ≥ 1, the $L_{2,1}$ norm can be generalized to the $L_{p,q}$ norm as follows:

$\|A\|_{p,q}=\left(\sum _{j=1}^{n}\left(\sum _{i=1}^{m}|a_{ij}|^{p}\right)^{\frac {q}{p}}\right)^{\frac {1}{q}}.$

Frobenius norm

Main article: Hilbert–Schmidt operator

See also: Frobenius inner product

When p = q = 2 for the $L_{p,q}$ norm, it is called the Frobenius norm or the Hilbert–Schmidt norm, though the latter term is used more frequently in the context of operators on (possibly infinite-dimensional) Hilbert space. This norm can be defined in various ways:

$\|A\|_{\text{F}}={\sqrt {\sum _{i=1}^{m}\sum _{j=1}^{n}|a_{ij}|^{2}}}={\sqrt {\operatorname {trace} \left(A^{*}A\right)}}={\sqrt {\sum _{i=1}^{\min\{m,n\}}\sigma _{i}^{2}(A)}},$

where $\sigma _{i}(A)$ are the singular values of $A$. Recall that the trace function returns the sum of diagonal entries of a square matrix.

The Frobenius norm is an extension of the Euclidean norm to $K^{n\times n}$ and comes from the Frobenius inner product on the space of all matrices.

The Frobenius norm is sub-multiplicative and is very useful for numerical linear algebra. The sub-multiplicativity of Frobenius norm can be proved using Cauchy–Schwarz inequality.

Frobenius norm is often easier to compute than induced norms, and has the useful property of being invariant under rotations (and unitary operations in general). That is, $\|A\|_{\text{F}}=\|AU\|_{\text{F}}=\|UA\|_{\text{F}}$ for any unitary matrix $U$. This property follows from the cyclic nature of the trace ($\operatorname {trace} (XYZ)=\operatorname {trace} (ZXY)$):

$\|AU\|_{\text{F}}^{2}=\operatorname {trace} \left((AU)^{*}AU\right)=\operatorname {trace} \left(U^{*}A^{*}AU\right)=\operatorname {trace} \left(UU^{*}A^{*}A\right)=\operatorname {trace} \left(A^{*}A\right)=\|A\|_{\text{F}}^{2},$

and analogously:

$\|UA\|_{\text{F}}^{2}=\operatorname {trace} \left((UA)^{*}UA\right)=\operatorname {trace} \left(A^{*}U^{*}UA\right)=\operatorname {trace} \left(A^{*}A\right)=\|A\|_{\text{F}}^{2},$

where we have used the unitary nature of $U$ (that is, $U^{*}U=UU^{*}=\mathbf {I}$).

It also satisfies

$\|A^{*}A\|_{\text{F}}=\|AA^{*}\|_{\text{F}}\leq \|A\|_{\text{F}}^{2}$

and

$\|A+B\|_{\text{F}}^{2}=\|A\|_{\text{F}}^{2}+\|B\|_{\text{F}}^{2}+2\langle A,B\rangle _{\text{F}},$

where $\langle A,B\rangle _{\text{F}}$ is the Frobenius inner product.

Max norm

The max norm is the elementwise norm with p = q = ∞:

$\|A\|_{\max }=\max _{ij}|a_{ij}|.$

This norm is not sub-multiplicative.

Note that in some literature (such as Communication complexity), an alternative definition of max-norm, also called the $\gamma _{2}$-norm, refers to the factorization norm:

$\gamma _{2}(A)=\min _{U,V:A=UV^{T}}\|U\|_{2,\infty }\|V\|_{2,\infty }=\min _{U,V:A=UV^{T}}\max _{i,j}\|U_{i,:}\|_{2}\|V_{j,:}\|_{2}$

Schatten norms

Further information: Schatten norm

The Schatten p-norms arise when applying the p-norm to the vector of singular values of a matrix. If the singular values of the $m\times n$ matrix $A$ are denoted by σ_i, then the Schatten p-norm is defined by

$\|A\|_{p}=\left(\sum _{i=1}^{\min\{m,n\}}\sigma _{i}^{p}(A)\right)^{\frac {1}{p}}.$

These norms again share the notation with the induced and entry-wise p-norms, but they are different.

All Schatten norms are sub-multiplicative. They are also unitarily invariant, which means that $\|A\|=\|UAV\|$ for all matrices $A$ and all unitary matrices $U$ and $V$.

The most familiar cases are p = 1, 2, ∞. The case p = 2 yields the Frobenius norm, introduced before. The case p = ∞ yields the spectral norm, which is the operator norm induced by the vector 2-norm (see above). Finally, p = 1 yields the nuclear norm (also known as the trace norm, or the Ky Fan 'n'-norm), defined as

$\|A\|_{*}=\operatorname {trace} \left({\sqrt {A^{*}A}}\right)=\sum _{i=1}^{\min\{m,n\}}\sigma _{i}(A),$

where ${\sqrt {A^{*}A}}$ denotes a positive semidefinite matrix $B$ such that $BB=A^{*}A$. More precisely, since $A^{*}A$ is a positive semidefinite matrix, its square root is well-defined. The nuclear norm $\|A\|_{*}$ is a convex envelope of the rank function ${\text{rank}}(A)$, so it is often used in mathematical optimization to search for low rank matrices.

Monotone norms

A matrix norm $\|\cdot \|$ is called monotone if it is monotonic with respect to the Loewner order. Thus, a matrix norm is increasing if

$A\preccurlyeq B\Rightarrow \|A\|\leq \|B\|.$

The Frobenius norm and spectral norm are examples of monotone norms.

Cut norms

Another source of inspiration for matrix norms arises from considering a matrix as the adjacency matrix of a weighted, directed graph. The so-called "cut norm" measures how close the associated graph is to being bipartite:

$\|A\|_{\Box }=\max _{S\subseteq [n],T\subseteq [m]}{\left|\sum _{s\in S,t\in T}{A_{t,s}}\right|}$

where A ∈ K^m×n. The cut-norm is equivalent to the induced operator norm ‖·‖_∞→1, which is itself equivalent to the another norm, called the Grothendieck norm.

To define the Grothendieck norm, first note that a linear operator K¹→K¹ is just a scalar, and thus extends to a linear operator on any K^k→K^k. Moreover, given any choice of basis for Kⁿ and K^m, any linear operator Kⁿ→K^m extends to a linear operator (K^k)ⁿ→(K^k)^m, by letting each matrix element on elements of K^k via scalar multiplication. In symbols:

$\|A\|_{G,k}=\sup _{{\text{each }}u_{j},v_{j}\in K^{k};\|u_{j}\|=\|v_{j}\|=1}{\sum _{j\in [n],l\in [m]}{(u_{j}\cdot v_{j})A_{l,j}}}$

Equivalence of norms

See also: Equivalent norms

For any two matrix norms $\|\cdot \|_{\alpha }$ and $\|\cdot \|_{\beta }$, we have that:

$r\|A\|_{\alpha }\leq \|A\|_{\beta }\leq s\|A\|_{\alpha }$

for some positive numbers r and s, for all matrices $A\in K^{m\times n}$. In other words, all norms on $K^{m\times n}$ are equivalent; they induce the same topology on $K^{m\times n}$. This is true because the vector space $K^{m\times n}$ has the finite dimension $m\times n$.

Moreover, for every vector norm $\|\cdot \|$ on $\mathbb {R} ^{n\times n}$, there exists a unique positive real number $k$ such that $l\|\cdot \|$ is a sub-multiplicative matrix norm for every $l\geq k$.

A sub-multiplicative matrix norm $\|\cdot \|_{\alpha }$ is said to be minimal, if there exists no other sub-multiplicative matrix norm $\|\cdot \|_{\beta }$ satisfying $\|\cdot \|_{\beta }<\|\cdot \|_{\alpha }$.

Examples of norm equivalence

Let $\|A\|_{p}$ once again refer to the norm induced by the vector p-norm (as above in the Induced Norm section).

For matrix $A\in \mathbb {R} ^{m\times n}$ of rank $r$, the following inequalities hold:

• $\|A\|_{2}\leq \|A\|_{F}\leq {\sqrt {r}}\|A\|_{2}$
• $\|A\|_{F}\leq \|A\|_{*}\leq {\sqrt {r}}\|A\|_{F}$
• $\|A\|_{\max }\leq \|A\|_{2}\leq {\sqrt {mn}}\|A\|_{\max }$
• ${\frac {1}{\sqrt {n}}}\|A\|_{\infty }\leq \|A\|_{2}\leq {\sqrt {m}}\|A\|_{\infty }$
• ${\frac {1}{\sqrt {m}}}\|A\|_{1}\leq \|A\|_{2}\leq {\sqrt {n}}\|A\|_{1}.$

Another useful inequality between matrix norms is

$\|A\|_{2}\leq {\sqrt {\|A\|_{1}\|A\|_{\infty }}},$

which is a special case of Hölder's inequality.

See also

• Dual norm
• Logarithmic norm

Bibliography

• James W. Demmel, Applied Numerical Linear Algebra, section 1.7, published by SIAM, 1997.
• Carl D. Meyer, Matrix Analysis and Applied Linear Algebra, published by SIAM, 2000. [1]
• John Watrous, Theory of Quantum Information, 2.3 Norms of operators, lecture notes, University of Waterloo, 2011.
• Kendall Atkinson, An Introduction to Numerical Analysis, published by John Wiley & Sons, Inc 1989