# Universal approximation theorem

(Redirected from Cybenko Theorem)

In the mathematical theory of artificial neural networks, the universal approximation theorem states[1] that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of Rn, under mild assumptions on the activation function. The theorem thus states that simple neural networks can represent a wide variety of interesting functions when given appropriate parameters; however, it does not touch upon the algorithmic learnability of those parameters.

One of the first versions of the theorem was proved by George Cybenko in 1989 for sigmoid activation functions.[2]

Kurt Hornik showed in 1991[3] that it is not the specific choice of the activation function, but rather the multilayer feedforward architecture itself which gives neural networks the potential of being universal approximators. The output units are always assumed to be linear. For notational convenience, only the single output case will be shown. The general case can easily be deduced from the single output case.

## Formal statement

The theorem[2][3][4][5] in mathematical terms:

Let ${\displaystyle \varphi (\cdot )}$ be a nonconstant, bounded, and monotonically-increasing continuous function. Let ${\displaystyle I_{m}}$ denote the m-dimensional unit hypercube ${\displaystyle [0,1]^{m}}$. The space of continuous functions on ${\displaystyle I_{m}}$ is denoted by ${\displaystyle C(I_{m})}$. Then, given any function ${\displaystyle f\in C(I_{m})}$ and ${\displaystyle \varepsilon >0}$, there exists an integer ${\displaystyle N}$, real constants ${\displaystyle v_{i},b_{i}\in \mathbb {R} }$ and real vectors ${\displaystyle w_{i}\in \mathbb {R} ^{m}}$, where ${\displaystyle i=1,\cdots ,N}$, such that we may define:

${\displaystyle F(x)=\sum _{i=1}^{N}v_{i}\varphi \left(w_{i}^{T}x+b_{i}\right)}$

as an approximate realization of the function ${\displaystyle f}$ where ${\displaystyle f}$ is independent of ${\displaystyle \varphi }$; that is,

${\displaystyle |F(x)-f(x)|<\varepsilon }$

for all ${\displaystyle x\in I_{m}}$. In other words, functions of the form ${\displaystyle F(x)}$ are dense in ${\displaystyle C(I_{m})}$.

This still holds when replacing ${\displaystyle I_{m}}$ with any compact subset of ${\displaystyle \mathbb {R} ^{m}}$.