# Functional data analysis

Functional data analysis (FDA) is a branch of statistics that analyzes data providing information about curves, surfaces or anything else varying over a continuum. In its most general form, under an FDA framework each sample element is considered to be a function. The physical continuum over which these functions are defined is often time, but may also be spatial location, wavelength, probability, etc.

## History

Functional data analysis has roots going back to work by Grenander and Karhunen in the 1940s and 1950s.[1][2][3] They considered the decomposition of square-integrable continuous time stochastic process into eigencomponents, now known as the Karhunen-Loève decomposition. A rigorous analysis of functional principal components analysis was done in the 1970s by Kleffe, Dauxois and Pousse including results about the asymptotic distribution of the eigenvalues.[4][5][6] More recently in the 1990s and 2000s the field has focused more on application and understanding the effects of dense and sparse observations schemes. Core contributions in this era were made by James O. Ramsay (who coined the term "functional data analysis" in this period), Bernard Silverman and John Rice.[7][8][9]

## Mathematical formalism

Random functions can be viewed as random elements taking values in a Hilbert space, or as a stochastic process. The former is mathematically convenient, whereas the latter is somewhat more suitable from an applied perspective. These two approaches coincide if the random functions are continuous and a condition called mean-squared continuity is satisfied. For more on the probabilistic underpinnings of functional data analysis, see Chapter 7.[10]

### Hilbertian random variables

In the Hilbert space viewpoint, one considers an ${\displaystyle H}$-valued random element ${\displaystyle X}$, where ${\displaystyle H}$ is a separable Hilbert space such as the space of square-integrable functions ${\displaystyle L^{2}[0,1]}$. Under the integrability condition that ${\displaystyle \mathbb {E} \|X\|}$ is finite, one can define the mean of ${\displaystyle X}$ as the unique element ${\displaystyle \mu \in H}$ satisfying

${\displaystyle \mathbb {E} \langle X,h\rangle =\langle \mu ,h\rangle ,\qquad h\in H.}$

This formulation is the Pettis integral but the mean can also be defined as ${\displaystyle \mu =\mathbb {E} X}$ the Bochner sense. Under the integrability condition that ${\displaystyle \mathbb {E} \|X\|^{2}}$ is finite, the covariance operator of ${\displaystyle X}$ is a linear operator ${\displaystyle {\mathcal {C}}:H\to H}$ that is uniquely defined by the relation

${\displaystyle {\mathcal {C}}h=\mathbb {E} [\langle h,X-\mu \rangle (X-\mu )],\qquad h\in H,}$

or, in tensor form, ${\displaystyle {\mathcal {C}}=\mathbb {E} [(X-\mu )\otimes (X-\mu )]}$. The spectral theorem allows to decompose ${\displaystyle X}$ as the Karhunen-Loève decomposition

${\displaystyle X=\mu +\sum _{i=1}^{\infty }\langle X,\varphi _{i}\rangle \varphi _{i},}$

where ${\displaystyle \varphi _{i}}$ are eigenvectors of ${\displaystyle {\mathcal {S}}}$, corresponding to the nonnegative eigenvalues of ${\displaystyle {\mathcal {S}}}$, in a nonincreasing order. Truncating this infinite series to a finite order underpins functional principal component analysis.

### Stochastic processes

The Hilbertian point of view is mathematically convenient, but abstract; the above considerations do not necessarily even view ${\displaystyle X}$ as a function at all, since common choices of ${\displaystyle H}$ like ${\displaystyle L^{2}[0,1]}$ and Sobolev spaces consist of equivalence classes, not functions. The stochastic process perspective views ${\displaystyle X}$ as a collection of random variables

${\displaystyle \{X(t)\}_{t\in [0,1]}}$

indexed by the unit interval (or more generally some compact metric space ${\displaystyle K}$). The mean and covariance functions are defined in a pointwise manner as

${\displaystyle \mu (t)=\mathbb {E} X(t),\qquad c(s,t)={\textrm {Cov}}[(X(s),X(t)],\qquad s,t\in [0,1]}$

(if ${\displaystyle \mathbb {E} [X(t)^{2}]<\infty }$ for all ${\displaystyle t\in [0,1]}$). We can hope to view ${\displaystyle X}$ as a random element on the Hilbert function space ${\displaystyle H=L^{2}[0,1]}$. However, additional conditions are required for such a pursuit to be fruitful, since if we let ${\displaystyle X(t)}$ be Gaussian white noise, i.e. ${\displaystyle X(t)}$ is standard Gaussian and independent from ${\displaystyle X(s)}$ for any ${\displaystyle s,t\in [0,1]}$, it is clear that we have no hope of viewing this as a square integrable function.

A convenient sufficient condition is mean square continuity, stipulating that ${\displaystyle \mu }$ and ${\displaystyle c}$ are continuous functions. In this case ${\displaystyle c}$ defines a covariance operator ${\displaystyle {\mathcal {C}}:H\to H}$ by

${\displaystyle ({\mathcal {C}}f)(t)=\int _{K}c(s,t)f(s)\,\mathrm {d} s.}$

The spectral theorem applies to ${\displaystyle {\mathcal {C}}}$, yielding eigenpairs ${\displaystyle (\lambda _{j},\varphi _{j})}$, so that in tensor product notation ${\displaystyle {\mathcal {C}}}$ writes

${\displaystyle {\mathcal {C}}=\sum _{j=1}^{\infty }\lambda _{j}\varphi _{j}\otimes \varphi _{j}.}$

Moreover, since ${\displaystyle {\mathcal {S}}f}$ is continuous for all ${\displaystyle f\in H}$, all the ${\displaystyle \varphi _{j}}$'s are continuous. Mercer's theorem then states that the covariance function ${\displaystyle c}$ admits an analogous decomposition

${\displaystyle \sup _{s,t\in [0,1]}\left|c(s,t)-\sum _{j=1}^{N}\lambda _{j}\varphi _{j}(s)\varphi _{j}(t)\right|\to 0,\qquad N\to \infty .}$

Finally, under the extra assumption that ${\displaystyle X}$ has continuous sample paths, namely that with probability one, the random function ${\displaystyle X:[0,1]\to \mathbb {R} }$ is continuous, the Karhunen-Loève expansion above holds for ${\displaystyle X}$ and the Hilbert space machinery can be subsequently applied. Continuity of sample paths can be shown using Kolmogorov continuity theorem.

## Regression methods for functional data

Several methods have been developed for simple functional data.

### Scalar-on-function regression

A well-studied model for scalar-on-function regression is a generalisation of linear regression. Classical linear regression assumes that a scalar variable of interest ${\displaystyle Y}$ is related to a ${\displaystyle p}$-dimensional covariate vector ${\displaystyle X}$ through the equation

${\displaystyle Y=\langle X,\beta \rangle +\varepsilon =X_{1}\beta _{1}+\dots +X_{p}\beta _{p}+\varepsilon }$

for a ${\displaystyle p}$-dimensional vector of coefficients ${\displaystyle \beta }$ and a scalar noise variable ${\displaystyle \varepsilon }$, where ${\displaystyle \langle X,\beta \rangle }$ denotes the standard inner product on ${\displaystyle \mathbb {R} ^{p}}$. If we instead observe a functional variable ${\displaystyle X}$, which we assume is an element of the space of square-integrable functions on the unit interval, we can consider the same linear regression model as above using the ${\displaystyle L^{2}[0,1]}$ inner product. In other words, we consider the model

${\displaystyle Y=\int _{0}^{1}X(t)\beta (t)\,\mathrm {d} t+\varepsilon }$

for a square-integrable coefficient function ${\displaystyle \beta }$ and ${\displaystyle \varepsilon }$ as before (see Chapter 13[7]).

### Function-on-scalar regression

Analogously to the scalar-on-function regression model, we can consider a functional ${\displaystyle Y}$ and ${\displaystyle p}$-dimensional covariate vector ${\displaystyle X}$ and again draw inspiration from the usual linear regression model by modelling ${\displaystyle Y}$ as a linear combination of ${\displaystyle p}$ functions. In other words, we assume that the relationship between ${\displaystyle Y}$ and ${\displaystyle X}$ is

${\displaystyle Y(t)=X_{1}\beta _{1}(t)+\dots X_{p}\beta _{p}(t)+\varepsilon (t)}$

for functions ${\displaystyle \beta _{1},\dots ,\beta _{p}}$ and functional error term ${\displaystyle \varepsilon }$.

### Function-on-function regression

Both of the previous regression models can be viewed as instances of a general linear model between Hilbert spaces. Assuming that ${\displaystyle X}$ and ${\displaystyle Y}$ are elements of Hilbert spaces ${\displaystyle H_{X}}$ and ${\displaystyle H_{Y}}$, the Hilbertian linear model assumes that

${\displaystyle Y={\mathcal {S}}X+\varepsilon }$

for a Hilbert-Schmidt operator ${\displaystyle {\mathcal {S}}:H_{X}\to H_{Y}}$ and a noise variable ${\displaystyle \varepsilon }$ taking values in ${\displaystyle H_{Y}}$. If ${\displaystyle H_{X}=L^{2}[0,1]}$ and ${\displaystyle H_{Y}=\mathbb {R} }$, we get the scalar-on-function regression model above. Similarly, if ${\displaystyle H_{X}=\mathbb {R} ^{p}}$ and ${\displaystyle H_{Y}=L^{2}[0,1]}$ then we get the function-on-scalar regression model above. If we let ${\displaystyle H_{X}=H_{Y}=L^{2}[0,1]}$, we get the function-on-function linear regression model which can equivalently be written

${\displaystyle Y(t)=\int _{0}^{1}X(s)\beta (s,t)\,\mathrm {d} s+\varepsilon (t)}$

for a square-integrable coefficient function ${\displaystyle \beta }$ and functional noise variable ${\displaystyle \varepsilon }$.

### Practical considerations

While the presentation of the models above assumes fully-observed functions, software is available for fitting the models with discretely-observed functions in software such as R. Packages for R include refund [11] and FDboost [12] that utilise reformulations of the functional models as generalized additive models and boosted models, respectively.

• Ramsay, J. O. and Silverman, B.W. (2002) Applied functional data analysis : methods and case studies, Springer series in statistics, New York ; London : Springer, ISBN 0-387-95414-7
• Ramsay, J. O. and Silverman, B.W. (2005) Functional data analysis, 2nd ed., New York : Springer, ISBN 0-387-40080-X
• James O. Ramsay; Giles Hooker; Spencer Graves (2009). Functional data analysis with R and Matlab. Springer Science+Business Media. ISBN 978-0-387-98184-0. Wikidata Q105545530.
• Hsing, T. and Eubank, R. (2015) Theoretical Foundations of Functional Data Analysis, with an Introduction to Linear Operators, Wiley series in probability and statistics, John Wiley & Sons, Ltd, ISBN 978-0-470-01691-6
• Morris, J. (2015) Functional Regression, Annual Review of Statistics and Its Application, Vol. 2, 321 - 359, https://doi.org/10.1146/annurev-statistics-010814-020413
• Wang et al. (2016) Functional Data Analysis, Annual Review of Statistics and Its Application, Vol. 3, 257-295, https://doi.org/10.1146/annurev-statistics-041715-033624

## References

1. ^ Grenander, Ulf (1950). "Stochastic processes and statistical inference". Arkiv för Matematik. 1 (3): 195–277. doi:10.1007/BF02590638. ISSN 0004-2080. Retrieved 2021-01-27.
2. ^ Müller, Hans-Georg (2016). "Peter Hall, functional data analysis and random objects". The Annals of Statistics. 44 (5): 1867–1887. doi:10.1214/16-AOS1492. ISSN 0090-5364. Retrieved 2021-01-27.
3. ^ Karhunen, Kari (1946). "Zur Spektraltheorie stochastischer Prozesse". Annales Academiae Scientiarum Fennicae. Series A. I. Mathematica-Physica. 1946 (34): 7. ISSN 0365-2300. MR 0023012. Retrieved 2021-01-27.
4. ^ Kleffe, Jürgen (1973). "Principal components of random variables with values in a seperable hilbert space". Mathematische Operationsforschung und Statistik. 4 (5): 391–406. doi:10.1080/02331887308801137. ISSN 0047-6277. Retrieved 2021-01-27.
5. ^ Dauxois, J.; Pousse, A. (1976). Les analyses factorielles en calcul des probabilités et en statistique: essai d'étude synthétique.
6. ^ Dauxois, J.; Pousse, A.; Romain, Y. (1982). "Asymptotic theory for the principal component analysis of a vector random function: Some applications to statistical inference". Journal of Multivariate Analysis. 12 (1): 136–154. doi:10.1016/0047-259X(82)90088-4. ISSN 0047-259X. Retrieved 2021-01-27.
7. ^ a b Ramsay, James O.; Silverman, Bernard W. (2005). Functional Data Analysis (2nd ed.). Springer. ISBN 0-387-40080-X.
8. ^ Rice, John A.; Silverman, B. W. (1991). "Estimating the Mean and Covariance Structure Nonparametrically When the Data are Curves". Journal of the Royal Statistical Society, Series B (Methodological). 53 (1): 233–243. doi:10.1111/j.2517-6161.1991.tb01821.x. ISSN 2517-6161. Retrieved 2021-01-27.
9. ^ Rice, John A.; Wu, Colin O. (2001). "Nonparametric Mixed Effects Models for Unequally Sampled Noisy Curves". Biometrics. 57 (1): 253–259. doi:10.1111/j.0006-341X.2001.00253.x. ISSN 0006-341X. PMID 11252607. Retrieved 2021-01-27.
10. ^ Hsing, Tailen; Eubank, Randall (2015-05-18). Theoretical Foundations of Functional Data Analysis, with an Introduction to Linear Operators: Hsing/Theoretical Foundations of Functional Data Analysis, with an Introduction to Linear Operators. Wiley Series in Probability and Statistics. Chichester, UK: John Wiley & Sons, Ltd. doi:10.1002/9781118762547. ISBN 978-0-470-01691-6. Retrieved 2021-02-04.
11. ^
12. ^