View source for Maximum likelihood estimation

{{Short description|Method of estimating the parameters of a statistical model, given observations}}
{{About|the statistical techniques|computer data storage|partial-response maximum-likelihood}}

In [[statistics]], '''maximum likelihood estimation''' ('''MLE''') is a method of [[estimation theory|estimating]] the [[Statistical parameter|parameters]] of an assumed [[probability distribution]], given some observed data. This is achieved by [[Mathematical optimization|maximizing]] a [[likelihood function]] so that, under the assumed [[statistical model]], the [[Realization (probability)|observed data]] is most probable. The [[point estimate|point]] in the [[parameter space]] that maximizes the likelihood function is called the maximum likelihood estimate.<ref>{{cite book |last=Rossi |first=Richard J. |title=Mathematical Statistics: An Introduction to Likelihood Based Inference |location=New York |publisher=John Wiley & Sons |year=2018 |isbn=978-1-118-77104-4 |page=227 }}</ref> The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of [[statistical inference]].<ref>{{cite book |first1=David F. |last1=Hendry |author-link=David Forbes Hendry |first2=Bent |last2=Nielsen |title=Econometric Modeling: A Likelihood Approach |location=Princeton |publisher=Princeton University Press |year=2007 |isbn=978-0-691-13128-3 }}</ref><ref>{{cite book |first1=Raymond L. |last1=Chambers |first2=David G. |last2=Steel |first3=Suojin |last3=Wang |first4=Alan |last4=Welsh |title=Maximum Likelihood Estimation for Sample Surveys |location=Boca Raton |publisher=CRC Press |year=2012 |isbn=978-1-58488-632-7 }}</ref><ref>{{cite book |first1=Michael Don |last1=Ward |author-link=Michael D. Ward |first2=John S. |last2=Ahlquist |title=Maximum Likelihood for Social Science: Strategies for Analysis |location=New York |publisher=Cambridge University Press |year=2018 |isbn=978-1-107-18582-1 }}</ref>

If the likelihood function is [[Differentiable function|differentiable]], the [[derivative test]] for finding maxima can be applied. In some cases, the first-order conditions of the likelihood function can be solved analytically; for instance, the [[ordinary least squares]] estimator for a [[linear regression]] model maximizes the likelihood when the random errors are assumed to have [[Normal distribution|normal]] distributions with the same variance.<ref>{{cite book |last1=Press |first1=W.H. |last2=Flannery |first2=B.P. |last3=Teukolsky |first3=S.A. |last4=Vetterling |first4=W.T. |chapter=Least Squares as a Maximum Likelihood Estimator |title=Numerical Recipes in FORTRAN: The Art of Scientific Computing |edition=2nd |location=Cambridge |publisher=Cambridge University Press |pages=651–655 |year=1992 |isbn=0-521-43064-X |chapter-url=https://books.google.com/books?id=gn_4mpdN9WkC&pg=PA651 }}</ref>

From the perspective of [[Bayesian inference]], MLE is generally equivalent to [[maximum a posteriori estimation|maximum a posteriori (MAP) estimation]] with a [[Prior probability|prior distribution]] that is [[uniform distribution (continuous)|uniform]] in the region of interest. In [[frequentist inference]], MLE is a special case of an [[extremum estimator]], with the objective function being the likelihood.

== Principles ==

We model a set of observations as a random [[Sample (statistics)|sample]] from an unknown [[joint probability distribution]] which is expressed in terms of a set of [[statistical parameters|parameters]]. The goal of maximum likelihood estimation is to determine the parameters for which the observed data have the highest joint probability. We write the parameters governing the joint distribution as a vector <math>\; \theta = \left[ \theta_{1},\, \theta_2,\, \ldots,\, \theta_k \right]^{\mathsf{T}} \;</math> so that this distribution falls within a [[parametric family]] <math>\; \{ f(\cdot\,;\theta) \mid \theta \in \Theta \} \;,</math> where <math>\, \Theta \,</math> is called the ''[[parameter space]]'', a finite-dimensional subset of [[Euclidean space]]. Evaluating the joint density at the observed data sample <math>\; \mathbf{y} = (y_1, y_2, \ldots, y_n) \;</math> gives a real-valued function,
:<math>\mathcal{L}_{n}(\theta) = \mathcal{L}_{n}(\theta; \mathbf{y}) = f_{n}(\mathbf{y}; \theta) \;,</math>
which is called the [[likelihood function]]. For [[independent and identically distributed random variables]], <math>f_{n}(\mathbf{y}; \theta)</math> will be the product of univariate [[Probability density function|density functions]]:
:<math>f_{n}(\mathbf{y}; \theta) =  \prod_{k=1}^n \, f_k^\mathsf{univar}(y_k; \theta) ~.</math>

The goal of maximum likelihood estimation is to find the values of the model parameters that maximize the likelihood function over the parameter space,<ref name=":0">{{cite journal |last = Myung |first = I.J. |year = 2003 |title = Tutorial on maximum likelihood Estimation |journal = [[Journal of Mathematical Psychology]] |volume=47 |issue=1 |pages=90–100 |doi = 10.1016/S0022-2496(02)00028-7 }}</ref> that is
:<math>\hat{\theta} = \underset{\theta\in\Theta}{\operatorname{arg\;max}}\,\mathcal{L}_{n}(\theta\,;\mathbf{y}) ~.</math>
Intuitively, this selects the parameter values that make the observed data most probable. The specific value <math>~ \hat{\theta} = \hat{\theta}_{n}(\mathbf{y}) \in \Theta ~</math> that maximizes the likelihood function <math>\, \mathcal{L}_{n} \,</math> is called the maximum likelihood estimate. Further, if the function <math>\; \hat{\theta}_{n} : \mathbb{R}^{n} \to \Theta \;</math> so defined is [[measurable function|measurable]], then it is called the maximum likelihood [[estimator]]. It is generally a function defined over the [[sample space]], i.e. taking a given sample as its argument. A [[Necessity and sufficiency|sufficient but not necessary]] condition for its existence is for the likelihood function to be [[Continuous function|continuous]] over a parameter space <math>\, \Theta \,</math> that is [[Compact space|compact]].<ref>{{cite book |first1=Christian |last1=Gourieroux |first2=Alain |last2=Monfort |year=1995 |title=Statistics and Econometrics Models |publisher=Cambridge University Press |isbn=0-521-40551-3 |page=[https://archive.org/details/statisticseconom00gour_434/page/n172 161] |url=https://archive.org/details/statisticseconom00gour_434 |url-access=limited}}</ref> For an [[Open set|open]] <math>\, \Theta \,</math> the likelihood function may increase without ever reaching a supremum value.

In practice, it is often convenient to work with the [[natural logarithm]] of the likelihood function, called the [[log-likelihood]]:
: <math>
  \ell(\theta\,;\mathbf{y}) =  \ln \mathcal{L}_{n}(\theta\,;\mathbf{y}) ~.
  </math>
Since the logarithm is a [[monotonic function]], the maximum of <math>\; \ell(\theta\,;\mathbf{y}) \;</math> occurs at the same value of <math>\theta</math> as does the maximum of <math>\, \mathcal{L}_{n} ~.</math><ref>{{cite book |first=Edward J. |last=Kane |year=1968 |title=Economic Statistics and Econometrics |location=New York, NY |publisher=Harper & Row |page=[https://archive.org/details/economicstatisti00kane/page/n200 179] |url=https://archive.org/details/economicstatisti00kane |url-access=registration}}</ref> If <math>\ell(\theta\,;\mathbf{y})</math> is [[Differentiable function|differentiable]] in <math>\, \Theta \,,</math> [[Derivative test|sufficient conditions]] for the occurrence of a maximum (or a minimum) are
:<math>\frac{\partial \ell}{\partial \theta_{1}} = 0, \quad \frac{\partial \ell}{\partial \theta_{2}} = 0, \quad \ldots, \quad \frac{\partial \ell}{\partial \theta_{k}} = 0 ~,</math>
known as the likelihood equations. For some models, these equations can be explicitly solved for <math>\, \widehat{\theta\,} \,,</math> but in general no closed-form solution to the maximization problem is known or available, and an MLE can only be found via [[Mathematical optimization|numerical optimization]]. Another problem is that in finite samples, there may exist multiple [[Zero of a function|roots]] for the likelihood equations.<ref>{{cite book |first1=Christoper G. |last1=Small |first2=Jinfang |last2=Wang |year=2003 |chapter=Working with roots |title=Numerical Methods for Nonlinear Estimating Equations |publisher=Oxford University Press |isbn=0-19-850688-0 |pages=74–124 |chapter-url=https://books.google.com/books?id=hMrwQVllY5AC&pg=PA74 }}</ref> Whether the identified root <math>\, \widehat{\theta\,} \,</math> of the likelihood equations is indeed a (local) maximum depends on whether the matrix of second-order partial and cross-partial derivatives, the so-called [[Hessian matrix]]

:<math>\mathbf{H}\left(\widehat{\theta\,}\right) = \begin{bmatrix} \left. \frac{\partial^2 \ell}{\partial \theta_1^2} \right|_{\theta=\widehat{\theta\,}} & \left. \frac{\partial^2 \ell}{\partial \theta_1 \, \partial \theta_2} \right|_{\theta=\widehat{\theta\,}} & \dots & \left. \frac{\partial^2 \ell}{\partial \theta_1 \, \partial \theta_k} \right|_{\theta=\widehat{\theta\,}} \\ \left. \frac{\partial^2 \ell}{\partial \theta_2 \, \partial \theta_1} \right|_{\theta=\widehat{\theta\,}} & \left. \frac{\partial^2 \ell}{\partial \theta_2^2} \right|_{\theta=\widehat{\theta\,}} & \dots & \left. \frac{\partial^2 \ell}{\partial \theta_2 \, \partial \theta_k} \right|_{\theta=\widehat{\theta\,}} \\ \vdots & \vdots & \ddots & \vdots \\ \left. \frac{\partial^2 \ell}{\partial \theta_k \, \partial \theta_1} \right|_{\theta=\widehat{\theta\,}} & \left. \frac{\partial^2 \ell}{\partial \theta_k \, \partial \theta_2} \right|_{\theta=\widehat{\theta\,}} & \dots & \left. \frac{\partial^2 \ell}{\partial \theta_k^2} \right|_{\theta=\widehat{\theta\,}} \end{bmatrix} ~,</math>

is [[negative semi-definite]] at <math>\widehat{\theta\,}</math>, as this indicates local [[Concave function|concavity]]. Conveniently, most common [[probability distribution]]s – in particular the [[exponential family]] – are [[Logarithmically concave function|logarithmically concave]].<ref>
{{cite book
 |first1=Robert E. |last1=Kass
 |first2=Paul W. |last2=Vos
 |year=1997
 |title=Geometrical Foundations of Asymptotic Inference
 |page=14
 |location=New York, NY
 |publisher=John Wiley & Sons
 |isbn=0-471-82668-5
 |url=https://books.google.com/books?id=e43EAIfUPCwC&pg=PA14
}}
</ref><ref>
{{cite web
 |first=Alecos |last=Papadopoulos
 |date=25 September 2013
 |title=Why we always put log() before the joint pdf when we use MLE (Maximum likelihood Estimation)?
 |website=[[Stack Exchange]]
 |url=https://stats.stackexchange.com/q/70975
}}
</ref>

=== Restricted parameter space ===
{{Distinguish|restricted maximum likelihood}}
While the domain of the likelihood function—the [[parameter space]]—is generally a finite-dimensional subset of [[Euclidean space]], additional [[Restriction (mathematics)|restriction]]s sometimes need to be incorporated into the estimation process. The parameter space can be expressed as
:<math>\Theta = \left\{ \theta : \theta \in \mathbb{R}^{k},\; h(\theta) = 0 \right\} ~,</math>

where <math>\; h(\theta) = \left[ h_{1}(\theta), h_{2}(\theta), \ldots, h_{r}(\theta) \right] \;</math> is a [[vector-valued function]] mapping <math>\, \mathbb{R}^{k} \,</math> into <math>\; \mathbb{R}^{r} ~.</math> Estimating the true parameter <math>\theta</math> belonging to <math>\Theta</math> then, as a practical matter, means to find the maximum of the likelihood function subject to the [[Constraint (mathematics)|constraint]] <math>~h(\theta) = 0 ~.</math>

Theoretically, the most natural approach to this [[constrained optimization]] problem is the method of substitution, that is "filling out" the restrictions <math>\; h_{1}, h_{2}, \ldots, h_{r} \;</math> to a set <math>\; h_{1}, h_{2}, \ldots, h_{r}, h_{r+1}, \ldots, h_{k} \;</math> in such a way that <math>\; h^{\ast} = \left[ h_{1}, h_{2}, \ldots, h_{k} \right] \;</math> is a [[one-to-one function]] from <math>\mathbb{R}^{k}</math> to itself, and reparameterize the likelihood function by setting <math>\; \phi_{i} = h_{i}(\theta_{1}, \theta_{2}, \ldots, \theta_{k}) ~.</math><ref name="Silvey p79">{{cite book |first=S. D. |last=Silvey |year=1975 |title=Statistical Inference |location=London, UK |publisher=Chapman and Hall |isbn=0-412-13820-4 |page=79 |url=https://books.google.com/books?id=qIKLejbVMf4C&pg=PA79 }}</ref> Because of the equivariance of the maximum likelihood estimator, the properties of the MLE apply to the restricted estimates also.<ref>{{cite web |first=David |last=Olive |year=2004 |title=Does the MLE maximize the likelihood? |website=Southern Illinois University |url=http://lagrange.math.siu.edu/Olive/simle.pdf }}</ref> For instance, in a [[multivariate normal distribution]] the [[covariance matrix]] <math>\, \Sigma \,</math> must be [[Positive-definite matrix|positive-definite]]; this restriction can be imposed by replacing <math>\; \Sigma = \Gamma^{\mathsf{T}} \Gamma \;,</math> where <math>\Gamma</math> is a real [[upper triangular matrix]] and <math>\Gamma^{\mathsf{T}}</math> is its [[transpose]].<ref>{{cite journal |first=Daniel P. |last=Schwallie |year=1985 |title=Positive definite maximum likelihood covariance estimators |journal=Economics Letters |volume=17 |issue=1–2 |pages=115–117 |doi=10.1016/0165-1765(85)90139-9 }}</ref>

In practice, restrictions are usually imposed using the method of Lagrange which, given the constraints as defined above, leads to the ''restricted likelihood equations''
:<math>\frac{\partial \ell}{\partial \theta} - \frac{\partial h(\theta)^\mathsf{T}}{\partial \theta} \lambda = 0</math> and <math>h(\theta) = 0 \;,</math> 

where <math>~ \lambda = \left[ \lambda_{1}, \lambda_{2}, \ldots, \lambda_{r}\right]^\mathsf{T} ~</math> is a column-vector of [[Lagrange multiplier]]s and <math>\; \frac{\partial h(\theta)^\mathsf{T}}{\partial \theta} \;</math> is the {{mvar|k × r}} [[Jacobian matrix]] of partial derivatives.<ref name="Silvey p79"/> Naturally, if the constraints are not binding at the maximum, the Lagrange multipliers should be zero.<ref>{{cite book |first=Jan R. |last=Magnus |year=2017 |title=Introduction to the Theory of Econometrics |location=Amsterdam |publisher=VU University Press |pages=64–65 |isbn=978-90-8659-766-6}}</ref> This in turn allows for a statistical test of the "validity" of the constraint, known as the [[Lagrange multiplier test]].

=== Nonparametric maximum likelihood estimation ===
Nonparametric maximum likelihood estimation can be performed using the [[empirical likelihood]].

== Properties ==
A maximum likelihood estimator is an [[extremum estimator]] obtained by maximizing, as a function of ''θ'', the [[objective function]] <math>\widehat{\ell\,}(\theta\,;x)</math>. If the data are [[independent and identically distributed]], then we have 
: <math>
    \widehat{\ell\,}(\theta\,;x)= \sum_{i=1}^n \ln f(x_i\mid\theta),
  </math>
this being the sample analogue of the expected log-likelihood <math>\ell(\theta) = \operatorname{\mathbb E}[\, \ln f(x_i\mid\theta) \,]</math>, where this expectation is taken with respect to the true density.

Maximum-likelihood estimators have no optimum properties for finite samples, in the sense that (when evaluated on finite samples) other estimators may have greater concentration around the true parameter-value.<ref>{{harvtxt |Pfanzagl |1994 |p=206 }}</ref> However, like other estimation methods, maximum likelihood estimation possesses a number of attractive [[Asymptotic theory (statistics)|limiting properties]]: As the sample size increases to infinity, sequences of maximum likelihood estimators have these properties:
* [[consistency of an estimator|Consistency]]: the sequence of MLEs converges in probability to the value being estimated.
* [[Invariant estimator|Equivariance]]: If <math>
    \hat{\theta}
  </math> is the maximum likelihood estimator for <math>
    \theta
  </math>, and if <math>
    g(\theta)
  </math> is a bijective transform of <math>
    \theta
  </math>, then the maximum likelihood estimator for <math>
    \alpha = g(\theta )
  </math> is <math>
    \hat{\alpha} = g(\hat{\theta} )
  </math>. The equivariance property can be generalized to non-bijective transforms, although it applies in that case on the maximum of an induced likelihood function which is not the true likelihood in general.
* [[efficient estimator|Efficiency]], i.e. it achieves the [[Cramér–Rao lower bound]] when the sample size tends to infinity. This means that no consistent estimator has lower asymptotic [[mean squared error]] than the MLE (or other estimators attaining this bound), which also means that MLE has [[Local asymptotic normality|asymptotic normality]].
* Second-order efficiency after correction for bias.

=== Consistency ===
Under the conditions outlined below, the maximum likelihood estimator is [[consistent estimator|consistent]]. The consistency means that if the data were generated by <math>f(\cdot\,;\theta_0)</math> and we have a sufficiently large number of observations ''n'', then it is possible to find the value of ''θ''<sub>0</sub> with arbitrary precision. In mathematical terms this means that as ''n'' goes to infinity the estimator <math>\widehat{\theta\,}</math> [[convergence in probability|converges in probability]] to its true value:
: <math>
    \widehat{\theta\,}_\mathrm{mle}\ \xrightarrow{\text{p}}\ \theta_0.
</math>

Under slightly stronger conditions, the estimator converges [[almost sure convergence|almost surely]] (or ''strongly''):
: <math>
    \widehat{\theta\,}_\mathrm{mle}\ \xrightarrow{\text{a.s.}}\ \theta_0.
</math>

In practical applications, data is never generated by <math>f(\cdot\,;\theta_0)</math>. Rather, <math>f(\cdot\,;\theta_0)</math> is a model, often in idealized form, of the process generated by the data. It is a common aphorism in statistics that ''[[all models are wrong]]''. Thus, true consistency does not occur in practical applications. Nevertheless, consistency is often considered to be a desirable property for an estimator to have.

To establish consistency, the following conditions are sufficient.<ref>By Theorem 2.5 in {{cite book
  | last1 = Newey | first1 = Whitney K.
  | last2 = McFadden | first2 = Daniel | author-link2 = Daniel McFadden
  | chapter = Chapter 36: Large sample estimation and hypothesis testing
  | editor1-first= Robert | editor1-last=Engle | editor2-first=Dan | editor2-last=McFadden
  | title = Handbook of Econometrics, Vol.4
  | year = 1994
  | publisher = Elsevier Science
  | pages = 2111–2245
  | isbn=978-0-444-88766-5
  }}</ref>
{{ordered list
|1= [[Identifiability|Identification]] of the model:
: <math>
    \theta \neq \theta_0 \quad \Leftrightarrow \quad f(\cdot\mid\theta)\neq f(\cdot\mid\theta_0).
  </math>
In other words, different parameter values ''θ'' correspond to different distributions within the model. If this condition did not hold, there would be some value ''θ''<sub>1</sub> such that ''θ''<sub>0</sub> and ''θ''<sub>1</sub> generate an identical distribution of the observable data. Then we would not be able to distinguish between these two parameters even with an infinite amount of data—these parameters would have been [[observational equivalence|observationally equivalent]]. <br />
The identification condition is absolutely necessary for the ML estimator to be consistent. When this condition holds, the limiting likelihood function ''ℓ''(''θ''{{!}}·) has unique global maximum at ''θ''<sub>0</sub>.

|2= Compactness: the parameter space Θ of the model is [[compact set|compact]].
[[File:Ee noncompactness.svg|240px|right]]
The identification condition establishes that the log-likelihood has a unique global maximum. Compactness implies that the likelihood cannot approach the maximum value arbitrarily close at some other point (as demonstrated for example in the picture on the right).

Compactness is only a sufficient condition and not a necessary condition. Compactness can be replaced by some other conditions, such as:
{{unordered list
| both [[Concave function|concavity]] of the log-likelihood function and compactness of some (nonempty) upper [[level set]]s of the log-likelihood function, or
| existence of a compact [[Neighbourhood (mathematics)|neighborhood]] {{mvar|N}} of {{mvar|θ}}<sub>0</sub> such that outside of {{mvar|N}} the log-likelihood function is less than the maximum by at least some {{nowrap|{{mvar|ε}} > 0}}.
}}
|3= Continuity: the function {{math| ln ''f''(''x''&nbsp;{{!}}&nbsp;''θ'')}} is continuous in {{mvar|θ}} for almost all values of {{mvar|x}}:
: <math>
    \operatorname{\mathbb P} \Bigl[\; \ln f(x\mid\theta) \;\in\; C^0(\Theta) \;\Bigr] = 1.
  </math>
The continuity here can be replaced with a slightly weaker condition of [[upper semi-continuous|upper semi-continuity]].

|4= Dominance: there exists {{math|''D''(''x'')}} integrable with respect to the distribution {{math|''f''(''x''&nbsp;{{!}}&nbsp;''θ''<sub>0</sub>)}} such that
: <math>
    \Bigl|\ln f(x\mid\theta)\Bigr| < D(x) \quad \text{ for all } \theta\in\Theta.
  </math>

By the [[uniform law of large numbers]], the dominance condition together with continuity establish the uniform convergence in probability of the log-likelihood:
: <math>
    \sup_{\theta\in\Theta} \left|\widehat{\ell\,}(\theta\mid x) - \ell(\theta)\,\right|\ \xrightarrow{\text{p}}\ 0.
  </math>
}}

The dominance condition can be employed in the case of [[i.i.d.]] observations. In the non-i.i.d. case, the uniform convergence in probability can be checked by showing that the sequence <math>\widehat{\ell\,}(\theta\mid x)</math> is [[stochastic equicontinuity|stochastically equicontinuous]].

If one wants to demonstrate that the ML estimator <math>\widehat{\theta\,}</math> converges to ''θ''<sub>0</sub> [[almost sure convergence|almost surely]], then a stronger condition of uniform convergence almost surely has to be imposed:
: <math>
    \sup_{\theta\in\Theta} \left\|\;\widehat{\ell\,}(\theta\mid x) - \ell(\theta)\;\right\| \ \xrightarrow{\text{a.s.}}\ 0.
  </math>

Additionally, if (as assumed above) the data were generated by <math>f(\cdot\,;\theta_0)</math>, then under certain conditions, it can also be shown that the maximum likelihood estimator [[Convergence in distribution|converges in distribution]] to a normal distribution. Specifically,<ref name=":1">By Theorem 3.3 in {{cite book
  | last1 = Newey | first1 = Whitney K.
  | last2 = McFadden | first2 = Daniel | author-link2 = Daniel McFadden
  | chapter = Chapter 36: Large sample estimation and hypothesis testing
  | editor1-first= Robert | editor1-last=Engle | editor2-first=Dan | editor2-last=McFadden
  | title = Handbook of Econometrics, Vol.4
  | year = 1994
  | publisher = Elsevier Science
  | pages = 2111–2245
  | isbn=978-0-444-88766-5
  }}</ref>
: <math>
    \sqrt{n}\left(\widehat{\theta\,}_\mathrm{mle} - \theta_0\right)\ \xrightarrow{d}\ \mathcal{N}\left(0,\, I^{-1}\right)
  </math>
where {{math|''I''}} is the [[Fisher information|Fisher information matrix]].

=== Functional invariance ===
The maximum likelihood estimator selects the parameter value which gives the observed data the largest possible probability (or probability density, in the continuous case). If the parameter consists of a number of components, then we define their separate maximum likelihood estimators, as the corresponding component of the MLE of the complete parameter. Consistent with this, if <math>\widehat{\theta\,}</math> is the MLE for <math>\theta</math>, and if <math>g(\theta)</math> is any transformation of <math>\theta</math>, then the MLE for <math>\alpha=g(\theta)</math> is by definition<ref>{{cite book |first=Shelemyahu |last=Zacks |title=The Theory of Statistical Inference |location=New York |publisher=John Wiley & Sons |year=1971 |isbn=0-471-98103-6 |page=223 }}</ref>

:<math>\widehat{\alpha} = g(\,\widehat{\theta\,}\,). \,</math>

It maximizes the so-called [[Likelihood function#Profile likelihood|profile likelihood]]:

:<math>\bar{L}(\alpha) = \sup_{\theta: \alpha = g(\theta)} L(\theta). \, </math>

The MLE is also equivariant with respect to certain transformations of the data.  If <math>y=g(x)</math> where <math>g</math> is one to one and does not depend on the parameters to be estimated, then the density functions satisfy

:<math>f_Y(y) = f_X(g^{-1}(y)) \, |(g^{-1}(y))^{\prime}| </math>

and hence the likelihood functions for <math>X</math> and <math>Y</math> differ only by a factor that does not depend on the model parameters.

For example, the MLE parameters of the log-normal distribution are the same as those of the normal distribution fitted to the logarithm of the data. In fact, in the log-normal case if <math>X\sim\mathcal{N}(0, 1)</math>, then <math>Y=g(X)=e^{X} </math> follows a [[log-normal distribution]]. The density of Y follows with <math> f_X</math> standard [[Normal distribution|Normal]] and <math>g^{-1}(y) = \log(y) </math>, <math>|(g^{-1}(y))^{\prime}| = \frac{1}{y} </math> for <math> y > 0</math>.

=== Efficiency ===
As assumed above, if the data were generated by <math>~f(\cdot\,;\theta_0)~,</math> then under certain conditions, it can also be shown that the maximum likelihood estimator [[Convergence in distribution|converges in distribution]] to a normal distribution. It is {{sqrt|''n''&nbsp;}}-consistent and asymptotically efficient, meaning that it reaches the [[Cramér–Rao bound]]. Specifically,<ref name=":1"/>

: <math>
    \sqrt{n\,} \, \left( \widehat{\theta\,}_\text{mle} - \theta_0 \right)\ \ \xrightarrow{d}\ \ \mathcal{N} \left( 0,\ \mathcal{I}^{-1} \right) ~,
  </math>
where <math>~\mathcal{I}~</math> is the [[Fisher information matrix]]:
: <math>
    \mathcal{I}_{jk} = \operatorname{\mathbb E} \, \biggl[ \; -{ \frac{\partial^2\ln f_{\theta_0}(X_t)}{\partial\theta_j\,\partial\theta_k } } 
             \; \biggr] ~.
  </math>

In particular, it means that the [[bias of an estimator|bias]] of the maximum likelihood estimator is equal to zero up to the order {{sfrac|1|{{sqrt|{{mvar|n}}&nbsp;}}}}.

=== Second-order efficiency after correction for bias ===
However, when we consider the higher-order terms in the [[Edgeworth expansion|expansion]] of the distribution of this estimator, it turns out that {{math|''θ''<sub>mle</sub>}} has bias of order {{frac|1|{{mvar|n}}}}. This bias is equal to (componentwise)<ref>See formula&nbsp;20 in
{{cite journal
  | last1 = Cox | first1 = David R. | author-link1=David R. Cox
  | last2 = Snell | first2 = E. Joyce | author-link2 = Joyce Snell
  | title = A general definition of residuals
  | year = 1968
  | journal = [[Journal of the Royal Statistical Society, Series B]]
  | pages = 248–275
  | jstor = 2984505
  | volume=30
  | issue = 2
}}
</ref>

:<math>
    b_h \; \equiv \; \operatorname{\mathbb E} \biggl[ \; \left( \widehat\theta_\mathrm{mle} - \theta_0 \right)_h \; \biggr]
       \; = \; \frac{1}{\,n\,} \, \sum_{i, j, k = 1}^m \; \mathcal{I}^{h i} \; \mathcal{I}^{j k} \left( \frac{1}{\,2\,} \, K_{i j k} \; + \; J_{j,i k} \right)
  </math>

where <math>\mathcal{I}^{j k}</math> (with superscripts) denotes the (''j,k'')-th component of the ''inverse'' Fisher information matrix <math>\mathcal{I}^{-1}</math>, and

:<math>
     \frac{1}{\,2\,} \, K_{i j k} \; + \; J_{j,i k} \; = \; \operatorname{\mathbb E}\,\biggl[\;
             \frac12 \frac{\partial^3 \ln f_{\theta_0}(X_t)}{\partial\theta_i\;\partial\theta_j\;\partial\theta_k} +
             \frac{\;\partial\ln f_{\theta_0}(X_t)\;}{\partial\theta_j}\,\frac{\;\partial^2\ln f_{\theta_0}(X_t)\;}{\partial\theta_i \, \partial\theta_k}
             \; \biggr] ~ .
  </math>

Using these formulae it is possible to estimate the second-order bias of the maximum likelihood estimator, and ''correct'' for that bias by subtracting it:
: <math>
    \widehat{\theta\,}^*_\text{mle} = \widehat{\theta\,}_\text{mle} - \widehat{b\,} ~ .
  </math>
This estimator is unbiased up to the terms of order {{sfrac|1|&thinsp;{{mvar|n}}&thinsp;}}, and is called the '''bias-corrected maximum likelihood estimator'''.

This bias-corrected estimator is {{em|second-order efficient}} (at least within the curved exponential family), meaning that it has minimal mean squared error among all second-order bias-corrected estimators, up to the terms of the order {{sfrac|1|&thinsp;{{mvar|n}}<sup>2</sup>&thinsp;}}&nbsp;. It is possible to continue this process, that is to derive the third-order bias-correction term, and so on. However, the maximum likelihood estimator is ''not'' third-order efficient.<ref>
{{cite journal
 |last    = Kano
 |first   = Yutaka
 |title   = Third-order efficiency implies fourth-order efficiency
 |year    = 1996
 |journal = Journal of the Japan Statistical Society
 |volume  = 26  |pages = 101–117
 |doi     = 10.14490/jjss1995.26.101
 |doi-access= free
 }}
</ref>

=== Relation to Bayesian inference ===
A maximum likelihood estimator coincides with the [[Maximum a posteriori|most probable]] [[Bayesian estimator]] given a [[Uniform distribution (continuous)|uniform]] [[prior probability|prior distribution]] on the [[parameter space|parameters]]. Indeed, the [[maximum a posteriori estimate]] is the parameter {{mvar|θ}} that maximizes the probability of {{mvar|θ}} given the data, given by Bayes' theorem:

: <math>
    \operatorname{\mathbb P}(\theta\mid x_1,x_2,\ldots,x_n) = \frac{f(x_1,x_2,\ldots,x_n\mid\theta)\operatorname{\mathbb P}(\theta)}{\operatorname{\mathbb P}(x_1,x_2,\ldots,x_n)}
  </math>

where <math>\operatorname{\mathbb P}(\theta)</math> is the prior distribution for the parameter {{mvar|θ}} and where <math>\operatorname{\mathbb P}(x_1,x_2,\ldots,x_n)</math> is the probability of the data averaged over all parameters. Since the denominator is independent of {{mvar|θ}}, the Bayesian estimator is obtained by maximizing <math>f(x_1,x_2,\ldots,x_n\mid\theta)\operatorname{\mathbb P}(\theta)</math> with respect to {{mvar|θ}}. If we further assume that the prior <math>\operatorname{\mathbb P}(\theta)</math> is a uniform distribution, the Bayesian estimator is obtained by maximizing the likelihood function <math>f(x_1,x_2,\ldots,x_n\mid\theta)</math>. Thus the Bayesian estimator coincides with the maximum likelihood estimator for a uniform prior distribution <math>\operatorname{\mathbb P}(\theta)</math>.

==== Application of maximum-likelihood estimation in Bayes decision theory ====
In many practical applications in [[machine learning]], maximum-likelihood estimation is used as the model for parameter estimation.

The Bayesian Decision theory is about designing a classifier that minimizes total expected risk, especially, when the costs (the loss function) associated with different decisions are equal, the classifier is minimizing the error over the whole distribution.<ref>{{cite web |last=Christensen |first=Henrikt I. |title=Pattern Recognition |publisher=Georgia Tech |series=Bayesian Decision Theory - CS 7616 |url=https://www.cc.gatech.edu/~hic/CS7616/pdf/lecture2.pdf |type=lecture}}</ref>

Thus, the Bayes Decision Rule is stated as
:"decide <math>\;w_1\;</math> if <math>~\operatorname{\mathbb P}(w_1|x) \; > \; \operatorname{\mathbb P}(w_2|x)~;~</math> otherwise decide <math>\;w_2\;</math>"
where <math>\;w_1\,, w_2\;</math> are predictions of different classes. From a perspective of minimizing error, it can also be stated as
:<math>w = \underset{ w }{\operatorname{arg\;max}} \; \int_{-\infty}^\infty \operatorname{\mathbb P}(\text{ error}\mid x)\operatorname{\mathbb P}(x)\,\operatorname{d}x~</math>
where
:<math>\operatorname{\mathbb P}(\text{ error}\mid x) = \operatorname{\mathbb P}(w_1\mid x)~</math>
if we decide <math>\;w_2\;</math> and <math>\;\operatorname{\mathbb P}(\text{ error}\mid x) = \operatorname{\mathbb P}(w_2\mid x)\;</math> if we decide <math>\;w_1\;.</math>

By applying [[Bayes' theorem]]
:<math>\operatorname{\mathbb P}(w_i \mid x) = \frac{\operatorname{\mathbb P}(x \mid w_i) \operatorname{\mathbb P}(w_i)}{\operatorname{\mathbb P}(x)}</math>,
and if we further assume the zero-or-one loss function, which is a same loss for all errors, the Bayes Decision rule can be reformulated as:
:<math>h_\text{Bayes} = \underset{ w }{\operatorname{arg\;max}} \, \bigl[\, \operatorname{\mathbb P}(x\mid w)\,\operatorname{\mathbb P}(w) \,\bigr]\;,</math>
where <math>h_\text{Bayes}</math> is the prediction and <math>\;\operatorname{\mathbb P}(w)\;</math> is the [[prior probability]].

=== Relation to minimizing Kullback–Leibler divergence and cross entropy ===
Finding <math>\hat \theta</math> that maximizes the likelihood is asymptotically equivalent to finding the <math>\hat \theta</math> that defines a probability distribution (<math>Q_{\hat \theta}</math>) that has a minimal distance, in terms of [[Kullback–Leibler divergence]], to the real probability distribution from which our data were generated (i.e., generated by <math>P_{\theta_0}</math>).<ref>cmplx96 (https://stats.stackexchange.com/users/177679/cmplx96), Kullback–Leibler divergence, URL (version: 2017-11-18): https://stats.stackexchange.com/q/314472 (at the youtube video, look at minutes 13 to 25)</ref> In an ideal world, P and Q are the same (and the only thing unknown is <math>\theta</math> that defines P), but even if they are not and the model we use is misspecified, still the MLE will give us the "closest" distribution (within the restriction of a model Q that depends on <math>\hat \theta</math>) to the real distribution <math>P_{\theta_0}</math>.<ref>[https://web.stanford.edu/class/stats200/Lecture16.pdf Introduction to Statistical Inference | Stanford (Lecture 16 — MLE under model misspecification)]</ref>

{| role="presentation" class="wikitable mw-collapsible mw-collapsed"
| '''Proof.'''
|-
|
For simplicity of notation, let's assume that P=Q. Let there be ''n'' [[i.i.d]] data samples <math>\mathbf{y} = (y_1, y_2, \ldots, y_n)</math> from some probability <math>y \sim P_{\theta_0}</math>, that we try to estimate by finding <math>\hat \theta</math> that will maximize the likelihood using <math>P_{\theta}</math>, then:
:<math>\begin{align}
\hat \theta &= \underset{\theta}{\operatorname{arg\,max}}\, L_{P_{\theta}}(\mathbf{y}) = \underset{\theta}{\operatorname{arg\,max}}\, P_{\theta} (\mathbf{y}) = \underset{\theta}{\operatorname{arg\,max}}\, P (\mathbf{y} \mid  \theta)\\
 &= \underset{\theta}{\operatorname{arg\,max}}\, \prod_{i=1}^n P (y_i \mid \theta) = \underset{\theta}{\operatorname{arg\,max}}\, \sum_{i=1}^n \log P (y_i \mid \theta) \\
 &= \underset{\theta}{\operatorname{arg\,max}}\, \left( \sum_{i=1}^n \log P (y_i \mid \theta) - \sum_{i=1}^n \log P (y_i \mid \theta_0) \right) = \underset{\theta}{\operatorname{arg\,max}}\,  \sum_{i=1}^n \left( \log P (y_i \mid \theta) - \log P (y_i \mid \theta_0) \right) \\
 &=  \underset{\theta}{\operatorname{arg\,max}}\,  \sum_{i=1}^n \log\frac{P (y_i \mid  \theta)}{P (y_i \mid  \theta_0)} = \underset{\theta}{\operatorname{arg\,min}}\,  \sum_{i=1}^n \log \frac{P (y_i \mid  \theta_0)}{P (y_i \mid  \theta)}
  = \underset{\theta}{\operatorname{arg\,min}}\,  \frac{1}{n} \sum_{i=1}^n  \log \frac{P (y_i \mid  \theta_0)}{P (y_i \mid  \theta)} \\
 &= \underset{\theta}{\operatorname{arg\,min}}\,  \frac{1}{n} \sum_{i=1}^n  h_{\theta}(y_i)  \quad \underset{n\to\infty}{\longrightarrow} \quad \underset{\theta}{\operatorname{arg\,min}}\,  E [ h_{\theta}(y) ]  \\
 &=\underset{\theta}{\operatorname{arg\,min}}\,  \int  P_{\theta_0}(y) h_\theta(y) dy  =  \underset{\theta}{\operatorname{arg\,min}}\,  \int  P_{\theta_0}(y) \log \frac{P (y \mid  \theta_0)}{P (y \mid  \theta)}  dy\\
 &= \underset{\theta}{\operatorname{arg\,min}}\,  D_\text{KL}(P_{\theta_0} \parallel P_{\theta}) 
\end{align}</math>

Where <math>h_{\theta}(x) = \log \frac{P (x \mid \theta_0)}{P (x \mid \theta)}</math>. Using ''h'' helps see how we are using the [[law of large numbers]] to move from the average of ''h''(''x'') to the [[expectancy]] of it using the [[law of the unconscious statistician]]. The first several transitions have to do with laws of [[logarithm]] and that finding <math>\hat \theta</math> that maximizes some function will also be the one that maximizes some monotonic transformation of that function (i.e.: adding/multiplying by a constant).

Since [[Kullback–Leibler divergence#Cross entropy|cross entropy]] is just [[Entropy (information theory)|Shannon's entropy]] plus KL divergence, and since the entropy of <math>P_{\theta_0}</math> is constant, then the MLE is also asymptotically minimizing cross entropy.<ref>Sycorax says Reinstate Monica (https://stats.stackexchange.com/users/22311/sycorax-says-reinstate-monica), the relationship between maximizing the likelihood and minimizing the cross-entropy, URL (version: 2019-11-06): https://stats.stackexchange.com/q/364237</ref>
|}

== Examples ==
=== Discrete uniform distribution ===
{{Main|German tank problem}}
Consider a case where ''n'' tickets numbered from 1 to ''n'' are placed in a box and one is selected at random (''see [[Uniform distribution (discrete)|uniform distribution]]''); thus, the sample size is 1.  If ''n'' is unknown, then the maximum likelihood estimator <math>\widehat{n}</math> of ''n'' is the number ''m'' on the drawn ticket. (The likelihood is 0 for ''n''&nbsp;<&nbsp;''m'', {{frac|1|''n''}} for ''n''&nbsp;≥&nbsp;''m'', and this is greatest when ''n''&nbsp;=&nbsp;''m''. Note that the maximum likelihood estimate of ''n'' occurs at the lower extreme of possible values {''m'',&nbsp;''m''&nbsp;+&nbsp;1,&nbsp;...}, rather than somewhere in the "middle" of the range of possible values, which would result in less bias.) The [[expected value]] of the number ''m'' on the drawn ticket, and therefore the expected value of <math>\widehat{n}</math>, is (''n''&nbsp;+&nbsp;1)/2. As a result, with a sample size of 1, the maximum likelihood estimator for ''n'' will systematically underestimate ''n'' by (''n''&nbsp;−&nbsp;1)/2.

=== Discrete distribution, finite parameter space ===
Suppose one wishes to determine just how biased an [[unfair coin]] is.  Call the probability of tossing a '[[Obverse and reverse|head]]' ''p''. The goal then becomes to determine ''p''.

Suppose the coin is tossed 80 times: i.e. the sample might be something like ''x''<sub>1</sub>&nbsp;=&nbsp;H, ''x''<sub>2</sub>&nbsp;=&nbsp;T, ..., ''x''<sub>80</sub>&nbsp;=&nbsp;T, and the count of the number of [[Obverse and reverse|heads]] "H" is observed.

The probability of tossing [[Obverse and reverse|tails]] is 1&nbsp;−&nbsp;''p'' (so here ''p'' is ''θ'' above). Suppose the outcome is 49&nbsp;heads and 31&nbsp;[[Obverse and reverse|tails]], and suppose the coin was taken from a box containing three coins: one which gives heads with probability ''p''&nbsp;=&nbsp;{{frac|1|3}}, one which gives heads with probability ''p''&nbsp;=&nbsp;{{frac|1|2}} and another which gives heads with probability ''p''&nbsp;=&nbsp;{{frac|2|3}}. The coins have lost their labels, so which one it was is unknown. Using maximum likelihood estimation, the coin that has the largest likelihood can be found, given the data that were observed. By using the [[probability mass function]] of the [[binomial distribution]] with sample size equal to 80, number successes equal to 49 but for different values of ''p'' (the "probability of success"), the likelihood function (defined below) takes one of three values:

:<math>
\begin{align}
\operatorname{\mathbb P}\bigl[\;\mathrm{H} = 49 \mid p=\tfrac{1}{3}\;\bigr] & = \binom{80}{49}(\tfrac{1}{3})^{49}(1-\tfrac{1}{3})^{31} \approx 0.000, \\[6pt]
\operatorname{\mathbb P}\bigl[\;\mathrm{H} = 49 \mid p=\tfrac{1}{2}\;\bigr] & = \binom{80}{49}(\tfrac{1}{2})^{49}(1-\tfrac{1}{2})^{31} \approx 0.012, \\[6pt]
\operatorname{\mathbb P}\bigl[\;\mathrm{H} = 49 \mid p=\tfrac{2}{3}\;\bigr] & = \binom{80}{49}(\tfrac{2}{3})^{49}(1-\tfrac{2}{3})^{31} \approx 0.054~.
\end{align}
</math>

The likelihood is maximized when {{mvar|p}}&nbsp;=&nbsp;{{frac|2|3}}, and so this is the ''maximum likelihood estimate'' for&nbsp;{{mvar|p}}.

=== Discrete distribution, continuous parameter space ===
Now suppose that there was only one coin but its {{mvar|p}} could have been any value {{nowrap| 0 ≤ {{mvar|p}} ≤ 1 .}} The likelihood function to be maximised is
:<math>
L(p) = f_D(\mathrm{H} = 49 \mid p) = \binom{80}{49} p^{49}(1 - p)^{31}~,
</math>

and the maximisation is over all possible values {{nowrap|0 ≤ {{mvar|p}} ≤ 1 .}}

[[File:MLfunctionbinomial-en.svg|thumb|200px|Likelihood function for proportion value of a binomial process ({{mvar|n}}&nbsp;=&nbsp;10)]]

One way to maximize this function is by [[derivative|differentiating]] with respect to {{mvar|p}} and setting to zero:

:<math>
\begin{align}
0 & = \frac{\partial}{\partial p} \left( \binom{80}{49} p^{49}(1-p)^{31} \right)~, \\[8pt]
0 & = 49 p^{48}(1-p)^{31} - 31 p^{49}(1-p)^{30} \\[8pt]
  & = p^{48}(1-p)^{30}\left[ 49 (1-p) - 31 p \right]  \\[8pt]
  & = p^{48}(1-p)^{30}\left[ 49 - 80 p \right]~.
\end{align}
</math>

This is a product of three terms.  The first term is 0 when {{mvar|p}}&nbsp;=&nbsp;0.  The second is 0 when {{mvar|p}}&nbsp;=&nbsp;1.  The third is zero when {{mvar|p}}&nbsp;=&nbsp;{{frac|49|80}}. The solution that maximizes the likelihood is clearly {{mvar|p}}&nbsp;=&nbsp;{{frac|49|80}} (since {{mvar|p}}&nbsp;=&nbsp;0 and {{mvar|p}}&nbsp;=&nbsp;1 result in a likelihood of 0). Thus the ''maximum likelihood estimator'' for {{mvar|p}} is {{frac|49|80}}.

This result is easily generalized by substituting a letter such as {{mvar|s}} in the place of 49 to represent the observed number of 'successes' of our [[Bernoulli trial]]s, and a letter such as {{mvar|n}} in the place of 80 to represent the number of Bernoulli trials. Exactly the same calculation yields {{frac|{{mvar|s}}|{{mvar|n}}}} which is the maximum likelihood estimator for any sequence of {{mvar|n}} Bernoulli trials resulting in {{mvar|s}} 'successes'.

=== Continuous distribution, continuous parameter space ===
For the [[normal distribution]] <math>\mathcal{N}(\mu, \sigma^2)</math> which has [[probability density function]]

:<math>f(x\mid \mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}\ }
                               \exp\left(-\frac {(x-\mu)^2}{2\sigma^2} \right), </math>

the corresponding [[probability density function]] for a sample of {{mvar|n}} [[independent identically distributed]] normal random variables (the likelihood) is

:<math>f(x_1,\ldots,x_n \mid \mu,\sigma^2) = \prod_{i=1}^n f( x_i\mid  \mu, \sigma^2) = \left( \frac{1}{2\pi\sigma^2} \right)^{n/2} \exp\left( -\frac{ \sum_{i=1}^n (x_i-\mu)^2}{2\sigma^2}\right).</math>

This family of distributions has two parameters: {{math|''θ''&nbsp;{{=}} (''μ'',&nbsp;''σ'')}}; so we maximize the likelihood, <math>\mathcal{L} (\mu,\sigma^2) = f(x_1,\ldots,x_n \mid \mu, \sigma^2)</math>, over both parameters simultaneously, or if possible, individually.

Since the [[natural logarithm|logarithm]] function itself is a [[continuous function|continuous]] [[strictly increasing]] function over the [[range (statistics)|range]] of the likelihood, the values which maximize the likelihood will also maximize its logarithm (the log-likelihood itself is not necessarily strictly increasing). The log-likelihood can be written as follows:

:<math>
   \log\Bigl( \mathcal{L} (\mu,\sigma^2)\Bigr) = -\frac{\,n\,}{2} \log(2\pi\sigma^2)
   - \frac{1}{2\sigma^2} \sum_{i=1}^n (\,x_i-\mu\,)^2
</math>

(Note: the log-likelihood is closely related to [[information entropy]] and [[Fisher information]].)

We now compute the derivatives of this log-likelihood as follows.

:<math>
\begin{align}
0 & = \frac{\partial}{\partial \mu} \log\Bigl( \mathcal{L} (\mu,\sigma^2)\Bigr) =
 0 - \frac{\;-2 n(\bar{x}-\mu)\;}{2\sigma^2}.
\end{align}
</math>
where <math> \bar{x} </math> is the [[sample mean]]. This is solved by

:<math>\widehat\mu = \bar{x} = \sum^n_{i=1} \frac{\,x_i\,}{n}. </math>

This is indeed the maximum of the function, since it is the only turning point in {{mvar|μ}} and the second derivative is strictly less than zero. Its [[expected value]] is equal to the parameter {{mvar|μ}} of the given distribution,

:<math>\operatorname{\mathbb E}\bigl[\;\widehat\mu\;\bigr] = \mu, \, </math>

which means that the maximum likelihood estimator <math>\widehat\mu</math> is unbiased.

Similarly we differentiate the log-likelihood with respect to {{mvar|σ}} and equate to zero:

:<math>
\begin{align}
0 & = \frac{\partial}{\partial \sigma} \log\Bigl( \mathcal{L} (\mu,\sigma^2)\Bigr) = -\frac{\,n\,}{\sigma} 
   + \frac{1}{\sigma^3} \sum_{i=1}^{n} (\,x_i-\mu\,)^2.
\end{align}
</math>

which is solved by

:<math>\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^n(x_i-\mu)^2.</math>

Inserting the estimate <math>\mu = \widehat\mu</math> we obtain

:<math>\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^n (x_i - \bar{x})^2 = \frac{1}{n}\sum_{i=1}^n x_i^2 -\frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n x_i x_j.</math>

To calculate its expected value, it is convenient to rewrite the expression in terms of zero-mean random variables ([[statistical error]])  <math>\delta_i \equiv \mu - x_i</math>. Expressing the estimate in these variables yields

: <math>\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^n (\mu - \delta_i)^2 -\frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n (\mu - \delta_i)(\mu - \delta_j).</math>

Simplifying the expression above, utilizing the facts that <math>\operatorname{\mathbb E}\bigl[\;\delta_i\;\bigr] = 0 </math> and <math>\operatorname{E}\bigl[\;\delta_i^2\;\bigr] = \sigma^2 </math>, allows us to obtain

:<math>\operatorname{\mathbb E}\bigl[\;\widehat\sigma^2\;\bigr]= \frac{\,n-1\,}{n}\sigma^2.</math>

This means that the estimator <math>\widehat\sigma^2</math> is biased for <math>\sigma^2</math>. It can also be shown that <math>\widehat\sigma</math> is biased for <math>\sigma</math>, but that both <math>\widehat\sigma^2</math> and <math>\widehat\sigma</math> are consistent.

Formally we say that the ''maximum likelihood estimator'' for <math>\theta=(\mu,\sigma^2)</math> is

:<math>\widehat{\theta\,} = \left(\widehat{\mu},\widehat{\sigma}^2\right).</math>

In this case the MLEs could be obtained individually.  In general this may not be the case, and the MLEs would have to be obtained simultaneously.

The normal log-likelihood at its maximum takes a particularly simple form:

:<math>
   \log\Bigl( \mathcal{L}(\widehat\mu,\widehat\sigma)\Bigr) = \frac{\,-n\;\;}{2} \bigl(\,\log(2\pi\widehat\sigma^2) +1\,\bigr)
</math>

This maximum log-likelihood can be shown to be the same for more general [[least squares]], even for [[non-linear least squares]].  This is often used in determining likelihood-based approximate [[confidence interval]]s and [[confidence region]]s, which are generally more accurate than those using the asymptotic normality discussed above.

== Non-independent variables ==
It may be the case that variables are correlated, that is, not independent. Two random variables <math>y_1</math> and <math>y_2</math> are independent only if their joint probability density function is the product of the individual probability density functions, i.e.

:<math>f(y_1,y_2)=f(y_1)f(y_2)\,</math>

Suppose one constructs an order-''n'' Gaussian vector out of random variables <math>(y_1,\ldots,y_n)</math>, where each variable has means given by <math>(\mu_1, \ldots, \mu_n)</math>. Furthermore, let the [[covariance matrix]] be denoted by <math>\mathit\Sigma</math>. The joint probability density function of these ''n'' random variables then follows a [[multivariate normal distribution]] given by:

:<math>f(y_1,\ldots,y_n)=\frac{1}{(2\pi)^{n/2}\sqrt{\det(\mathit\Sigma)}} \exp\left( -\frac{1}{2} \left[y_1-\mu_1,\ldots,y_n-\mu_n\right]\mathit\Sigma^{-1}     \left[y_1-\mu_1,\ldots,y_n-\mu_n\right]^\mathrm{T} \right)</math>

In the [[Bivariate analysis|bivariate]] case, the joint probability density function is given by:

:<math> f(y_1,y_2) = \frac{1}{2\pi \sigma_{1} \sigma_2 \sqrt{1-\rho^2}} \exp\left[ -\frac{1}{2(1-\rho^2)} \left(\frac{(y_1-\mu_1)^2}{\sigma_1^2} - \frac{2\rho(y_1-\mu_1)(y_2-\mu_2)}{\sigma_1\sigma_2} + \frac{(y_2-\mu_2)^2}{\sigma_2^2}\right) \right] </math>

In this and other cases where a joint density function exists, the likelihood function is defined as above, in the section "[[Maximum likelihood#Principles|principles]]," using this density.

=== Example ===
<math>X_1,\ X_2,\ldots,\ X_m</math> are counts in cells / boxes 1 up to m; each box has a different probability (think of the boxes being bigger or smaller) and we fix the number of balls that fall to be <math>n</math>:<math>x_1+x_2+\cdots+x_m=n</math>. The probability of each box is <math>p_i</math>, with a constraint: <math>p_1+p_2+\cdots+p_m=1</math>. This is a case in which the <math>X_i</math> ''s'' are not independent, the joint probability of a vector <math>x_1,\ x_2,\ldots,x_m</math> is called the multinomial and has the form:

: <math>f(x_1,x_2,\ldots,x_m\mid p_1,p_2,\ldots,p_m)=\frac{n!}{\prod x_i!}\prod p_i^{x_i}= \binom{n}{x_1,x_2,\ldots,x_m} p_1^{x_1} p_2^{x_2} \cdots p_m^{x_m}</math>

Each box taken separately against all the other boxes is a binomial and this is an extension thereof.

The log-likelihood of this is:

: <math>\ell(p_1,p_2,\ldots,p_m)=\log n!-\sum_{i=1}^m \log x_i!+\sum_{i=1}^m x_i\log p_i</math>

The constraint has to be taken into account and use the Lagrange multipliers:

: <math>L(p_1,p_2,\ldots,p_m,\lambda)=\ell(p_1,p_2,\ldots,p_m)+\lambda\left(1-\sum_{i=1}^m p_i\right)</math>

By posing all the derivatives to be 0, the most natural estimate is derived

: <math>\hat{p}_i=\frac{x_i}{n}</math>

Maximizing log likelihood, with and without constraints, can be an unsolvable problem in closed form, then we have to use iterative procedures.

== Iterative procedures ==
Except for special cases, the likelihood equations
:<math>\frac{\partial \ell(\theta;\mathbf{y})}{\partial \theta} = 0</math>

cannot be solved explicitly for an estimator <math>\widehat{\theta} = \widehat{\theta}(\mathbf{y})</math>. Instead, they need to be solved [[Iterative method|iteratively]]: starting from an initial guess of <math>\theta</math> (say <math>\widehat{\theta}_{1}</math>), one seeks to obtain a convergent sequence <math>\left\{ \widehat{\theta}_{r} \right\}</math>. Many methods for this kind of [[optimization problem]] are available,<ref>
{{cite book
 |first=R. |last=Fletcher
 |year=1987
 |title=Practical Methods of Optimization
 |location=New York, NY
 |publisher=John Wiley & Sons
 |edition=Second
 |isbn=0-471-91547-5
 |url=https://archive.org/details/practicalmethods0000flet
 |url-access=registration
}}
</ref><ref>
{{cite book
 |first1=Jorge |last1=Nocedal |author-link=Jorge Nocedal
 |first2=Stephen J. |last2=Wright
 |year=2006
 |title=Numerical Optimization
 |location=New York, NY
 |publisher=Springer
 |edition=Second
 |isbn=0-387-30303-0
}}
</ref> but the most commonly used ones are algorithms based on an updating formula of the form
:<math>\widehat{\theta}_{r+1} = \widehat{\theta}_{r} + \eta_{r} \mathbf{d}_r\left(\widehat{\theta}\right)</math>

where the vector <math>\mathbf{d}_{r}\left(\widehat{\theta}\right)</math> indicates the [[descent direction]] of the <var>r</var>th "step," and the scalar <math>\eta_{r}</math> captures the "step length,"<ref>{{cite book |first=Carlos |last=Daganzo |title=Multinomial Probit: The Theory and its Application to Demand Forecasting |location=New York |publisher=Academic Press |year=1979 |isbn=0-12-201150-3 |pages=61–78 }}</ref><ref>{{cite book |first1=William |last1=Gould |first2=Jeffrey |last2=Pitblado |first3=Brian |last3=Poi |title=Maximum Likelihood Estimation with Stata |location=College Station |publisher=Stata Press |year=2010 |edition=Fourth |isbn=978-1-59718-078-8 |pages=13–20 }}</ref> also known as the [[learning rate]].<ref>{{cite book |first=Kevin P. |last=Murphy |title=Machine Learning: A Probabilistic Perspective |location=Cambridge |publisher=MIT Press |year=2012 |isbn=978-0-262-01802-9 |page=247 |url=https://books.google.com/books?id=NZP6AQAAQBAJ&pg=PA247 }}</ref>

=== [[Gradient descent]] method ===
(Note: here it is a maximization problem, so the sign before gradient is flipped)

:<math>\eta_r\in \R^+</math>  that is small enough for convergence and <math>\mathbf{d}_r\left(\widehat{\theta}\right) = \nabla\ell\left(\widehat{\theta}_r;\mathbf{y}\right)</math>

Gradient descent method requires to calculate the gradient at the rth iteration, but no need to calculate the inverse of second-order derivative, i.e., the Hessian matrix. Therefore, it is computationally faster than Newton-Raphson method.

=== [[Newton's method|Newton–Raphson method]] ===
:<math>\eta_r = 1</math> and <math>\mathbf{d}_r\left(\widehat{\theta}\right) = -\mathbf{H}^{-1}_r\left(\widehat{\theta}\right) \mathbf{s}_r\left(\widehat{\theta}\right)</math>

where <math>\mathbf{s}_{r}(\widehat{\theta})</math> is the [[Score (statistics)|score]] and <math>\mathbf{H}^{-1}_r \left(\widehat{\theta}\right)</math> is the [[Invertible matrix|inverse]] of the [[Hessian matrix]] of the log-likelihood function, both evaluated the <var>r</var>th iteration.<ref>{{cite book |first=Takeshi |last=Amemiya |author-link=Takeshi Amemiya |title=Advanced Econometrics |location=Cambridge |publisher=Harvard University Press |year=1985 |isbn=0-674-00560-0 |pages=[https://archive.org/details/advancedeconomet00amem/page/137 137–138] |url=https://archive.org/details/advancedeconomet00amem/page/137 }}</ref><ref>{{cite book |first=Denis |last=Sargan |author-link=Denis Sargan |chapter=Methods of Numerical Optimization |title=Lecture Notes on Advanced Econometric Theory |location=Oxford |publisher=Basil Blackwell |year=1988 |isbn=0-631-14956-2 |pages=161–169 }}</ref> But because the calculation of the Hessian matrix is [[Computational complexity|computationally costly]], numerous alternatives have been proposed. The popular [[Berndt–Hall–Hall–Hausman algorithm]] approximates the Hessian with the [[outer product]] of the expected gradient, such that

:<math>\mathbf{d}_r\left(\widehat{\theta}\right) = - \left[ \frac{1}{n} \sum_{t=1}^n \frac{\partial \ell(\theta;\mathbf{y})}{\partial \theta} \left( \frac{\partial \ell(\theta;\mathbf{y})}{\partial \theta} \right)^{\mathsf{T}} \right]^{-1} \mathbf{s}_r \left(\widehat{\theta}\right)</math>

=== [[Quasi-Newton method]]s ===
Other quasi-Newton methods use more elaborate secant updates to give approximation of Hessian matrix.

==== [[Davidon–Fletcher–Powell formula]] ====
DFP formula finds a solution that is symmetric, positive-definite and closest to the current approximate value of second-order derivative:
: <math>\mathbf{H}_{k+1} =
  \left(I - \gamma_k y_k s_k^\mathsf{T}\right) \mathbf{H}_k \left(I - \gamma_k s_k y_k^\mathsf{T}\right) + \gamma_k y_k y_k^\mathsf{T},
</math>

where

: <math>y_k = \nabla\ell(x_k + s_k) - \nabla\ell(x_k),</math>
: <math>\gamma_k = \frac{1}{y_k^T s_k},</math>
: <math>s_k = x_{k+1} - x_k.</math>

==== [[Broyden–Fletcher–Goldfarb–Shanno algorithm]] ====
BFGS also gives a solution that is symmetric and positive-definite:

: <math>B_{k+1} = B_k + \frac{y_k y_k^\mathsf{T}}{y_k^\mathsf{T} s_k} - \frac{B_k s_k s_k^\mathsf{T} B_k^\mathsf{T}}{s_k^\mathsf{T} B_k s_k}\ ,</math>

where

: <math>y_k = \nabla\ell(x_k + s_k) - \nabla\ell(x_k),</math>
: <math>s_k = x_{k+1} - x_k.</math>

BFGS method is not guaranteed to converge unless the function has a quadratic [[Taylor expansion]] near an optimum. However, BFGS can have acceptable performance even for non-smooth optimization instances

==== [[Scoring algorithm|Fisher's scoring]] ====
Another popular method is to replace the Hessian with the [[Fisher information matrix]], <math>\mathcal{I}(\theta) = \operatorname{\mathbb E}\left[\mathbf{H}_r \left(\widehat{\theta}\right)\right]</math>, giving us the Fisher scoring algorithm. This procedure is standard in the estimation of many methods, such as [[generalized linear models]].

Although popular, quasi-Newton methods may converge to a [[stationary point]] that is not necessarily a local or global maximum,<ref>See theorem 10.1 in
{{cite book
 |first=Mordecai |last=Avriel
 |year=1976
 |title=Nonlinear Programming: Analysis and Methods
 |pages=293–294
 |location=Englewood Cliffs, NJ
 |publisher=Prentice-Hall
 |isbn=978-0-486-43227-4
 |url=https://books.google.com/books?id=byF4Xb1QbvMC&pg=PA293
}}
</ref> but rather a local minimum or a [[saddle point]]. Therefore, it is important to assess the validity of the obtained solution to the likelihood equations, by verifying that the Hessian, evaluated at the solution, is both [[negative definite]] and [[well-conditioned]].<ref>
{{cite book
 |first1=Philip E. |last1=Gill
 |first2=Walter |last2=Murray
 |first3=Margaret H. |last3=Wright |author-link3=Margaret H. Wright
 |year=1981
 |title=Practical Optimization
 |location=London, UK
 |publisher=Academic Press
 |pages=[https://archive.org/details/practicaloptimiz00gill/page/n329 312]–313
 |isbn=0-12-283950-1
 |url=https://archive.org/details/practicaloptimiz00gill
 |url-access=limited
}}
</ref>

== History ==
[[File:Youngronaldfisher2.JPG|thumb|right|[[Ronald Fisher]] in 1913]]
Early users of maximum likelihood include [[Carl Friedrich Gauss]], [[Pierre-Simon Laplace]], [[Thorvald N. Thiele]], and [[Francis Ysidro Edgeworth]].<ref>
{{cite journal
  | doi = 10.2307/2339293
  | last = Edgeworth | first = Francis Y. | author-link = Francis Ysidro Edgeworth
  | title = On the probable errors of frequency-constants
  |date=Sep 1908
  | ref= CITEREFEdgeworthSeptember_1908
  | journal = Journal of the Royal Statistical Society
  | volume = 71 | issue = 3
  | pages = 499–512
  | jstor = 2339293
  | url = https://zenodo.org/record/1449468 }}</ref><ref>{{cite journal
  | doi = 10.2307/2339378
  | last = Edgeworth | first = Francis Y.
  | title = On the probable errors of frequency-constants
  |date=Dec 1908
  | ref= CITEREFEdgeworthDecember_1908
  | journal = Journal of the Royal Statistical Society
  | volume = 71 | issue = 4
  | pages = 651–678
  | jstor = 2339378
  | url = https://zenodo.org/record/1449468
}}
</ref> It was [[Ronald Fisher]] however, between 1912 and 1922, who singlehandedly created the modern version of the method.<ref name="Pfanzagl">
{{cite book
 |last1=Pfanzagl |first1=Johann
 |year=1994
 |title=Parametric Statistical Theory
 |publisher=[[Walter de Gruyter]]
 |isbn=978-3-11-013863-4
 |pages=207–208
 |doi=10.1515/9783110889765
 |mr=1291393
}}</ref><ref>{{Cite journal |last=Hald |first=Anders |date=1999 |title=On the History of Maximum Likelihood in Relation to Inverse Probability and Least Squares |url=https://www.jstor.org/stable/2676741 |journal=Statistical Science |volume=14 |issue=2 |pages=214–222 |issn=0883-4237}}</ref>

Maximum-likelihood estimation finally transcended [[heuristic]] justification in a proof published by [[Samuel S. Wilks]] in 1938, now called [[Wilks' theorem]].<ref>{{cite journal |last=Wilks |first=S.S. |year=1938 |title=The large-sample distribution of the likelihood ratio for testing composite hypotheses |journal=[[Annals of Mathematical Statistics]] |volume=9 |pages=60–62 |doi=10.1214/aoms/1177732360 |doi-access=free}}</ref> The theorem shows that the error in the logarithm of likelihood values for estimates from multiple independent observations is asymptotically [[chi-squared distribution|''χ''<sup>&thinsp;2</sup>-distributed]], which enables convenient determination of a [[confidence region]] around any estimate of the parameters. The only difficult part of Wilks' proof depends on the expected value of the [[Fisher information]] matrix, which is provided by a theorem proven by Fisher.<ref>
{{cite book
 |last=Owen |first=Art B.
 |year=2001
 |title=Empirical Likelihood
 |place=London, UK; Boca Raton, FL
 |publisher=Chapman & Hall; CRC Press
 |isbn=978-1-58488-071-4
}}
</ref> Wilks continued to improve on the generality of the theorem throughout his life, with his most general proof published in 1962.<ref>
{{cite book
 |last=Wilks |first=Samuel S.
 |year=1962
 |title=Mathematical Statistics
 |place=New York, NY
 |publisher=John Wiley & Sons
 |isbn=978-0-471-94650-2
}}
</ref>

Reviews of the development of maximum likelihood estimation have been provided by a number of authors.<ref>
{{cite journal
  | doi = 10.1214/aos/1176343456
  | last = Savage | first = Leonard J. | author-link = Leonard J. Savage
  | title = On rereading R.A. Fisher
  | year = 1976
  | journal = The Annals of Statistics
  | volume = 4 | issue = 3
  | pages = 441–500
  | jstor = 2958221
  | doi-access = free
}}
</ref><ref>
{{cite journal
  | doi = 10.1214/aos/1176343457
  | last = Pratt | first = John W.
  | title = F. Y. Edgeworth and R. A. Fisher on the efficiency of maximum likelihood estimation
  | year = 1976
  | journal = The Annals of Statistics
  | volume = 4 | issue = 3
  | pages = 501–514
  | jstor = 2958222
  | doi-access = free
}}
</ref><ref>
{{cite journal
  | doi = 10.2307/2344804
  | last = Stigler | first = Stephen M. | author-link = Stephen M. Stigler
  | title = Francis Ysidro Edgeworth, statistician
  | year = 1978
  | journal = Journal of the Royal Statistical Society, Series A
  | volume = 141 | issue = 3
  | pages = 287–322
  | jstor = 2344804
  }}
</ref><ref>
{{cite book
  | last = Stigler  | first = Stephen M.
  | title = The history of statistics: the measurement of uncertainty before 1900
  | year = 1986
  | publisher = Harvard University Press
  | isbn = 978-0-674-40340-6
  | url = https://archive.org/details/historyofstatist00stig
}}
</ref><ref>
{{cite book
  | last = Stigler | first = Stephen M.
  | title = Statistics on the table: the history of statistical concepts and methods
  | year = 1999
  | publisher = Harvard University Press
  | isbn = 978-0-674-83601-3
  }}</ref><ref>{{cite book
  | last = Hald | first = Anders | author-link=Anders Hald
  | title = A history of mathematical statistics from 1750 to 1930
  | year = 1998
  | publisher = Wiley
  | location = New York, NY
  | isbn = 978-0-471-17912-2
  }}
</ref><ref>
{{cite journal
  | last = Hald | first = Anders
  | title = On the history of maximum likelihood in relation to inverse probability and least squares
  | year = 1999
  | journal = Statistical Science
  | volume = 14 | issue = 2
  | pages =214–222
  | jstor = 2676741
  | doi=10.1214/ss/1009212248
  | url = http://projecteuclid.org/download/pdf_1/euclid.ss/1009212248
  | doi-access = free
}}
</ref><ref>
{{cite journal
  | last = Aldrich | first = John
  | title = R.A. Fisher and the making of maximum likelihood 1912–1922
  | year = 1997
  | journal = Statistical Science
  | volume = 12 | issue = 3
  | pages = 162–176
  | doi = 10.1214/ss/1030037906
  | mr = 1617519
  | doi-access = free
}}
</ref>

== See also ==
{{Portal|Mathematics}}

=== Related concepts===
* [[Akaike information criterion]]: a criterion to compare statistical models, based on MLE 
* [[Extremum estimator]]: a more general class of estimators to which MLE belongs
* [[Fisher information]]: information matrix, its relationship to covariance matrix of ML estimates
* [[Mean squared error]]: a measure of how 'good' an estimator of a distributional parameter is (be it the maximum likelihood estimator or some other estimator)
* [[RANSAC]]: a method to estimate parameters of a mathematical model given data that contains [[outliers]]
* [[Rao–Blackwell theorem]]: yields a process for finding the best possible unbiased estimator (in the sense of having minimal [[mean squared error]]); the MLE is often a good starting place for the process
* [[Likelihood-ratio test#Asymptotic distribution: Wilks' theorem|Wilks' theorem]]: provides a means of estimating the size and shape of the region of roughly equally-probable estimates for the population's parameter values, using the information from a single sample, using a [[chi-squared distribution]]

=== Other estimation methods ===
* [[Generalized method of moments]]: methods related to the likelihood equation in maximum likelihood estimation
* [[M-estimator]]: an approach used in robust statistics
* [[Maximum a posteriori]] (MAP) estimator: for a contrast in the way to calculate estimators when prior knowledge is postulated
* [[Maximum spacing estimation]]: a related method that is more robust in many situations
* [[Principle of maximum entropy|Maximum entropy estimation]]
* [[Method of moments (statistics)]]: another popular method for finding parameters of distributions
* [[Method of support]], a variation of the maximum likelihood technique
* [[Minimum-distance estimation]]
* [[Partial likelihood methods for panel data]]
* [[Quasi-maximum likelihood]] estimator: an MLE estimator that is misspecified, but still consistent
* [[Restricted maximum likelihood]]: a variation using a likelihood function calculated from a transformed set of data

== References ==
{{reflist}}

== Further reading ==
* {{cite book
 |first=J.S. |last=Cramer |author-link=Mars Cramer
 |year=1986
 |title=Econometric Applications of Maximum Likelihood Methods
 |location=New York, NY
 |publisher=Cambridge University Press
 |isbn=0-521-25317-9
 |url=https://archive.org/details/econometricappli0000cram
}}
* {{cite book
 |first=Scott R. |last=Eliason
 |year=1993
 |title=Maximum Likelihood Estimation: Logic and Practice
 |location=Newbury Park |publisher=Sage
 |isbn=0-8039-4107-2
}}	
* {{cite book
 |last=King |first=Gary |author-link=Gary King (political scientist)
 |year=1989
 |title=Unifying Political Methodology: the Likehood Theory of Statistical Inference
 |publisher=Cambridge University Press
 |isbn=0-521-36697-6
}}
* {{cite journal
 |last=Le Cam |first=Lucien |author-link=Lucien Le Cam
 |year=1990
 |title=Maximum likelihood: An Introduction
 |journal=ISI Review
 |volume=58 |issue=2 |pages=153–171
 |doi=10.2307/1403464 |jstor=1403464
}}
* {{cite book
 |first=Jan R. |last=Magnus
 |year=2017
 |chapter=Maximum Likelihood
 |title=Introduction to the Theory of Econometrics
 |pages=53–68
 |location=Amsterdam, NL
 |publisher=VU University Press
 |isbn=978-90-8659-766-6
}}
* {{cite book
 | last= Millar | first= Russell B.
 | year= 2011
 | title=Maximum Likelihood Estimation and Inference
 | publisher= Wiley
 | location=Hoboken, NJ
 | isbn=978-0-470-09482-2
}}
* {{cite book
 |first=Andrew |last=Pickles
 |year=1986
 |title=An Introduction to Likelihood Analysis
 |location=Norwich
 |publisher=W. H. Hutchins & Sons
 |isbn=0-86094-190-6
 |url-access=registration
 |url=https://archive.org/details/introductiontoli0000pick
}}
* {{cite book
 |first=Thomas A. |last=Severini
 |year=2000
 |title=Likelihood Methods in Statistics
 |location=New York, NY
 |publisher=Oxford University Press
 |isbn=0-19-850650-3
}}
* {{cite book
 |first1=Michael D. |last1=Ward |author-link=Michael D. Ward
 |first2=John S. |last2=Ahlquist
 |year=2018
 |title=Maximum Likelihood for Social Science: Strategies for Analysis
 |publisher=Cambridge University Press
 |isbn=978-1-316-63682-4
}}

== External links ==
* Tilevik, Andreas (2022). [https://www.youtube.com/watch?v=bhTIpGtWtzQ Maximum likelihood vs least squares in linear regression] (video)
* {{springer |title=Maximum-likelihood method |id=p/m063100}}
* {{cite web |url=http://statgen.iop.kcl.ac.uk/bgim/mle/sslike_1.html |title=Maximum Likelihood Estimation |first=S. |last=Purcell }} 
* {{cite web |first1=Thomas |last1=Sargent |author-link=Thomas J. Sargent |first2=John |last2=Stachurski |title=Maximum Likelihood Estimation |work=Quantitative Economics with [[Python (programming language)|Python]] |url= https://intro.quantecon.org/mle.html }}
* {{cite web |first1=Ott |last1=Toomet |first2=Arne |last2=Henningsen |title=maxLik: A package for maximum likelihood estimation in R |url=https://cran.r-project.org/package=maxLik |date=2019-05-19 }}
* {{cite web |first=Lawrence M. |last=Lesser |title='MLE' song lyrics |year=2007 |website=University of Texas |location=El Paso, TX |department=Mathematical Sciences / College of Science |url=http://www.math.utep.edu/Faculty/lesser/MLE.html |access-date=2021-03-06}}

{{Statistics|inference}}

{{DEFAULTSORT:Maximum likelihood}}
[[Category:Maximum likelihood estimation| ]]
[[Category:M-estimators]]
[[Category:Probability distribution fitting]]