Inside–outside algorithm

From Wikipedia, the free encyclopedia
Jump to: navigation, search

In computer science, the inside–outside algorithm is a way of re-estimating production probabilities in a probabilistic context-free grammar. It was introduced James K. Baker in 1979 as a generalization of the forward–backward algorithm for parameter estimation on hidden Markov models to stochastic context-free grammars. It is used to compute expectations, for example as part of the expectation–maximization algorithm (an unsupervised learning algorithm).

Inside and outside probabilities[edit]

The inside probability \beta_j(p,q) is the total probability of generating words w_p \cdots w_q, given the root nonterminal N^j and a grammar G:[1]

\beta_j(p,q) = P(w_{pq}|N^j_{pq}, G)

The outside probability \alpha_j(p,q) is the total probability of beginning with the start symbol N^1 and generating the nonterminal N^j_{pq} and all the words outside w_p \cdots w_q, given a grammar G:[1]

\alpha_j(p,q) = P(w_{1(p-1)}, N^j_{pq}, w_{(q+1)m}|G)

Computing Inside probabilities[edit]

Base Case:

\beta_j(p,p) = P(w_{p}|N^j, G)

General Case:

Suppose there is a rule  N_j \rightarrow N_r N_s in the grammar, then the probability of generating w_p \cdots w_q starting with a subtree rooted at N_j is:

\sum_{k=p}^{k=q-1} P(N_j \rightarrow N_r N_s)\beta_r(p,k) \beta_s(k+1,q)

The inside probability \beta_j(p,q) is just the sum over all such possible rules:

\beta_j(p,q) = \sum_{N_r,N_s} \sum_{k=p}^{k=q-1} P(N_j \rightarrow N_r N_s)\beta_r(p,k) \beta_s(k+1,q)

Computing Outside probabilities[edit]

Base Case:

\alpha_j(1,n) = \begin{cases}
1 & \mbox{if } j=1 \\
0 & \mbox{otherwise}

Here the start symbol is N_1.


  1. ^ a b Manning, Christopher D.; Hinrich Schütze (1999). Foundations of Statistical Natural Language Processing. Cambridge, MA, USA: MIT Press. pp. 388–402. ISBN 0-262-13360-1. 
  • J. Baker (1979): Trainable grammars for speech recognition. In J. J. Wolf and D. H. Klatt, editors, Speech communication papers presented at the 97th meeting of the Acoustical Society of America, pages 547–550, Cambridge, MA, June 1979. MIT.
  • Karim Lari, Steve J. Young (1990): The estimation of stochastic context-free grammars using the inside–outside algorithm. Computer Speech and Language, 4:35–56.
  • Karim Lari, Steve J. Young (1991): Applications of stochastic context-free grammars using the Inside–Outside algorithm. Computer Speech and Language, 5:237–257.
  • Fernando Pereira, Yves Schabes (1992): Inside–outside reestimation from partially bracketed corpora. Proceedings of the 30th annual meeting on Association for Computational Linguistics, Association for Computational Linguistics, 128–135.

External links[edit]