Second-order co-occurrence pointwise mutual information

From Wikipedia, the free encyclopedia
Jump to: navigation, search

In computational linguistics, second-order co-occurrence pointwise mutual information is a semantic similarity measure. To assess the degree of association between two given words, it uses pointwise mutual information (PMI) to sort lists of important neighbor words of the two target words from a large corpus.


The PMI-IR method[clarification needed] used AltaVista's Advanced Search query syntax to calculate probabilities. Note that the "NEAR" search operator of AltaVista is an essential operator in the PMI-IR method.[citation needed] However, it is no longer in use in AltaVista; this means that, from the implementation point of view, it is not possible to use the PMI-IR method in the same form in new systems. In any case, from the algorithmic point of view, the advantage of using SOC-PMI is that it can calculate the similarity between two words that do not co-occur frequently, because they co-occur with the same neighboring words. For example, the British National Corpus (BNC) has been used as a source of frequencies and contexts.


The method considers the words that are common in both lists and aggregate their PMI values (from the opposite list) to calculate the relative semantic similarity. We define the pointwise mutual information function for only those words having ,

where tells us how many times the type appeared in the entire corpus, tells us how many times word appeared with word in a context window and is total number of tokens in the corpus. Now, for word , we define a set of words, , sorted in descending order by their PMI values with and taken the top-most words having .

The set , contains words ,

, where and

A rule of thumb is used to choose the value of . The -PMI summation function of a word is defined with respect to another word. For word with respect to word it is:

where which sums all the positive PMI values of words in the set also common to the words in the set . In other words, this function actually aggregates the positive PMI values of all the semantically close words of which are also common in 's list. should have a value greater than 1. So, the -PMI summation function for word with respect to word having and the -PMI summation function for word with respect to word having are



Finally, the semantic PMI similarity function between the two words, and , is defined as

The semantic word similarity is normalized, so that it provides a similarity score between and inclusively. The normalization of semantic similarity algorithm returns a normalized score of similarity between two words. It takes as arguments the two words, and , and a maximum value, , that is returned by the semantic similarity function, Sim(). It returns a similarity score between 0 and 1 inclusively. For example, the algorithm returns 0.986 for words cemetery and graveyard with (for SOC-PMI method).