Natural language processing: Difference between revisions

Content deleted Content added

Inline

Revision as of 22:01, 9 November 2006

Natural language processing (NLP) is a subfield of artificial intelligence and linguistics. It studies the problems of automated generation and understanding of natural human languages. Natural language generation systems convert information from computer databases into normal-sounding human language, and natural language understanding systems convert samples of human language into more formal representations that are easier for computer programs to manipulate.

Tasks and limitations

In theory natural language processing is a very attractive method of human-computer interaction. Early systems such as SHRDLU, working in restricted "blocks worlds" with restricted vocabularies, worked extremely well, leading researchers to excessive optimism which was soon lost when the systems were extended to more realistic situations with real-world ambiguity and complexity.

Natural language understanding is sometimes referred to as an AI-complete problem, because natural language recognition seems to require extensive knowledge about the outside world and the ability to manipulate it. The definition of "understanding" is one of the major problems in natural language processing.

Concrete problems

Some examples of the problems faced by natural language understanding systems:

The sentences We gave the monkeys the bananas because they were hungry and We gave the monkeys the bananas because they were over-ripe have the same surface grammatical structure. However, in one of them the word they refers to the monkeys, in the other it refers to the bananas: the sentence cannot be understood properly without knowledge of the properties and behaviour of monkeys and bananas.

A string of words may be interpreted in myriad ways. For example, the string Time flies like an arrow may be interpreted in a variety of ways:
- time moves quickly just like an arrow does;
- measure the speed of flying insects like you would measure that of an arrow - i.e. (You should) time flies like you would an arrow.;
- measure the speed of flying insects like an arrow would - i.e. Time flies in the same way that an arrow would (time them).;
- measure the speed of flying insects that are like arrows - i.e. Time those flies that are like arrows;
- a type of flying insect, "time-flies," enjoy arrows (compare Fruit flies like a banana.)

English is particularly challenging in this regard because it has little inflectional morphology to distinguish between parts of speech.

English and several other languages don't specify which word an adjective applies to. For example, in the string "pretty little girls' school".
- Does the school look little?
- Do the girls look little?
- Do the girls look pretty?
- Does the school look pretty?

Subproblems

Speech segmentation: In most spoken languages, the sounds representing successive letters blend into each other, so the conversion of the analog signal to discrete characters can be a very difficult process. Also, in natural speech there are hardly any pauses between successive words; the location of those boundaries usually must take into account grammatical and semantical constraints, as well as the context.

Text segmentation: Some written languages like Chinese, Japanese and Thai do not have signal word boundaries either, so any significant text parsing usually requires the identification of word boundaries, which is often a non-trivial task.

Word sense disambiguation: Many words have more than one meaning; we have to select the meaning which makes the most sense in context.

Syntactic ambiguity: The grammar for natural languages is ambiguous, i.e. there are often multiple possible parse trees for a given sentence. Choosing the most appropriate one usually requires semantic and contextual information. Specific problem components of syntactic ambiguity include sentence boundary disambiguation.

Imperfect or irregular input: Foreign or regional accents and vocal impediments in speech; typing or grammatical errors, OCR errors in texts.

Speech acts and plans: Sentences often don't mean what they literally say; for instance a good answer to "Can you pass the salt" is to pass the salt; in most contexts "Yes" is not a good answer, although "No" is better and "I'm afraid that I can't see it" is better yet. Or again, if a class was not offered last year, "The class was not offered last year" is a better answer to the question "How many students failed the class last year?" than "None" is.

Statistical NLP

Statistical natural language processing uses stochastic, probabilistic and statistical methods to resolve some of the difficulties discussed above, especially those which arise because longer sentences are highly ambiguous when processed with realistic grammars, yielding thousands or millions of possible analyses. Methods for disambiguation often involve the use of corpora and Markov models. The technology for statistical NLP comes mainly from machine learning and data mining, both of which are fields of artificial intelligence that involve learning from data.

The major tasks in NLP

Evaluation of natural language processing

This section is a stub and needs to be expanded

History of evaluation in NLP
Intrinsic evaluation
Extrinsic evaluation
Automatic evaluation
Manual evaluation
Shared tasks

Organizations and Conferences

Association for Computational Linguistics
Association for Machine Translation in the Americas
AFNLP - Asian Federation of Natural Language Processing Associations

External links

Resources

Research and development groups

Implementations

Cypher - Generates RDF and SeRQL representation of natural language statements and phrases
OpenNLP
DELPH-IN: integrated technology for deep language processing
LinguaStream: a generic platform for Natural Language Processing experimentation
GATE - a Java Library for Text Engineering
Mind.html
Natural Language ToolKit for Python - comprehensive tutorial
MARF: Modular Audio Recognition Framework for voice and statistical NLP processing
FreeLing: an open source suite of language analyzers
LingPipe: Java Natural Language Processing Toolkit
The wraetlic toolkit
Antelope framework for Microsoft .NET 2.0
Internet Business Logic has a lightweight approach in which meaning is assigned to a whole sentence. A sentence is either a table heading, with place holders such as this-dept or some-number, or it is defined in terms of other sentences, using a rule that is similar to a classical syllogism. The approach, while simple, is robust, in that it simply avoids many of the problems inherent in full dictioinary- and grammar-based language processing. An unusual feature is that the vocabluary is open, and so, to a large extent is the syntax.

@@ Line 141: / Line 141: @@
 * [http://www.ii.uam.es/%7eealfon/eng/research/wraetlic.html The wraetlic toolkit]
 * [http://www.proxem.com Antelope framework] for [[Microsoft_.Net|Microsoft .NET 2.0]]
+* [http://www.reengineeringllc.com Internet Business Logic] has a lightweight approach in which meaning is assigned to a whole sentence. A sentence is either a table heading, with place holders such as ''this-dept'' or ''some-number'', or it is defined in terms of other sentences, using a rule that is similar to a classical syllogism. The approach, while simple, is robust, in that it simply avoids many of the problems inherent in full dictioinary- and grammar-based language processing. An unusual feature is that the vocabluary is ''open'', and so, to a large extent is the syntax.
 [[Category:Artificial intelligence]]