User:Lminnes/sandbox

David E. Rumelhart

David E. Rumelhart was an American psychologist and a pioneer in the field of human cognition. His work within the frameworks of mathematical psychology and artificial intelligence led to the development of the back-propagation learning algorithm and the Parallel Distributed Processing Model. Rumelhart was known as a "father of connectionism" and developed models of motor control, story understanding, and letter recognition.”

File:David E. Rumelhart.jpg

An early photograph of psychologist David E. Rumelhart

Biography

Career

In 1963, Rumelhart chose to attend the University of South Dakota, where he earned a degree in psychology and mathematics. Continuing on his education, he chose to complete his Ph.D in mathematical psychology at Stanford University. Afterwards, he became a faculty member at the University of California, San Diego for 20 years. In 1987, he returned to Stanford and continued teaching as professor until he retired in 1998. A great deal of the research conducted by Rumelhart was done with his fellow research companion, James McClelland. McClelland was not only a colleague, but a friend. They wrote numerous papers and books together, and were able to build computer programs and devise algorithms that became central topics of discussion in the field. Their research on Parallel Distributed Processing, generated controversial thought and became a staple theory in cognitive science research. Rumelhart won many distinct professional awards including the MacArthur Genius Award, the Warren Medal of the Society of Experimental Psychologists, the IEEE Neural Networks Pioneer Award, and the APA Distinguished Scientific Contribution Award. He was also elected to the National Academy of Sciences.

Personal Life

David E. Rumelhart was born on June 6th, 1942, in Wessington Springs, South Dakota.His mother, Elma was a librarian, and his father, Everett, was a printer. He was the eldest of three sons and lived in a house full of constant competition where he was able to develop his strong self-reliant thinking and independence. Rumelhart was married to Marilyn Austin, but the marriage ended in divorce. With his ex-wife he has two sons, Karl and Peter, and four grandchildren. In the 1990’s, Rumelhart’s health began to decline and the symptoms of his neurodegenerative condition, Pick’s disease, became too much for him to continue to teach. Pick’s disease is a debilitating disease that strikes nerve cells in the brain. The cells progressively destruct due to excessive protein build up. It is known to be a genetic disease, which attacks the frontal and temporal lobes and causes them to slowly deteriorate. The sufferer may experience great behavioural and personality changes and overtime will experience speech impairment. The disease progressively gets worse and commonly causes death within 2-10 years. When the disease became too disabling, Rumelhart was taken in by his brother, Donald Rumelhart, and his wife, Judy Rumelhart. David Rumelhart passed away in Chelsea, Michigan on March 13th, 2011. The lasting impression he has made in the field of cognitive science will be greatly remembered and his contributions have paved the way for many to come. .

Research and Theory

Interactive Activation Model

The Interactive Activation Model was the first model of Rumelhart and McClelland. Their study An interactive activation model of context effects in letter perception: I. an account of basic findings, displayed their beliefs that perception occurred in a multilevel processing system. Their model of processing contained three levels: a visual feature level, letter level, and word level. The process of this model is interactive. It consists of both bottom-up and top-down information processing, due to contextual factors in perceptual processing. Their model of word perception suggests parallel processing, meaning that visual processing occurs at several different levels at the same time. Information flow is continuous, as opposed to the alternative view that information processing occurs through sequences of discrete steps. It is a positive feedback system that was designed to show how we account for specific aspects of word perception.

Back-propagation Learning Algorithm

Perhaps Rumelhart’s most notable contribution is known as the back-propagation learning model. This model is a profound algorithm that describes patterns and representations of how we learn regularities in language. This model was the starting point of further research in the developing fields of neural networks, investigations of cognitive science, and machine working. In Rumelhart, Durbin, Golden and Chauvin's Backpropgation: Theory, architectures, and applications, they explain how back-propagation is training based on error feedback, and is supervised by a teacher or “target”. Connections of units are distributed amongst three nodes: input, hidden and output. Input units activate hidden units, which then activate outputs units. The teacher or “target” then compares the output to a desired response. If there is a difference between the two, an error signal is sent back to the network in order for the weight of the connections to be changed and adjusted so that the difference is minimized. This cycle is repeated until the error signal drops below threshold and the network approaches a relatively ideal function.

Parallel Distributed Processing

A topic that appears continuously throughout the work of Rumelhart and which has had a large impact on the field has to do with the notion of parallel distributed processing. This theory generally postulates that the brain is able to carry out multiple levels of activity simultaneously and thus several processes can take place at the same time. This was observed in the previously discussed Interactive Activation Model. As stated in the article Parallel distributed processing: Explorations in the microstructure of cognition, there are 8 major aspects of a parallel distributed processing model:

• A set of processing units
• A state of activation
• An output function for each unit
• A pattern of connectivity among units
• A propagation rule for propagating patterns of activities through the network of connectivities
• An activation rule for combining unit inputs with the current state of that unit to produce a new level of activation for the unit
• A learning rule whereby patterns of connectivity are modified by experience
• An environment in which the system must operate

The framework of Parallel Distributed Processing suggests that information is not stored in localized structures, but rather is distributed over a collection of nodes. Learning is not explicit; instead it relies on the connections between units, and gradually changes in connection strength by experience.

Rumelhart and McClelland’s 1986 study On Learning the Past Tenses of English Verbs, proposed a model of language acquisition. They were able to train an Artificial Neural Network to learn the past tense of verbs. They conducted their study by using past tense forms of verbs that are frequently used and not frequently used, and also forms that are both regular and irregular. Through back-propagation, the inputs and outputs of many verb repetitions were compared and the weighting was modified. The network was then able to produce correct past tense forms for the training verbs, and also able to generate correct forms for unfamiliar verbs. Rumelhart and McClelland claim that the model had learned the English past tense, to a remarkable degree, as a young child would learn and acquire language. Through generalizations, our tendency to develop patterns and the evidence of U-shaped development, their network was able to mimic child language acquisition. This study provided an alternative view to the dominating perspective that children learn the past tense of verbs through explicit rules. The connectionist viewpoint suggests that there are no rules in language acquisition, and that we need not decide whether a verb is regular or irregular. Instead, a uniform procedure is applied for producing the past test of verbs.

Components of Learning

In the 1970’s, Rumelhart began to collaborate with Peter Lindsey, Donald Norman, and the LNR research group to develop a research project on memory and cognition. This work led to their book “Explorations in Cognitions”, and sparked debate between researchers. Their overall goal was to create a computer model that would be able to understand and operate effectively with linguistic information. As psychologists, they wanted the model to simulate human behaviour and concerned themselves with comparing the correlations between the two. The computer, named MEMOD, was an active structural network that was able to represent both procedural and declarative knowledge. Procedural knowledge represents knowing how to do something and can be applied to a certain task, while declarative knowledge represents knowing about something, more factual knowledge. MEMOD encoded information, converted it into network representations, and interpreted the information to direct the behaviour of the system. MEMOD can locate and retrieve information, answer questions and make inferences.

Rumelhart collaborated once again with Donald Norman, author and cognitive scientist, to study analogical processes in learning. They theorized three components of learning: accretion, tuning, and restructuring. They believed it is through accretion that we encode new terms, relevant to our pre-existing memories. Accretion allows us to add new data to our existing stored information, store and later reconstruct the original experience by “remembering” the data. Through tuning, a schema is modified to conform increasingly better to situations. Finally, it is through restructuring where new schemata are created. This occurs when the existing memory structures are not enough to adequately represent new knowledge. Accretion is the most common form of learning and requires the least amount of effort, while tuning and restructuring occurs less often and requires more time and effort. Rumelhart and Norman theorized that schemata aids learning in multiple ways, by highlighting important events and as serving as cues in order to remember past events.

Criticisms

There are many researchers and scientists that oppose the theories and views of Rumelhart. In Donald Broadbent’s A question of levels: Comment on McClelland and Rumelhart, he critiques Parallel Distributed Processing by suggesting that it is relevant only to the implementational level of description and not to the psychological computational level. Broadbent believes it is unclear whether Parallel Distributed Processing should be considered a cognitive theory. In James Hampton’s Context, categories and modality: Challenges for the Rumelhart model, he criticizes Rumelhart’s models, his use of context layers, and his ways of differentiating information.

Serial Processing

Serial memory processing, as opposed to parallel distributed processing, is based on the belief that there is an explicit order in which operations occur, with no overlapping. Meaning, the result of one action is known before another begins. A paper by Steinberg (1996) disagreed with parallel processing through his research on short-term memory search reaction times, and Snodgrass and Townsend’s Comparing Parallel and Serial Models: Theory and Implementation, questioned the limited capacity of the parallel processing system.

Connectionism vs Computationalism Debate

The computational and connectionism debate has become prevalent in the field of cognitive science. As connectionism grew and became increasingly popular, nativists such as Steven Pinker and others, believed it had become a threat to the development and continuous progression being made in the field of computationalism. Computationalism argues that the mind operates by performing purely operations, programmed on a computer or fully mathematically formulated. Computational models generally focus on mental models and rules, as opposed to connectionism that focuses on the connection strength of neurons and environmental stimuli. Computationalists model brain structures that are not relative to actual brain models, while connectionists attempt to simulate the neurological structures of the brain.

Steven Pinker went on to write counter arguments on the research of Rumelhart and McClelland. His 1988 paper on Language and connectionism: Analysis of a parallel distributed processing model of language acquisition, challenged Rumelhart and McClellands On Learning the Past Tenses of English Verbs. Pinker claimed that the model cannot learn any rules, cannot master the past tense, cannot explain differences between irregular and regular verbs and he generally discredited the connectionist view of not needing rules to account for language acquisition.

The David E. Rumelhart Prize

The Rumelhart Prize was created in honour of David Rumelhart in 2001, and awarded each year to an individual or collaborative team that has made a significant contribution to the field of human cognition. Funded by the Robert J. Glushko and Pamela Samuelson Foundation, the recipient is awarded a certificate, a citation of the awardee’s contribution, and a $100,000 monetary reward. The most recent winner, Linda Smith, is one of the world’s most leading cognitive scientists, focusing on the field of development processes in early word learning. Other past recipients include:

• Geoffrey E. Hinton 2001
• Richard M. Shiffrin 2002
• Aravind Joshi 2003
• John Anderson 2004
• Paul Smolensky 2005
• Roger Shepard 2006
• Jeffrey L. Elman 2007
• Shimon Ullman 2008
• Susan Carey 2009
• Jay McClelland 2010
• Judea Pearl 2011
• Peter Dayan 2012
• Linda B. Smith 2013

Implications for Future Research

In addition to creating the MEMOD model, Rumelhart was determined to produce artificial intelligence programs. Although, it is difficult to rapidly produce these AI structures because the program has to satisfy the goal of simulating actual human behaviour. This long term progress will be continued, and the pioneering work of Rumelhart will be resurfaced in order to develop new AI models.