User:Mbuset14/Melanie's Sandbox

David E. Rumelhart

David E. Rumelhart is my topic for this assignment. I am interested in knowing more about his influence on mathematical psychology. One topic he researched was backpropogation.

Introduction

David E Rumelhart received his BA in psychology and mathematics at the University of South Dakota, 1963. He earned his Ph.D in mathematical psychology at Stanford in 1967. He taught at the University of California, San Diego, and Standford. Rumelhart believed that the gap between mathematical and computer based models of learning as well as linguistics needed to be brought together. One topic that Rumelhart was interested in, was unsupervised learning. He used the name "competitive learning" to refer to it. Competitive learning was examined by the use of computer simulation and formal analysis. Rumelhart & Zipser found that when competitive learning was applied to parallel networks mimicking neuron like elements, many learning tasks can be achieved. Jordan & Rumelhart (1992), explained how certain algorithms made two assumptions in regards to neural networking. The perceptron and LMS algorithms assumed that the only adaptive units in the network were output units and that a teacher provides desired states of all of said output units . However, once more research was done it was found that internal units adaptively recode the initial input representation of the environment. This finding eliminated the early assumption that the only output units were adaptive units in a network. Algorithms such as Boltzmann learning and backpropagation are considered training networks that use nonlinear internal units . Backpropagation strives to find the minimum error function, which is then considered to be the solution of the learning problem . Backpropagation differs from perceptrons by its use of activation function rather than step function . In addition to the first assumption being disproven, further research challenged the second as well. Unsupervised learning algorithms require no use of a “teacher”. These algorithms do their work by clustering and extracting features on the input data about statistical or topological properties .

Competitive Learning

Rumelhart Explored the problems of the processing capabilities of one level systems and the difficulty of developing learning schemed for multilayered systems.

Competitive learning is a scheme in which important features can be discovered at one level that a multilayered system can use to classify pattern sets which cannot be classified with a single level system. Rhumelhart reported that 35 years of experience has shown that getting neuron like elements to learn some easy thing is often quite straightforward, but designing system with powerful general learning properties is a difficult problem, and the competitive learning paradigm does not change this fact. Rumelhart hoped to show that competitive learning is a powerful strategy that cut down the amount of time it would take to complete difficult tasks.35-40 years ago, it was very difficult to see how anything resembling a neural network could learn at all. Prior to Hebb’s work, it was believed that some physical change must occur in a network to support learning, but it was unclear what this change could be. Hebb proposed that a reasonable and biologically plausible change would be to strengthen the connections between elements of the network only when both the pre and post-synaptic units were active simultaneously. The essential notion that the strength of connections between the units must change in response to some function of the correlated activity of the connected units still dominates learning models. Frank Rosenblatt invented a class of simple neuron like learning networks which he called perceptrons. Researchers thought that perceptrons may have actually corresponded to parts of more extended networks and biological systems; in this case. The results obtained will be directly applicable. More likely they represent extreme simplifications of the central nervous system, in which some properties are exaggerated and other suppressed. In this case successive perturbations and refinements of the system may yield a closer approximation .The essential structure that a competitive learning mechanism can discover is represented in the overlap of stimulus patterns. The "simplest stimulus population in which stimulus patterns could overlap with one another is one constructed out of dipoles-stimulus patterns consisting of exactly two active elements and the rest inactive" . If we have a total of N input units there are N(N-1)/2possible dipole stimuli. If the actual stimulus population consists of all N (N-1)/2 possibilities, there is no structure to be discovered. There are no clusters for out units to point at (unless we have one unit for each of the possible stimuli, in which case we can point a weight vector at each of the possible input stimuli). If, however we restrict the possible dipole stimuli in certain ways, then there can be meaningful groupings of the stimulus patterns that the system can find .

Interactive Activation and Competition Networks

Rumelhart worked a lot with colleague James McClelland. Together they developed an early word recognition connectionist model. The type of model is called “interactive activation and competition networks” (IAC) and is used during competitive learning for a number of applications such as speech perception, visual word recognition, and visual perception . The IAC network is comprised of three interconnected hierarchical levels. Each level is either excitatory or inhibitory. Excitatory connections make the connections more active and fast. Inhibitory connections weaken connections. The three levels are; input levels with visual features, intermediate levels where units are individual letters, and output levels where each unit is a word . At the input levels, the visual features of a word are considered and connections are made based on the physical feature of the first letter in the word. For example, if the word starts with a “M” all letters with the beginning letter “M” will be considered and this would produce an excitatory connection between neurons looking for the letter “M”. Other words that start with any other letter than “M” will not be considered and those connections will be inhibited. Depending on the second letter of the word, the same process will take place and so on and so forth until the final word is made. The exhausting of certain letters in certain words is called a process cycle. An example of the IAC network is seen in Figure 2.

Human Information Processing

Again, Rumelhart and McClelland worked on another neural network algorithm named the competitor of the logogen model. This model is a type of human information processing . Broadbent learning has proposed that memory is most likely diffuse being represented in many different brain regions. Rumelhart and McClelland created the logogen network in response to research done by Broadbent (1985) and his work done on memory. Rumelhart and McClelland believed that Broadbent had a good start but they were not completely satisfied with Broadbent’s official proposition explaining that his idea was much more mathematical and needed to be further understood by both psychologists and mathematicians in order to receive the best results. Rumelhart and McClelland recognized that Broadbent was adding onto previous research done by David Marr (1982). While Marr proposed three levels of his theory, the computational, the algorithmic, and the implementation levels, Broadbent only discussed two levels, leaving the algorithmic level absent. Rumelhart and McClelland’s proposal is mainly focused on the algorithmic level and focuses on the storing and retrieving of memory. The algorithmic level was of most importance to Rumelhart and McClelland focusing on issues such as, “efficiency, degradation of performance under noise or other adverse conditions, whether a particular problem is hard or difficult, which problems are solved quickly, which take a long time to solve, how information is represented, and so on” . They explained that these problems speak directly to psychologists and at the computational level it does not matter “whether the theory is stated as a program for a Turing machine, as a set of axioms, or as a set of rewrite rules” . What is important here and what matters is what function is being computed and now necessarily how it is being computed. Rumelhart and McClelland proposed that a proper algorithm must mimic that of neuronal functioning in the brain where more complex ideas take more time and simple ideas and memories take less time. An Example of the different levels is seen in Figure 1.

references

(Mbuset14 (talk) 01:55, 13 March 2012 (UTC))