Deep learning

From Wikipedia, the free encyclopedia
Jump to: navigation, search
For deep versus shallow learning in educational psychology, see Student approaches to learning

Deep learning (also called deep structural learning or hierarchical learning[1](p198)) is a set of algorithms in machine learning that attempt to model high-level abstractions in data by using model architectures composed of multiple non-linear transformations.[1][2][3][4]

Deep learning is part of a broader family of machine learning methods based on learning representations of data. An observation (e.g., an image) can be represented in many ways (e.g., a vector of intensity values, one per pixel), but some representations make it easier to learn tasks of interest (e.g., is this the image of a human face?) from examples, and research in this area attempts to define what makes better representations and how to create models to learn these representations.

Various deep learning architectures such as deep neural networks, convolutional deep neural networks, and deep belief networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, and music/audio signal recognition where they have been shown to produce state-of-the-art results on various tasks.

Alternatively, "deep learning" has been characterized as "just a buzzword for",[5] or "largely a rebranding of", neural networks.[6]



There are a number of ways that the field of deep learning has been characterized. Deep learning is a class of machine learning training algorithms that[1](pp199–200)

  • use many layers of nonlinear processing units for feature extraction and transformation. The algorithms may be supervised or unsupervised and applications include pattern recognition and statistical classification.
  • are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher level features are derived from lower level features to form a hierarchical representation.
  • are part of the broader machine learning field of learning representations of data.
  • learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts.
  • form a new field with the goal of moving toward artificial intelligence. The different levels of representation help make sense of data such as images, sounds and texts.

These definitions have in common (1) multiple layers of nonlinear processing units and (2) the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features.[1](p200) The composition of a layer of nonlinear processing units used in a deep belief algorithm depend on the problem to be solved. Layers that have been used in deep learning include hidden layers of an artificial neural network, restricted Boltzmann machines, and sets of complicated propositional formulas.[2]

Deep learning algorithms are contrasted with shallow learning algorithms by the number of parameterized transformations a signal encounters as it propagates from the input layer to the output layer, where a parameterized transformation is a processing unit that has trainable parameters, such as weights and thresholds.[4](p6) A chain of transformations from input to output is a credit assignment path (CAP). CAPs describe potentially causal connections between input and output and may vary in length. For a feedforward neural network, the depth of the CAPs, and thus the depth of the network, is the number of hidden layers plus one (the output layer is also parameterized). For recurrent neural networks, in which a signal may propagate through a layer more than once, the CAP is potentially unlimited in length. There is no universally agreed upon threshold of depth dividing shallow learning from deep learning, but most researchers in the field agree that deep learning has multiple nonlinear layers (CAP > 2) and Schmidhuber considers CAP > 10 to be very deep learning.[4](p7)

Fundamental concepts[edit]

Deep learning algorithms are based on distributed representations, a concept used in machine learning. The underlying assumption behind distributed representations is that observed data is generated by the interactions of many different factors on different levels. Deep learning adds the assumption that these factors are organized into multiple levels, corresponding to different levels of abstraction or composition. Varying numbers of layers and layer sizes can be used to provide different amounts of abstraction.[3]

Deep learning algorithms in particular exploit this idea of hierarchical explanatory factors. Different concepts are learned from other concepts, with the more abstract, higher level concepts being learned from the lower level ones. These architectures are often constructed with a greedy layer-by-layer method that models this idea. Deep learning helps to disentangle these abstractions and pick out which features are useful for learning.[3]

Many deep learning algorithms are framed as unsupervised learning problems. Because of this, these algorithms can make use of the unlabeled data that other algorithms cannot. Unlabeled data is usually more abundant than labeled data, making this a very important benefit of these algorithms. The deep belief network is an example of a deep structure that can be trained in an unsupervised manner.[3]


Deep learning architectures, specifically those built from artificial neural networks (ANN), date back at least to the Neocognitron introduced by Kunihiko Fukushima in 1980.[7] The ANNs themselves date back even further. In 1989, Yann LeCun et al. were able to apply the standard backpropagation algorithm, which had been around since 1974,[8] to a deep neural network with the purpose of recognizing handwritten ZIP codes on mail. Despite the success of applying the algorithm, the time to train the network on this dataset was approximately 3 days, making it impractical for general use.[9] Many factors contribute to the slow speed, one being due to the so-called vanishing gradient problem analyzed in 1991 by Jürgen Schmidhuber's student Sepp Hochreiter.[10][11] In combination with speed issues, ANNs fell out of favor in practical machine learning and simpler models such as support vector machines (SVMs) became the popular choice of the field in the 1990s and 2000s.

The term "deep learning" gained traction in the mid-2000s after a publication by Geoffrey Hinton and Ruslan Salakhutdinov showed how a many-layered feedforward neural network could be effectively pre-trained one layer at a time, treating each layer in turn as an unsupervised restricted Boltzmann machine, then using supervised backpropagation for fine-tuning.[12] In 1992, Schmidhuber had already implemented a very similar idea for the more general case of unsupervised deep hierarchies of recurrent neural networks, and also experimentally shown its benefits for speeding up supervised learning [13][14]

Since the resurgence of deep learning, it has become part of many state-of-the-art systems in different disciplines, particularly that of computer vision and automatic speech recognition (ASR). Results on commonly used evaluation sets such as TIMIT (ASR) and MNIST (image classification) as well as a range of large vocabulary speech recognition tasks are constantly being improved with new applications of deep learning.[15][16][17] Currently, it has been shown that deep learning architectures in the form of convolutional neural networks have been nearly best performing;[18] however, these are more widely used in computer vision than in ASR.

Advances in hardware have also been an important enabling factor for the renewed interest of deep learning. In particular, powerful graphics processing units (GPUs) are highly suited for the kind of number crunching, matrix/vector math involved in machine learning. GPUs have been shown to speed up training algorithms by orders of magnitude, bringing running times of weeks back to days.[19][20]

Deep learning in artificial neural networks[edit]

Some of the most successful deep learning methods involve artificial neural networks. Artificial neural networks are inspired by the 1959 biological model proposed by Nobel laureates David H. Hubel & Torsten Wiesel, who found two types of cells in the primary visual cortex: simple cells and complex cells. Many artificial neural networks can be viewed as cascading models[21] of cell types inspired by these biological observations.

Fukushima's Neocognitron introduced convolutional neural networks partially trained by unsupervised learning. Yann LeCun et al. (1989) applied supervised backpropagation to such architectures.[22]

With the advent of the back-propagation algorithm in the 1970s, many researchers tried to train supervised deep artificial neural networks from scratch, initially with little success. Sepp Hochreiter's diploma thesis of 1991[23][24] formally identified the reason for this failure in the "vanishing gradient problem," which not only affects many-layered feedforward networks, but also recurrent neural networks. The latter are trained by unfolding them into very deep feedforward networks, where a new layer is created for each time step of an input sequence processed by the network. As errors propagate from layer to layer, they shrink exponentially with the number of layers.

To overcome this problem, several methods were proposed. One is Jürgen Schmidhuber's multi-level hierarchy of networks (1992) pre-trained one level at a time through unsupervised learning, fine-tuned through backpropagation.[13] Here each level learns a compressed representation of the observations that is fed to the next level.

Another method is the long short term memory (LSTM) network of 1997 by Hochreiter & Schmidhuber.[25] In 2009, deep multidimensional LSTM networks demonstrated the power of deep learning with many nonlinear layers, by winning three ICDAR 2009 competitions in connected handwriting recognition, without any prior knowledge about the three different languages to be learned.[26][27]

Sven Behnke relied only on the sign of the gradient (Rprop) when training his Neural Abstraction Pyramid[28] to solve problems like image reconstruction and face localization.

Other methods also use unsupervised pre-training to structure a neural network, making it first learn generally useful feature detectors. Then the network is trained further by supervised back-propagation to classify labeled data. The deep model of Hinton et al. (2006) involves learning the distribution of a high level representation using successive layers of binary or real-valued latent variables. It uses a restricted Boltzmann machine (Smolensky, 1986[29]) to model each new layer of higher level features. Each new layer guarantees an increase on the lower-bound of the log likelihood of the data, thus improving the model, if trained properly. Once sufficiently many layers have been learned the deep architecture may be used as a generative model by reproducing the data when sampling down the model (an "ancestral pass") from the top level feature activations.[30] Hinton reports that his models are effective feature extractors over high-dimensional, structured data.[31]

The Google Brain team led by Andrew Ng and Jeff Dean created a neural network that learned to recognize higher-level concepts, such as cats, only from watching unlabeled images taken from YouTube videos.[32] [33]

Other methods rely on the sheer processing power of modern computers, in particular, GPUs. In 2010 it was shown by Dan Ciresan and colleagues[19] in Jürgen Schmidhuber's group at the Swiss AI Lab IDSIA that despite the above-mentioned "vanishing gradient problem," the superior processing power of GPUs makes plain back-propagation feasible for deep feedforward neural networks with many layers. The method outperformed all other machine learning techniques on the old, famous MNIST handwritten digits problem of Yann LeCun and colleagues at NYU.

As of 2011, the state of the art in deep learning feedforward networks alternates convolutional layers and max-pooling layers,[34][35] topped by several pure classification layers. Training is usually done without any unsupervised pre-training. Since 2011, GPU-based implementations[34] of this approach won many pattern recognition contests, including the IJCNN 2011 Traffic Sign Recognition Competition,[36] the ISBI 2012 Segmentation of neuronal structures in EM stacks challenge,[37] and others.

Such supervised deep learning methods also were the first artificial pattern recognizers to achieve human-competitive performance on certain tasks.[38]

Deep learning architectures[edit]

There are huge number of different variants of deep architectures, however, most of them are branched from some original parent architectures. It is not always possible to compare the performance of multiple architectures all together since, they are not all implemented on the same data set. It is important to mention that deep learning is a fast growing field that one may find some different architectures, variants, or algorithms every couple of weeks.

Deep neural networks[edit]

A deep neural network (DNN) is defined[2][4] to be an artificial neural network with multiple hidden layers of units between the input and output layers. Similar to shallow ANNs, DNNs can model complex non-linear relationships. The extra layers enable composition of features from lower layers, giving the potential of modeling complex data with fewer units than a similarly performing shallow network.[2] DNNs are typically designed as feedforward networks, but recent research has successfully applied the deep learning architecture to recurrent neural networks for applications such as language modeling.[39] Convolutional deep neural networks (CNNs) are used in computer vision where their success is well-documented.[40] More recently, CNNs have been applied to acoustic modeling for automatic speech recognition (ASR), where they have shown success over previous models.[41] For simplicity, a look at training DNNs is given here.

A DNN can be discriminatively trained with the standard backpropagation algorithm. The weight updates can be done via stochastic gradient descent using the following equation:

 \Delta w_{ij}(t + 1) = \Delta w_{ij}(t) + \eta\frac{\partial C}{\partial w_{ij}}

Here,  \eta is the learning rate, and  C is the cost function. The choice of the cost function depends on factors such as the learning type (supervised, unsupervised, reinforcement, etc.) and the activation function. For example, when performing supervised learning on a multiclass classification problem, common choices for the activation function and cost function are the softmax function and cross entropy function, respectively. The softmax function is defined as  p_j = \frac{\exp(x_j)}{\sum_k \exp(x_k)} where  p_j represents the class probability and  x_j and  x_k represent the total input to units  j and  k respectively. Cross entropy is defined as  C = -\sum_j d_j \log(p_j) where  d_j represents the target probability for output unit  j and  p_j is the probability output for  j after applying the activation function.[15]

Issues with deep neural networks[edit]

As with ANNs, many issues can arise with DNNs if they are naively trained. Two common issues are overfitting and computation time.

DNNs are prone to overfitting because of the added layers of abstraction, which allow them to model rare dependencies in the training data. Regularization methods such as weight decay ( \ell_2 -regularization) or sparsity ( \ell_1 -regularization) can be applied during training to help combat overfitting.[42] A more recent regularization method applied to DNNs is dropout regularization. In dropout, some number of units are randomly omitted from the hidden layers during training. This helps to break the rare dependencies that can occur in the training data [43]

Backpropagation and gradient descent have been the preferred method for training these structures due to the ease of implementation and their tendency to converge to better local optima in comparison with other training methods. However, these methods can be computationally expensive, especially when being used to train DNNs. There are many training parameters to be considered with a DNN, such as the size (number of layers and number of units per layer), the learning rate and initial weights. Sweeping through the parameter space for optimal parameters may not be feasible due to the cost in time and computational resources. Various 'tricks' such as using mini-batching (computing the gradient on several training examples at once rather than individual examples) [44] have been shown to speed up computation. The large processing throughput of GPUs has produced significant speedups in training, due to the matrix and vector computations required being well suited for GPUs.[4]

Deep belief networks[edit]

Main article: Deep belief network
A restricted Boltzmann machine (RBM) with fully connected visible and hidden units. Note there are no hidden-hidden or visible-visible connections.

A deep belief network (DBN) is a probabilistic, generative model made up of multiple layers of hidden units. It can be looked at as a composition of simple learning modules that make up each layer.[45]

A DBN can be used for generatively pre-training a DNN by using the learned weights as the initial weights. Backpropagation or other discriminative algorithms can then be applied for fine-tuning of these weights. This is particularly helpful in situations where limited training data is available, as poorly initialized weights can have significant impact on the performance of the final model. These pre-trained weights are in a region of the weight space that is closer to the optimal weights (as compared to just random initialization). This allows for both improved modeling capability and faster convergence of the fine-tuning phase.[46]

A DBN can be efficiently trained in an unsupervised, layer-by-layer manner where the layers are typically made of restricted Boltzmann machines (RBM). A description of training a DBN via RBMs is provided below. An RBM is an undirected, generative energy-based model with an input layer and single hidden layer. Connections only exist between the visible units of the input layer and the hidden units of the hidden layer; there are no visible-visible or hidden-hidden connections.

The training method for RBMs was initially proposed by Geoffrey Hinton for use with training "Product of Expert" models and is known as contrastive divergence (CD).[47] CD provides an approximation to the maximum likelihood method that would ideally be applied for learning the weights of the RBM.[44][48]

In training a single RBM, weight updates are performed with gradient ascent via the following equation:  \Delta w_{ij}(t+1) = w_{ij}(t) + \eta\frac{\partial \log(p(v))}{\partial w_{ij}} . Here, p(v) is the probability of a visible vector, which is given by p(v) = \frac{1}{Z}\sum_he^{-E(v,h)}.  Z is the partition function (used for normalizing) and E(v,h) is the energy function assigned to the state of the network. A lower energy indicates the network is in a more "desirable" configuration. The gradient \frac{\partial \log(p(v))}{\partial w_{ij}} has the simple form \langle v_ih_j\rangle_\text{data} - \langle v_ih_j\rangle_\text{model} where \langle\cdots\rangle_p represent averages with respect to distribution p. The issue arises in sampling \langle v_ih_j\rangle_\text{model} as this requires running alternating Gibbs sampling for a long time. CD replaces this step by running alternating Gibbs sampling for n steps (values of n = 1 have empirically been shown to perform well). After n steps, the data is sampled and that sample is used in place of \langle v_ih_j\rangle_\text{model}. The CD procedure works as follows:[44]

  1. Initialize the visible units to a training vector.
  2. Update the hidden units in parallel given the visible units: p(h_j = 1 \mid \textbf{V}) = \sigma(b_j + \sum_i v_iw_{ij}). \sigma represents the sigmoid function and b_j is the bias of h_j.
  3. Update the visible units in parallel given the hidden units: p(v_i = 1 \mid \textbf{H}) = \sigma(a_i + \sum_j h_jw_{ij}). a_i is the bias of v_i. This is called the "reconstruction" step.
  4. Reupdate the hidden units in parallel given the reconstructed visible units using the same equation as in step 2.
  5. Perform the weight update: \Delta w_{ij} \propto \langle v_ih_j\rangle_\text{data} - \langle v_ih_j\rangle_\text{reconstruction}.

Once an RBM is trained, another RBM can be "stacked" atop of it to create a multilayer model. Each time another RBM is stacked, the input visible layer is initialized to a training vector and values for the units in the already-trained RBM layers are assigned using the current weights and biases. The final layer of the already-trained layers is used as input to the new RBM. The new RBM is then trained with the procedure above, and then this whole process can be repeated until some desired stopping criterion is met.[2]

Despite the approximation of CD to maximum likelihood being very crude (CD has been shown to not follow the gradient of any function), empirical results have shown it to be an effective method for use with training deep architectures.[44]

Convolutional neural networks[edit]

A CNN is composed of one or more convolutional layers with fully connected layers (matching those in typical artificial neural networks) on top. It also uses tied weights and pooling layers. This architecture allows CNNs to take advantage of the 2D structure of input data. In comparison with other deep architectures, convolutional neural networks are starting to show superior results in both image and speech applications. They can also be trained with standard backpropagation. CNNs are easier to train than other regular, deep, feed-forward neural networks and have many fewer parameters to estimate, making them a highly attractive architecture to use.[49]

Convolutional Deep Belief Networks[edit]

A recent achievement in deep learning is from the use of convolutional deep belief networks (CDBN). A CDBN is very similar to normal Convolutional neural network in terms of its structure. Therefore, like CNNs they are also able to exploit the 2D structure of images combined with the advantage gained by pre-training in Deep belief network. They provide a generic structure which can be used in many image and signal processing tasks and can be trained in a way similar to that for Deep Belief Networks. Recently, many benchmark results on standard image datasets like CIFAR [50] have been obtained using CDBNs.[51]

Deep Boltzmann Machines[edit]

A Deep Boltzmann Machine (DBM) is a type of binary pairwise Markov random field (undirected probabilistic graphical models) with multiple layers of hidden random variables. It is a network of symmetrically coupled stochastic binary units. It comprises a set of visible units \boldsymbol{\nu} \in \{0,1\}^D, and a series of layers of hidden units \boldsymbol{h}^{(1)} \in \{0,1\}^{F_1}, \boldsymbol{h}^{(2)} \in \{0,1\}^{F_2}, \ldots, \boldsymbol{h}^{(L)} \in \{0,1\}^{F_L}. There is no connection between the units of the same layer (like RBM). For the DBM, we can write the probability which is assigned to vector \nu as:

p(\boldsymbol{\nu}) = \frac{1}{Z}\sum_h e^{\sum_{ij}W_{ij}^{(1)}\nu_i h_j^1 + \sum_{jl}W_{jl}^{(2)}h_j^{(1)}h_l^{(2)}+\sum_{lm}W_{lm}^{(3)}h_l^{(2)}h_m^{(3)}},

where \boldsymbol{h} = \{\boldsymbol{h}^{(1)}, \boldsymbol{h}^{(2)}, \boldsymbol{h}^{(3)} \} are the set of hidden units, and \theta = \{\boldsymbol{W}^{(1)}, \boldsymbol{W}^{(2)}, \boldsymbol{W}^{(3)} \} are the model parameters, representing visible-hidden and hidden-hidden symmetric interaction, since they are undirected links. As it is clear by setting \boldsymbol{W}^{(2)} = 0 and \boldsymbol{W}^{(3)} = 0 the network becomes the well-known restricted Boltzmann machine.[52]

The top two layers of a DBN form an undirected graph and the remaining layers form a belief net with directed, top-down connections. For a DBM, all the connections are undirected.[53]

There are several reasons which motivate us to take advantage of deep Boltzmann machine architectures. Like DBNs, they benefit from the ability of learning complex and abstract internal representations of the input in tasks such as object or speech recognition, with the use of limited number of labeled sensory data to fine-tune the representations which is built based on a large supply of unlabeled sensory input data. However, unlike DBNs and deep convolutional neural networks, they adopt the inference and training procedure in both directions, bottom-up and top-down pass, which enable the DBMs to better unveil the representations of the ambiguous and complex input structures,[54] .[55] Another important advantage of DBMs is the joint optimization of all layers using the approximate gradient of a variation lower-bound on the likelihood function which impacts greatly on the more proper learning of generative models.[53]

Since the exact maximum likelihood learning is intractable for the DBMs, we may perform the approximate maximum likelihood learning. One should note that this algorithm is rather slow, especially for those architectures with multiple layers of hidden units, where upper layers are quite remote from the visible layer.[54] There is another possibility, to use mean-field inference to estimate data-dependent expectations, incorporation with a Markov chain Monte Carlo (MCMC) based stochastic approximation technique to approximate the expected sufficient statistics of the model.[52] Again, there is the problem of random initialization of weights, since the mentioned learning procedure performance is quite poor for DBMs.[54] introduced a layer-wise pre-training algorithm to solve or modify this problem.

We can see the difference between DBNs and DBM. In DBNs, the top two layers form a restricted Boltzmann machine which is an undirected graphical model, but the lower layers form a directed generative model. A greedy layer-wise unsupervised learning algorithm was introduced in.[56]

The idea behind the pre-training algorithm is straightforward. When learning parameters of the first layer RBM, the bottom-up weights are constrained to be twice the top-down weights and tie the visible-hidden weights. Intuitively, using twice the weights when inferring the states of the hidden units h(1) compensates for the initial lack of top-down feedback. Conversely, when pre-training the last RBM in the stack, the top-down weights are constrained to be twice the bottom-up weights. For all the intermediate RBMs the weights are halved in both directions when composing them to form a DBM, they are symmetric. This trick, eliminates the double-counting problem once top-down and bottom-up inferences are subsequently combined. In this modified RBM with tied parameters, the conditional distributions over the hidden and visible states are defined as:[52]

p(\boldsymbol{\nu}) = \frac{1}{Z}\sum_h e^{\sum_{ij}W_{ij}^{(1)}\nu_i h_j^1 + \sum_{jl}W_{jl}^{(2)}h_j^{(1)}h_l^{(2)}+\sum_{lm}W_{lm}^{(3)}h_l^{(2)}h_m^{(3)}}.

This heuristic pre-training algorithm works surprisingly well in practice. However, it is solely motivated by the need to end up with a model that has symmetric weights, and does not provide any useful insights into what is happening during the pre-training stage. Furthermore, unlike the pre-training algorithm for Deep Belief Networks (DBNs), it lacks a proof that each time a layer is added to the DBM, the variation bound improves.[52]

Apart from all the advantages of DBMs discussed so far, they have a crucial disadvantage which limits the performance and functionality of this kind of architecture. The approximate inference, which is based on mean-field method, is about 25 to 50 times slower than a single bottom-up pass in DBNs. This time consuming task make the joint optimization, introduced before, quite impractical for large data sets, and also seriously restricts the use of DBMs in the tasks such as feature representations (the mean-field inference have to be performed for each new test input).[53]

Stacked (Denoising) Auto-Encoders[edit]

In the context of neural networks it is really common to believe that with several layers of non-linear processing one might be able to model even a complex model efficiently, and to generalize the performance on difficult recognition tasks.[57][58][59][60] This belief was inspired from both theoretical point of view such that in,[61][62] and also from the discoveries related to the biological models of human brain such as visual cortex.[55][63] The non-convex characteristics of the optimization in MLPs had limited the scientists and engineers, for a long time, to apply more than two layers of hidden units.[63][64] Consequently, the researchers had been carried out in shallow architectures so as to conceal the problem of optimization and cope with convex functions.

The auto encoder idea is motivated by the concept of good representation, in other words it states that not all the representations or features of the input are suitable to perform a specific task such as classification. Therefore, there is a need to have a clear understanding of what is a good representation and how we can distinguish it. For instance for the case of classifier it is possible to define that A good representation is one that will yield a better performing classifier. Despite the fact that whether this philosophy might be true, we may think of it as a pre-training stage with a defined criterion.

According to the definition introduced in,[60] encoder is referred to a deterministic mapping f_\theta that transforms an input vector x into hidden representation y, where \theta = \{\boldsymbol{W}, b\}, \boldsymbol{W} is the weight matrix and b is an offset vector (bias). On the contrary a decoder maps back the hidden representation y to the reconstructed input z via g_\theta. The whole process of auto encoding is to compare this reconstructed input to the original, apparently it needs an error criterion, and try to minimize this error to make the reconstructed value as close as possible to the original.

Here in the case of denoising auto encoders, we also focus on exploiting good representation. In this context we are seeking carefully the features which are useful for our specific task, and the rest (corrupting features) are not useful for the task at hand, so are to be denoised (cleaned). There are different strategies to distinguish and choose the good representations (features), such as restricting the representation by means of traditional bottleneck or sparse representation, maximizing the mutual information. However, in stacked denoising auto encoders, the partially corrupted output is cleaned (denoised). This fact has been introduced in [60] with a specific approach to good representation, a good representation is one that can be obtained robustly from a corrupted input and that will be useful for recovering the corresponding clean input. Implicit in this definition are the ideas of

  • The higher level representations are relatively stable and robust to the corruption of the input;
  • It is required to extract features that are useful for representation of the input distribution.

The algorithm is performed by small changes in the basic auto encoders described above. It consists of multiple steps, starts by a stochastic mapping of \boldsymbol{x} to \tilde{\boldsymbol{x}} through q_D(\tilde{\boldsymbol{x}}|\boldsymbol{x}), this is the corrupting step. Then the corrupted input \tilde{\boldsymbol{x}} passes through a basic auto encoder process and is mapped to a hidden representation \boldsymbol{y} = f_\theta(\tilde{\boldsymbol{x}}) = s(\boldsymbol{W}\tilde{\boldsymbol{x}}+b). From this hidden representation we can reconstruct \boldsymbol{z} = g_\theta(\boldsymbol{y}). In last stage a minimization algorithm is done in order to have a z as close as possible to uncorrupted input \boldsymbol{x}, the only difference with the conventional auto encoder is that z is a deterministic function of corrupted input rather than uncorrupted input. The reconstruction error LH(x, z) might be either the cross-entropy loss with an affine-sigmoid decoder, or the squared error loss with an affine decoder. Note that in this architecture parameters initialized randomly and adjusted using stochastic gradient descent algorithm.[60]

In order to make a deep architecture out of this denoising auto encoders, we have to stack them one on top of another, similar to what has been mentioned for the RBMs in DBNs or what is done in,[64][65][66] for conventional auto encoders. Once the encoding function f_\theta of the first denoising auto encoder is learned and used to uncorrupt the input (corrupted input), we can train the second level of stacked auto encoder.[60]

Once the stacked auto encoder is trained, its output might be used as the input to a supervised learning algorithm such as support vector machine classifier or a multiclass logistic regression.[60]

Deep Stacking Networks[edit]

One of the deep architectures recently introduced in,[67] which is based on building hierarchies with blocks of simplified neural network modules, is called deep convex network. They are called convex because of the formulation of the weights learning problem, which is a convex optimization problem with a closed-form solution, while the lower layer weights are initialized with a fixed RBM. However, the network was renamed the deep stacking network (DSN),[68] emphasizing on this fact that a similar mechanism as the stacked generalization is used.[69] Along with this new name, there is a change in the learning process, for better performance in tasks such as classification, the lower-layer weights are also learned, which causes the overall learning problem is not a convex problem any longer.[70]

The DSN blocks, each consisting of a simple, easy-to-learn module, are stacked to form the overall deep network. It can be trained block-wise in a supervised fashion without the need for back-propagation for the entire blocks.[63]

As designed in [67] each block consists of a simplified MLP with a single hidden layer. It comprises a weight matrix U as the connection between the logistic sigmoidal units of the hidden layer h to the linear output layer y, and a weight matrix W which connects each input of the blocks to their respective hidden layers. If we assume that the target vectors t be arranged to form the columns of T (the target matrix), let the input data vectors x be arranged to form the columns of X, let \boldsymbol{H} = \sigma(\boldsymbol{W}^T\boldsymbol{X}) denote the matrix of hidden units, and assume the lower-layer weights W are known (training layer-by-layer). The function performs the element-wise logistic sigmoid operation. Then learning the upper-layer weight matrix U can be formulated as a convex optimization problem:

\min_{U^T} f = ||\boldsymbol{U}^T \boldsymbol{H} - \boldsymbol{T}||^2_F,

which has a closed-form solution. The input to the first block X only contains the original data, however in the upper blocks in addition to this original (raw) data there is a copy of the lower-block(s) output y. It is possible to optimize the weights of the layer-layer (input-hidden links) using an accelerated gradient descent, this is done by minimizing the squared error objective. In this step we assume that upper-layers weights U are known (the optimal solution from the previous step) and try to update the W using iterative gradient algorithm, then again U shall be computed with new values.[71]

What is done in this method is in each block an estimate of the same final label class y is produced, then this estimated label concatenated with original input to form the expanded input for the upper block. Note that here in this architecture in contrast with other deep architectures, such as DBNs, the goal is not to discover the transformed feature representation. Regarding the structure of the hierarchy of this kind of architecture, it makes the parallel training possible. In purely discriminative tasks DSN performance is better than the conventional DBN.[68] Note that only in the final block the output y is used for the classification tasks, in other blocks the output is only used to form the expanded input for the upper blocks.

Tensor Deep Stacking Networks (T-DSN)[edit]

The architecture discussed here is an extension of the DSN. It is called tensor deep stacking network (TDSN). It improves the DSN in two important ways, using the higher order information by means of covariance statistics and also transforming the non-convex problem of the lower-layer to a convex sub-problem of the upper-layer.[70]

Unlike the DSN, the covariance statistics of the data is employed using a bilinear mapping from two distinct sets of hidden units in the same layer to predictions via a third-order tensor. Looking to the covariance structure of the data was also proposed in the works on mean-covariance RBM (mcRBM) architecture, however, there they use it on the raw data rather than on binary hidden feature layers as is done in TDSN,[72][73][74] In the mcRBM, the higher-order structure is represented in the visible data, while in the TDSN it is the hidden units which are responsible for this representation. Due to the learning complexity of the mcRBM models which are caused by the factorization, required for the reduction of the cubic growth in the size of the weight parameters, it is also very difficult to use mcRBM in deeper layers, usually it is used only in the bottom layer of a deep architecture. These difficulties of the factorization in addition to the high costs of the Hybrid Monte Carlo in learning are the very limiting factors in mcRBM to scale up to very large data sets. However, they are all removed in TDSN, and due to the specific architecture of TDSN the parallel training and closed-form solution for the upper-layer convex problems are straightforward. TDSN adopts small sizes of hidden layers so as to eliminate the factorization process. There are other differences between mcRBM and TDSN. The mcRBM is a generative model optimizing a maximum likelihood objective, while the TDSN is a discriminative model optimizing a least squares objective. In,[75] more advantages of the TDSN over other architectures are discussed.

The scalability and parallelization are the two important factors in the learning algorithms which are not considered seriously in the conventional DNNs,[76][77][78] In,[67][68] is noted that all the learning process for the DSN (and TDSN as well) is done on a batch-mode basis, so as to make the parallelization possible on a cluster of CPU and/or GPU nodes. Parallelization gives us the opportunity to scale up our design to larger (deeper) architectures and data sets, in a way different than what is done in [79] for deep sparse auto encoders.

It is important to note that the basic architecture is suitable for tasks such as classification, or regression. However, if it is used as a part of a hybrid architecture with HMM, a softmax layer to produce posterior probabilities is desirable.[70]

Spike-and-Slab RBMs (ssRBMs)[edit]

The need for real-valued inputs which are employed in Gaussian RBMs (GRBMs), motivates scientists seeking new methods. One of these methods is the spike and slab RBM (ssRBM), which models continuous-valued inputs with strictly binary latent variables.[80]

Similar to basic RBMs and its variants, the spike and slab RBM is a bipartite graph. Like GRBM, the visible units (input) are real-valued. The difference arises in the hidden layer, where each hidden unit come along with a binary spike variable and real-valued slab variable. These terms (spike and slab) come from the statistics literature,[81] and refer to a prior including a mixture of two components. One is a discrete probability mass at zero called spike, and the other is a density over continuous domain.[82][82]

There is also an extension of the ssRBM model, which is called µ-ssRBM. This variant provides extra modeling capacity to the architecture using additional terms in the energy function. One of these terms enable model to form a conditional distribution of the spike variables by means of marginalizing out the slab variables given an observation, which is quite similar to the conditional of mcRBM method [74] and also mPoT model.[83] The µ-ssRBM slab variables and input are jointly Gaussian with diagonal covariance matrix, given both observed and spike variables. The observations are Gaussian with diagonal covariance, given both the spike and slab variables. The µ-ssRBM is related to Gibbs sampling. These properties make this model as a good choice for the building blocks of deep structures such as DBM. However, there is no guarantee that the resulting model produce a valid density over the whole real-valued data space, this is one of the main shortcomings of ssRBM architectures.[82]

Compound Hierarchical-Deep Models[edit]

For a human brain in comparison with the current state-of-the-art artificial systems, fewer number of examples is needed to categorize and even extend the already existing categories for the novel instances (generalization).[84][85][86][87] This is the main motivation of this subsection, by means of learning abstract knowledge of the data and use them for novel cases in the future.[88]

The class of these new architectures is called compound HD models, where HD stands for Hierarchical-Deep. They are structured as a composition of non-parametric Bayesian models with deep networks. The features, learned by deep architectures such as DBNs,[56] DBMs,[54] deep auto encoders,[66] convolutional variants,[89][90] ssRBMs,[82] deep coding network,[91] DBNs with sparse feature learning,[92] recursive neural networks,[93] conditional DBNs,[94] denoising auto encoders,[95] are able to provide better representation for more rapid and accurate classification tasks with high-dimensional training data sets. However, they are not quite powerful in learning novel classes with few examples, themselves. In these architectures, all units through the network are involved in the representation of the input (distributed representations), and they have to be adjusted together (high degree of freedom). However, if we limit the degree of freedom, we make it easier for the model to learn new classes out of few training samples (less parameters to learn). Hierarchical Bayesian (HB) models, provide us learning from few examples as you may find in,[84][87][96][97][98] for computer vision, statistics, and cognitive science. These HB models are based on categorization of the training examples, and the generalization to the new classes at hands. However, they are all fed by hand-crafted features,[97][99] such as GiST, or SIFT features in computer vision, and MFCC features in speech perception domains. Another shortcoming of HB models is the fixed architecture it employs,[100][101] it does not discover the representation and the links between different parameters in an unsupervised fashion.[88]

There are several methods addressing the subject of learning with few examples, such as using several boosted detectors in a multi-task settings in,[102] crossgeneralization approach in,[103] which are both discriminative, boosting methods in,[104] and also HB approach in.[97]

Compound HD architectures try to integrate both characteristics of HB and deep networks. Here we introduce the compound HDP-DBM architecture, a hierarchical Dirichlet process (HDP) as a hierarchical model, incorporated with DBM architecture. It is a full generative model, generalized from abstract concepts flowing through the layers of the model, which is able to synthesize new examples in novel classes that look reasonably natural. Note that all the levels are learned jointly by maximizing a joint log-probability score.[88]

Consider a DBM with three hidden layers, the probability of a visible input \boldsymbol{\nu} is:

p(\boldsymbol{\nu}, \psi) = \frac{1}{Z}\sum_h e^{\sum_{ij}W_{ij}^{(1)}\nu_i h_j^1 + \sum_{jl}W_{jl}^{(2)}h_j^{1}h_l^{2}+\sum_{lm}W_{lm}^{(3)}h_l^{2}h_m^{3}},

where \boldsymbol{h} = \{\boldsymbol{h}^{(1)}, \boldsymbol{h}^{(2)}, \boldsymbol{h}^{(3)} \} are the set of hidden units, and \psi = \{\boldsymbol{W}^{(1)}, \boldsymbol{W}^{(2)}, \boldsymbol{W}^{(3)} \} are the model parameters, representing visible-hidden and hidden-hidden symmetric interaction terms.

After a DBM model has been learned, we have an undirected model that defines the joint distribution P(\nu, h^1, h^2, h^3). One way to express what has been learned is the conditional model P(\nu, h^1, h^2|h^3) and a prior term P(h^3). We can therefore rewrite the variation bound as:

\log P(\nu) \geq \sum_{h^1, h^2, h^3} Q(h|\nu; \mu) \log P(\nu, h^1, h^2|h^3) + \mathcal{H}(Q) + \sum_{h^3}Q(h^3|\nu;\mu)\log P(h^3)

This particular decomposition lies at the core of the greedy recursive pre-training algorithm: we keep the learned conditional model P(\nu, h^1, h^2|h^3), but maximize the variational lower-bound with respect to the last term.[56] Instead of adding an additional undirected layer, (e. g. a restricted Boltzmann machine), to model P(h^3), we can place a hierarchical Dirichlet process prior over h^3, that will allow us to learn category hierarchies, and more importantly, useful representations of classes that contain few training examples. The part we keep, P(\nu, h^1, h^2|h^3), represents a conditional DBM model, which can be viewed as a two-layer DBM but with bias terms given by the states of h^3:

P(\nu, h^1, h^2|h^3) = \frac{1}{Z(\psi, h^3)}e^{\sum_{ij}W_{ij}^{(1)}\nu_i h_j^1 + \sum_{jl}W_{jl}^{(2)}h_j^{1}h_l^{2}+\sum_{lm}W_{lm}^{(3)}h_l^{2}h_m^{3}}.

Deep Coding Networks[edit]

There are several advantages to have a model which can actively update itself to the context in data. One of these methods arises from this idea to have a model which is able to adjust its prior knowledge dynamically according to the context of the data. Deep coding network (DPCN) is a predictive coding scheme where top-down information are used to empirically adjust the priors needed for the bottom-up inference procedure by means of a deep locally-connected generative model. This is based on extracting sparse features out of time-varying observations using a linear dynamical model. Then with an extension to this feature extraction block, a pooling strategy is employed in order to learn invariant feature representations. Similar to other deep architectures, these blocks are the building elements of a deeper architecture where greedy layer-wise unsupervised learning are used. Note that the layers constitute a kind of Markov chain such that the states at any layer are only dependent on the succeeding and preceding layers. Since, it predicts the representation of the layer, by means of a top-down approach using the information in upper layer and also temporal dependencies from the previous states, it is called deep predictive coding network (DPCN).[105]

In,[106][107] a statistical model is also used to explain the cortical functions in the mammalian brain. Those models are in a close relation with the one introduced here, DPCN.[106] uses a kind of update procedure like Kalman filter for inference and a general framework consisting of all higher-order moments is used in.[107] The important problem with these methods is the lack of discriminative representation (sparse and invariant representation), helpful for task such as object recognition. However, in DPCN, and efficient inference algorithm is employed to extract locally invariant representation of the image sequences and more abstract information in higher layers. It is also possible to extend the DPCN to form a convolutional network.[105]

Deep Kernel Machines[edit]

Artificial neural network is not the only area conquered by deep concept. The Multilayer Kernel Machine (MKM) as introduced in [108] is one of those deep architectures which are not in the field of neural network, albeit, quite relevant. It is a way of learning highly nonlinear functions with the iterative applications of weakly nonlinear kernels. They use the kernel principle component analysis (KPCA), in,[109] as method for unsupervised greedy layer-wise pre-training step of the deep learning architecture.

In this method, layer l+1-th learns the representation of the previous layer l, extracting the n_l principle component (PC) of the projection layer l output in the feature domain induced by the kernel. For the sake of dimensionality reduction of the updated representation in each layer, a supervised strategy is proposed to select the best informative features among the ones extracted by KPCA. We can numerate this process as follows:

  • ranking the n_l features according to their mutual information with the class labels;
  • for different values of K and m_l \in\{1, \ldots, n_l\}, compute the classification error rate of a K-nearest neighbor (K-NN) classifier using only the m_l most informative features on a validation set;
  • the value of m_l with which the classifier has reached the lowest error rate determines the number of features to retain.

There are some drawbacks in using the KPCA method as the building cells of an MKM. It is a time consuming task, as the cross validation stage of the feature selection process needs quite long time. To circumvent this shortcoming, the use of a more efficient kernel method is used in,[110] called the kernel partial least squares (KPLS). This new approach remove the cross validation stage, and merge the selection process into the projection strategy.[111] The features are selected in an iterative supervised fashion, where the KPLS selects the j-th feature that most correlated with the class labels, solving an updated eigen-problem, at each iteration j. In this method, the eigenvalue \lambda_j of the extracted feature indicates the discriminative importance of this feature. The number of iterations to be done is equal to the number of features to extract by KPLS, and is determined by a thresholding of \lambda_j.[108]


Automatic speech recognition[edit]

The results shown in the table below are for automatic speech recognition on the popular TIMIT data set. This is a common data set used for initial evaluations of deep learning architectures. The entire set contains 630 speakers from eight major dialects of American English, with each speaker reading 10 different sentences.[112] Its small size allows many different configurations to be tried effectively with it. The error rates presented are phone error rates (PER).

Method PER (%)
Randomly Initialized RNN 26.1
Bayesian Triphone GMM-HMM 25.6
Hidden Trajectory (Generative) Model 24.8
Monophone Randomly Initialized DNN 23.4
Monophone DBN-DNN 22.4
Triphone GMM-HMM with BMMI Training 21.7
Monophone DBN-DNN on fbank 20.7
Convolutional DNN 20.0
Convolutional DNN w. Heterogeneous Pooling 18.7
Ensemble DNN/CNN/RNN [113] 18.2
Bidirectional LSTM 17.9

Extension of the success of deep learning from TIMIT to large vocabulary speech recognition occurred in 2010 by industrial researchers, where large output layers of the DNN based on context dependent HMM states constructed by decision trees were adopted. See comprehensive reviews of this development and of the state of the art as of October 2014 in the recent Springer book from Microsoft Research.[114] See also the related background of automatic speech recognition and the impact of various machine learning paradigms including notably deep learning in a recent overview article.[115]

Image classification[edit]

A common evaluation set for image classification is the MNIST database data set. MNIST is composed of handwritten digits and includes 60000 training examples and 10000 test examples. Similar to TIMIT, its small size allows multiple configurations to be tested. A comprehensive list of results on this set can be found in.[116] The current best result on MNIST is an error rate of 0.23%, achieved by Ciresan et al. in 2012.[117]

Natural language processing[edit]

Neural networks have been used for implementing language models since the early 2000s.[118] Key techniques in this field are negative sampling[119] and word embedding. A word embedding, such as word2vec, can be thought of as a representational layer in a deep learning architecture transforming an atomic word into a positional representation of the word relative to other words in the dataset; the position is represented as a point in a vector space. Using a word embedding as an input layer to a recursive neural network (RNN) allows for the training of the network to parse sentences and phrases using an effective compositional vector grammar. A compositional vector grammar can be thought of as probabilistic context free grammar (PCFG) implemented by a recursive neural network.[120] Recursive autoencoders built atop word embeddings have been trained to assess sentence similarity and detect paraphrasing.[120] Deep neural architectures have achieved state-of-the-art results in many tasks in natural language processing, such as constituency parsing[121] and sentiment analysis.[122]

Deep learning in the human brain[edit]

Computational deep learning is closely related to a class of theories of brain development (specifically, neocortical development) proposed by cognitive neuroscientists in the early 1990s.[123] An approachable summary of this work is Elman, et al.'s 1996 book "Rethinking Innateness" [124] (see also: Shrager and Johnson;[125] Quartz and Sejnowski [126]). As these theories were also instantiated in computational models, they are technical predecessors of purely computationally-motivated deep learning models. These models share the interesting property that various proposed learning dynamics in the brain (e.g., a wave of nerve growth factor) conspire to support the self-organization of just the sort of inter-related neural networks utilized in the later, purely computational deep learning models, and which appear to be analogous to one way of understanding the neocortex of the brain as a hierarchy of filters where each layer captures some of the information in the operating environment, and then passes the remainder, as well as modified base signal, to other layers further up the hierarchy. The result of this process is a self-organizing stack of transducers, well-tuned to their operating environment. As described in The New York Times in 1995: "...the infant's brain seems to organize itself under the influence of waves of so-called trophic-factors ... different regions of the brain become connected sequentially, with one layer of tissue maturing before another and so on until the whole brain is mature." [127]

The importance of deep learning with respect to the evolution and development of human cognition did not escape the attention of these researchers. One aspect of human development that distinguishes us from our nearest primate neighbors may be changes in the timing of development.[128] Among primates, the human brain remains relatively plastic until late in the post-natal period, whereas the brains of our closest relatives are more completely formed by birth. Thus, humans have greater access to the complex experiences afforded by being out in the world during the most formative period of brain development. This may enable us to "tune in" to rapidly changing features of the environment that other animals, more constrained by evolutionary structuring of their brains, are unable to take account of. To the extent that these changes are reflected in similar timing changes in hypothesized wave of cortical development, they may also lead to changes in the extraction of information from the stimulus environment during the early self-organization of the brain. Of course, along with this flexibility comes an extended period of immaturity, during which we are dependent upon our caretakers and our community for both support and training. The theory of deep learning therefore sees the coevolution of culture and cognition as a fundamental condition of human evolution.[129]

Publicity around deep learning[edit]

Deep learning is often presented as a step towards realising strong AI[130] and thus many organizations have become interested in its use for particular applications. Most recently, in December 2013, Facebook announced that it hired Yann LeCun to head its new artificial intelligence (AI) lab that will have operations in California, London, and New York. The AI lab will be used for developing deep learning techniques that will help Facebook do tasks such as automatically tagging uploaded pictures with the names of the people in them.[131]

In March 2013, Geoffrey Hinton and two of his graduate students, Alex Krizhevsky and Ilya Sutskever, were hired by Google. Their work will be focused on both improving existing machine learning products at Google and also help deal with the growing amount of data Google has. Google also purchased Hinton's company, DNNresearch.


A main criticism of deep learning concerns the lack of theory surrounding many of the methods. Most of the learning in deep architectures is just some form of gradient descent. While gradient descent has been understood for a while now, the theory surrounding other algorithms, such as contrastive divergence is less clear (i.e., Does it converge? If so, how fast? What is it approximating?). Deep learning methods are often looked at as a black box, with most confirmations done empirically, rather than theoretically.

Others point out that deep learning should be looked at as a step towards realizing strong AI, not as an all-encompassing solution. Despite the power of deep learning methods, they still lack much of the functionality needed for realizing this goal entirely. Research psychologist Gary Marcus has noted that:

"Realistically, deep learning is only part of the larger challenge of building intelligent machines. Such techniques lack ways of representing causal relationships (...) have no obvious ways of performing logical inferences, and they are also still a long way from integrating abstract knowledge, such as information about what objects are, what they are for, and how they are typically used. The most powerful A.I. systems, like Watson (...) use techniques like deep learning as just one element in a very complicated ensemble of techniques, ranging from the statistical technique of Bayesian inference to deductive reasoning." [132]

Deep learning software libraries[edit]

See also[edit]


  1. ^ a b c d L. Deng and D. Yu, "Deep Learning: Methods and Applications"
  2. ^ a b c d e Bengio, Yoshua (2009). "Learning Deep Architectures for AI". Foundations and Trends in Machine Learning 2 (1). 
  3. ^ a b c d Y. Bengio, A. Courville, and P. Vincent., "Representation Learning: A Review and New Perspectives," IEEE Trans. PAMI, special issue Learning Deep Architectures, 2013
  4. ^ a b c d e J. Schmidhuber, "Deep Learning in Neural Networks: An Overview", 2014
  5. ^ Ronan Collobert (May 6, 2011). "Deep Learning for Efficient Discriminative Parsing".  Ca. 7:45.
  6. ^ Gomes, Lee (20 October 2014). "Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Other Huge Engineering Efforts". IEEE Spectrum. 
  7. ^ K. Fukushima., "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position," Biol. Cybern., 36, 193–202, 1980
  8. ^ P. Werbos., "Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences," PhD thesis, Harvard University, 1974.
  9. ^ LeCun et al., "Backpropagation Applied to Handwritten Zip Code Recognition," Neural Computation, 1, pp. 541–551, 1989.
  10. ^ S. Hochreiter., "Untersuchungen zu dynamischen neuronalen Netzen," Diploma thesis. Institut f. Informatik, Technische Univ. Munich. Advisor: J. Schmidhuber, 1991.
  11. ^ S. Hochreiter et al., "Gradient flow in recurrent nets: the difficulty of learning long-term dependencies," In S. C. Kremer and J. F. Kolen, editors, A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001.
  12. ^ G. E. Hinton., "Learning multiple layers of representation," Trends in Cognitive Sciences, 11, pp. 428–434, 2007.
  13. ^ a b J. Schmidhuber., "Learning complex, extended sequences using the principle of history compression," Neural Computation, 4, pp. 234–242, 1992.
  14. ^ J. Schmidhuber., "My First Deep Learning System of 1991 + Deep Learning Timeline 1962–2013."
  15. ^ a b G. E. Hinton et al., "Deep Neural Networks for Acoustic Modeling in Speech Recognition: The shared views of four research groups," IEEE Signal Processing Magazine, pp. 82–97, November 2012.
  16. ^
  17. ^ L. Deng et al. Recent Advances in Deep Learning for Speech Research at Microsoft, ICASSP, 2013.
  18. ^ L. Deng, O. Abdel-Hamid, and D. Yu, A deep convolutional neural network using heterogeneous pooling for trading acoustic invariance with phonetic confusion, ICASSP, 2013.
  19. ^ a b D. C. Ciresan et al., "Deep Big Simple Neural Nets for Handwritten Digit Recognition," Neural Computation, 22, pp. 3207–3220, 2010.
  20. ^ R. Raina, A. Madhavan, A. Ng., "Large-scale Deep Unsupervised Learning using Graphics Processors," Proc. 26th Int. Conf. on Machine Learning, 2009.
  21. ^ M Riesenhuber, T Poggio. Hierarchical models of object recognition in cortex. Nature neuroscience, 1999(11) 1019–1025.
  22. ^ Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation, 1(4):541–551, 1989.
  23. ^ S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut f. Informatik, Technische Univ. Munich, 1991. Advisor: J. Schmidhuber
  24. ^ S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In S. C. Kremer and J. F. Kolen, editors, A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001.
  25. ^ Hochreiter, Sepp; and Schmidhuber, Jürgen; Long Short-Term Memory, Neural Computation, 9(8):1735–1780, 1997
  26. ^ Graves, Alex; and Schmidhuber, Jürgen; Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks, in Bengio, Yoshua; Schuurmans, Dale; Lafferty, John; Williams, Chris K. I.; and Culotta, Aron (eds.), Advances in Neural Information Processing Systems 22 (NIPS'22), December 7th–10th, 2009, Vancouver, BC, Neural Information Processing Systems (NIPS) Foundation, 2009, pp. 545–552
  27. ^ A. Graves, M. Liwicki, S. Fernandez, R. Bertolami, H. Bunke, J. Schmidhuber. A Novel Connectionist System for Improved Unconstrained Handwriting Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 5, 2009.
  28. ^ Sven Behnke (2003). Hierarchical Neural Networks for Image Interpretation.. Lecture Notes in Computer Science 2766. Springer. 
  29. ^ Smolensky, P. (1986). Information processing in dynamical systems: Foundations of harmony theory.. In D. E. Rumelhart, J. L. McClelland, & the PDP Research Group, Parallel Distributed Processing: Explorations in the Microstructure of Cognition. 1. pp. 194–281. 
  30. ^ Hinton, G. E.; Osindero, S.; Teh, Y. (2006). "A fast learning algorithm for deep belief nets". Neural Computation 18 (7): 1527–1554. doi:10.1162/neco.2006.18.7.1527. PMID 16764513. 
  31. ^ Hinton, G. (2009). "Deep belief networks". Scholarpedia 4 (5): 5947. doi:10.4249/scholarpedia.5947.  edit
  32. ^ John Markoff (25 June 2012). "How Many Computers to Identify a Cat? 16,000.". New York Times. 
  33. ^ Ng, Andrew; Dean, Jeff (2012). "Building High-level Features Using Large Scale Unsupervised Learning". 
  34. ^ a b D. C. Ciresan, U. Meier, J. Masci, L. M. Gambardella, J. Schmidhuber. Flexible, High Performance Convolutional Neural Networks for Image Classification. International Joint Conference on Artificial Intelligence (IJCAI-2011, Barcelona), 2011.
  35. ^ Martines, H., Bengio, Y., & Yannakakis, G. N. (2013). Learning Deep Physiological Models of Affect. I EEE Computational Intelligence, 8(2), 20.
  36. ^ D. C. Ciresan, U. Meier, J. Masci, J. Schmidhuber. Multi-Column Deep Neural Network for Traffic Sign Classification. Neural Networks, 2012.
  37. ^ D. Ciresan, A. Giusti, L. Gambardella, J. Schmidhuber. Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images. In Advances in Neural Information Processing Systems (NIPS 2012), Lake Tahoe, 2012.
  38. ^ D. C. Ciresan, U. Meier, J. Schmidhuber. Multi-column Deep Neural Networks for Image Classification. IEEE Conf. on Computer Vision and Pattern Recognition CVPR 2012.
  39. ^ T. Mikolov et al., "Recurrent neural network based language model," Interspeech, 2010.
  40. ^ Y. LeCun et al., "Gradient-based learning applied to document recognition," Proceedings of the IEEE, 86 (11), pp. 2278–2324.
  41. ^ T. Sainath et al., "Convolutional neural networks for LVCSR," ICASSP, 2013.
  42. ^ Y. Bengio et al., "Advances in optimizing recurrent networks," ICASSP, 2013.
  43. ^ G. Dahl et al., "Improving DNNs for LVCSR using rectified linear units and dropout," ICASSP, 2013.
  44. ^ a b c d G. E. Hinton., "A Practical Guide to Training Restricted Boltzmann Machines," Tech. Rep. UTML TR 2010-003, Dept. CS., Univ. of Toronto, 2010.
  45. ^ G.E. Hinton., "Deep belief networks," Scholarpedia, 4(5):5947.
  46. ^ H. Larochelle et al., "An empirical evaluation of deep architectures on problems with many factors of variation," in Proc. 24th Int. Conf. Machine Learning, pp. 473–480, 2007.
  47. ^ G. E. Hinton., "Training Product of Experts by Minimizing Contrastive Divergence," Neural Computation, 14, pp. 1771–1800, 2002.
  48. ^ A. Fischer and C. Igel. Training Restricted Boltzmann Machines: An Introduction. Pattern Recognition 47, pp. 25-39, 2014
  49. ^
  50. ^
  51. ^
  52. ^ a b c d Hinton, Geoffrey; Salakhutdinov, Ruslan (2012). "A better way to pretrain deep Boltzmann machines". Advances in Neural 3: 1–9. 
  53. ^ a b c Larochelle, Hugo; Salakhutdinov, Ruslan (2010). "Efficient Learning of Deep Boltzmann Machines". pp. 693–700. 
  54. ^ a b c d Hinton, Geoffrey; Salakhutdinov, Ruslan (2009). "Efficient Learning of Deep Boltzmann Machines" 3. pp. 448–455. 
  55. ^ a b Bengio, Yoshua; LeCun, Yann (2007). "Scaling Learning Algorithms towards AI" 1. pp. 1–41. 
  56. ^ a b c Hinton, Geoffrey; Osindero, Simon; Teh, Yee-Whye (2006). "A Fast Learning Algorithm for Deep Belief Nets". Neural Computation 1554: 1527–1554. 
  57. ^ McClelland, James; Feldman, J (1986). "Connectionist models and cognitive science: Goals, directions and implications". to the National Science Foundation:,+Directions+and+Implications#1. 
  58. ^ Hinton, Geoffrey (1989). "Connectionist learning procedures". Artificial intelligence. 40(1-3): 185–234. 
  59. ^ Utgoff, Paul; Stracuzzi, David (2002). "Many-layered learning". Neural computation. 14 (10): 2497–529. 
  60. ^ a b c d e f Vincent, Pascal; Larochelle, Hugo; Lajoie, Isabelle; Bengio, Yoshua; Manzagol, Pierre-Antoine (2010). "Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion". The Journal of Machine Learning Research 11: 3371–3408. 
  61. ^ Hastad, Johan (1986). "Almost optimal lower bounds for small depth circuits". Proceedings of the eighteenth annual ACM symposium on Theory of computing: 6–20. 
  62. ^ Hastad, Johan; Goldmann, Mikael (1991). "On the power of small-depth threshold circuits". Computational Complexity 1(2): 113–129. 
  63. ^ a b c Bengio, Yoshua (2009). "Learning deep architectures for AI". Foundations and trends in Machine Learning 2(1): 1–127. 
  64. ^ a b Bengio, Yoshua; Lamblin, Pascal; Popovici, Dan; Larochelle, Hugo (2007). "Greedy layer-wise training of deep networks". Advances in neural 1: 153–160. 
  65. ^ Ranzato, Marc’Aurelio; Poultney, Christopher; Chopra, Sumit; LeCun, Yann (2006). "Efficient learning of sparse representations with an energy-based model". Advances in neural: 1–8. 
  66. ^ a b Larochelle, Hugo; Bengio, Yoshua; Louradour, Jerdme; Lamblin, Pascal (2009). "Exploring Strategies for Training Deep Neural Networks". The Journal of Machine Learning Research 10: 1–40. 
  67. ^ a b c Deng, Li; Yu, Dong (2011). "Deep Convex Net: A Scalable Architecture for Speech Pattern Classification". Proceedings of the Interspeech: 2285–2288. 
  68. ^ a b c Deng, Li; Yu, Dong; Platt, John (2012). "Scalable stacking and learning for building deep architectures". 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): 2133–2136. 
  69. ^ David, Wolpert (1992). "Stacked generalization". Neural networks 5(2): 241–259. 
  70. ^ a b c Hutchinson, Brian; Deng, Li; Yu, Dong (2012). "Tensor deep stacking networks". IEEE transactions on pattern analysis and machine intelligence 1–15. 
  71. ^ Yu, Dong; Deng, Li (2011). "Accelerated Parallelizable Neural Network Learning Algorithm for Speech Recognition". INTERSPEECH: 2281–2284. 
  72. ^ Dahl, George; Ranzato, Marc’Aurelio; Mohamed, Abdel-rahman; Hinton, Geoffrey (2010). "Phone Recognition with the Mean-Covariance Restricted Boltzmann Machine". Advances in neural: 1–9. 
  73. ^ Ranzato, Marc’Aurelio; Krizhevsky, Alex; Hinton, Geoffrey (2010). "Factored 3-Way Restricted Boltzmann Machines For Modeling Natural Images". International 9: 621–628. 
  74. ^ a b Ranzato, Marc'Aurelio; Hinton, Geoffrey (2010). "Modeling pixel means and covariances using factorized third-order boltzmann machines". 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition: 2551–2558. 
  75. ^ Hutchinson, B; Yu, Dong (2012). "A deep architecture with bilinear modeling of hidden representations: Applications to phonetic recognition". Acoustics, Speech and Signal 3: 3–6. 
  76. ^ Hinton, Geoffrey; Salakhutdinov, Ruslan (2006). "Reducing the Dimensionality of Data with Neural Networks". Science 313: 504–507. 
  77. ^ Hinton, Geoffrey; Salakhutdinov, Ruslan (2012). "Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition". Audio, Speech, and ... 20(1): 30–42. 
  78. ^ Mohamed, Abdel-rahman; Dahl, George; Hinton, Geoffrey (2012). "Acoustic Modeling Using Deep Belief Networks". IEEE Transactions on Audio, Speech, and Language Processing. 20(1): 14–22. 
  79. ^ Le, Quoc; Ranzato, Marc'Aurelio (2011). "Building high-level features using large scale unsupervised learning". arXiv preprint arXiv. 
  80. ^ Courville, Aaron; Bergstra, James; Bengio, Yoshua (2011). "A Spike and Slab Restricted Boltzmann Machine". International . . . 15: 233–241. 
  81. ^ Mitchell, T; Beauchamp, J (1988). "Bayesian Variable Selection in Linear Regression". Journal of the American Statistical Association. 83 (404): 1023–1032. 
  82. ^ a b c d Courville, Aaron; Bergstra, James; Bengio, Yoshua (2011). "Unsupervised Models of Images by Spike-and-Slab RBMs". Proceedings of the . . . 10: 1–8. 
  83. ^ Ranzato, Marc'Aurelio; Mnih, Volodymyr (2010). "Generating more realistic images using gated MRF’s". . . . in Neural Information Processing . . .: 1–9. 
  84. ^ a b Kemp, Charles; Perfors, Amy; Tenenbaum, Joshua (2007). "Learning overhypotheses with hierarchical Bayesian models". Developmental science. 10(3): 307–21. 
  85. ^ Perfors, A; Tenenbaum, J (2009). "Learning to learn categories". Annual Conference 1955: 136–141. 
  86. ^ Smith, Linda; Jones, Susan; Landau, Barbara (2002). "Object name learning provides on-the-job training for attention". Psychol Sci. 13(1): 13–9. 
  87. ^ a b Xu, Fei; Tenenbaum, Joshua (2007). "Word learning as Bayesian inference". Psychol Rev. 114(2): 245–72. 
  88. ^ a b c Ruslan, Salakhutdinov; Joshua, Tenenbaum (2012). "Learning with Hierarchical-Deep Models". IEEE transactions on pattern analysis and machine intelligence: 1–14. 
  89. ^ Coates, Adam; Carpenter, Blake (2011). "Text Detection and Character Recognition in Scene Images with Unsupervised Feature Learning". pp. 440–445. 
  90. ^ Lee, Honglak; Grosse, Roger (2009). "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations". Proceedings of the 26th Annual International Conference on Machine Learning - ICML '09: 1–8. 
  91. ^ Lin, Yuanqing; Zhang, Tong (2010). "Deep Coding Network". Advances in Neural . . .: 1–9. 
  92. ^ Ranzato, Marc Aurelio; Boureau, Y-Lan (2007). "Sparse Feature Learning for Deep Belief Networks". Advances in neural information . . .: 1–8. 
  93. ^ Socher, Richard; Lin, Clif (2011). "Parsing Natural Scenes and Natural Language with Recursive Neural Networks". Proceedings of the . . . 
  94. ^ Taylor, Graham; Hinton, Geoffrey (2006). "Modeling Human Motion Using Binary Latent Variables". Advances in neural . . . 
  95. ^ Vincent, Pascal; Larochelle, Hugo (2008). "Extracting and composing robust features with denoising autoencoders". Proceedings of the 25th international conference on Machine learning - ICML '08: 1096–1103. 
  96. ^ Chen, Bo; Polatkan, Gungor (2011). "The Hierarchical Beta Process for Convolutional Factor Analysis and Deep Learning". Machine Learning . . . 
  97. ^ a b c Fei-Fei, Li; Fergus, Rob (2006). "One-shot learning of object categories". IEEE Trans Pattern Anal Mach Intell. 28(4): 594–611. 
  98. ^ Rodriguez, Abel; Dunson, David (2008). "The Nested Dirichlet Process". Journal of the American Statistical Association. 103(483): 1131–1154. 
  99. ^ Bart, Evgeniy (2008). "Unsupervised learning of visual taxonomies". 2008 IEEE Conference on Computer Vision and Pattern Recognition: 1–8. 
  100. ^ Canini, Kevin; Griffiths, Thomas (2009). "Modeling Human Transfer Learning with the Hierarchical Dirichlet Process". NIPS 2009 workshop: Nonparametric . . .: 1–3. 
  101. ^ Sudderth, Erik; Torralba, Antonio (2008). "Describing Visual Scenes Using Transformed Objects and Parts". International Journal of Computer Vision: 1–53. 
  102. ^ Torralba, Antonio; Murphy, Kevin (2006). "Shared Features for Multiclass Object Detection". Toward Category-Level Object Recognition. 
  103. ^ Bart, E.; Ullman, S. (2005). "Cross-generalization: learning novel classes from a single example by feature replacement". 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) 1: 672–679. 
  104. ^ Babenko, Boris; Branson, Steve (2009). "Similarity metrics for categorization: From monolithic to category specific". Computer Vision, 2009: 293–300. 
  105. ^ a b Chalasani, Rakesh; Principe, Jose (2013). "Deep Predictive Coding Networks". arXiv preprint arXiv: 1–13. 
  106. ^ a b Rao, R; Ballard, D (1997). "Dynamic model of visual recognition predicts neural response properties in the visual cortex". Neural computation 9(4): 721–63. 
  107. ^ a b Friston, Karl (2008). "Hierarchical Models in the Brain". PLoS Comput Biol. 4(11). 
  108. ^ a b Cho, Youngmin (2012). "Kernel Methods for Deep Learning". pp. 1–9. 
  109. ^ Scholkopf, B; Smola, Alexander (1998). "Nonlinear component analysis as a kernel eigenvalue problem". Neural computation (44). 
  110. ^ Yger, Florian; Berar, Maxime (2011). "A supervised strategy for deep kernel machine". ESANN 2011: 27–29. 
  111. ^ Rosipal, Roman; Trejo, Leonard (2003). "Kernel PLS-SVC for Linear and Nonlinear Classification". ICML. 
  112. ^ TIMIT Acoustic-Phonetic Continuous Speech Corpus Linguistic Data Consortium, Philadelphia.
  113. ^ Deng, L.; Platt, J. (2014). "Ensemble Deep Learning for Speech Recognition". Proc. Interspeech. 
  114. ^ Yu, D.; Deng, L. (2014). "Automatic Speech Recognition: A Deep Learning Approach (Publisher: Springer)". 
  115. ^ Deng, L.; Li, Xiao (2013). "Machine Learning Paradigms for Speech Recognition: An Overview". IEEE Transactions on Audio, Speech, and Language Processing. 
  116. ^
  117. ^ D. Ciresan, U. Meier, J. Schmidhuber., "Multi-column Deep Neural Networks for Image Classification," Technical Report No. IDSIA-04-12', 2012.
  118. ^ Y. Bengio, R. Ducharme, P. Vincent, C. Jauvin., "A Neural Probabilistic Language Model," Journal of Machine Learning Research 3 (2003) 1137–1155', 2003.
  119. ^ Goldberg, Yoav; Levy, Omar. "word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method". Arxiv. Retrieved 26 October 2014. 
  120. ^ a b Socher, Richard; Manning, Christopher. "Deep Learning for NLP". Retrieved 26 October 2014. 
  121. ^ Socher, Richard; Bauer, John; Manning, Christopher; Ng, Andrew (2013). "Parsing With Compositional Vector Grammars". Proceedings of the ACL 2013 conference. 
  122. ^ Socher, Richard (2013). "Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank". EMNLP 2013. 
  123. ^ P. E. Utgoff and D. J. Stracuzzi., "Many-layered learning," Neural Computation, 14, pp. 2497–2529, 2002.
  124. ^ J. Elman, et al., "Rethinking Innateness," 1996.
  125. ^ J. Shrager, MH Johnson., "Dynamic plasticity influences the emergence of function in a simple cortical array," Neural Networks, 9 (7), pp. 1119–1129, 1996
  126. ^ SR Quartz and TJ Sejnowski., "The neural basis of cognitive development: A constructivist manifesto," Behavioral and Brain Sciences, 20 (4), pp. 537–556, 1997.
  127. ^ S. Blakeslee., "In brain's early growth, timetable may be critical," The New York Times, Science Section, pp. B5–B6, 1995.
  128. ^ {BUFILL} E. Bufill, J. Agusti, R. Blesa., "Human neoteny revisited: The case of synaptic plasticity," American Journal of Human Biology, 23 (6), pp. 729–739, 2011.
  129. ^ J. Shrager and M. H. Johnson., "Timing in the development of cortical function: A computational approach," In B. Julesz and I. Kovacs (Eds.), Maturational windows and adult cortical plasticity, 1995.
  130. ^ D. Hernandez., "The Man Behind the Google Brain: Andrew Ng and the Quest for the New AI," Wired, 10 May 2013.
  131. ^ C. Metz., "Facebook's 'Deep Learning' Guru Reveals the Future of AI," Wired, 12 December 2013.
  132. ^ G. Marcus., "Is "Deep Learning" a Revolution in Artificial Intelligence?" The New Yorker, 25 November 2012.

External links[edit]