Recurrent neural network

A recurrent neural network (RNN) is a class of s where connections between nodes form a along a temporal sequence. This allows it to exhibit temporal dynamic behavior. Unlike, RNNs can use their internal state (memory) to process sequences of inputs. This makes them applicable to tasks such as unsegmented, connected or.

The term "recurrent neural network" is used indiscriminately to refer to two broad classes of networks with a similar general structure, where one is and the other is. Both classes of networks exhibit temporal. A finite impulse recurrent network is a that can be unrolled and replaced with a strictly feedforward neural network, while an infinite impulse recurrent network is a  that can not be unrolled.

Both finite impulse and infinite impulse recurrent networks can have additional stored state, and the storage can be under direct control by the neural network. The storage can also be replaced by another network or graph, if that incorporates time delays or has feedback loops. Such controlled states are referred to as gated state or gated memory, and are part of networks (LSTMs) and s.

History
Recurrent neural networks were based on 's work in 1986. - a special kind of RNN - were discovered by in 1982. In 1993, a neural history compressor system solved a "Very Deep Learning" task that required more than 1000 subsequent layers in an RNN unfolded in time.

LSTM
(LSTM) networks were discovered by and  in 1997 and set accuracy records in multiple applications domains.

Around 2007, LSTM started to revolutionize, outperforming traditional models in certain speech applications. In 2009, a (CTC)-trained LSTM network was the first RNN to win pattern recognition contests when it won several competitions in connected. In 2014, the Chinese search giant used CTC-trained RNNs to break the Switchboard Hub5'00 speech recognition benchmark without using any traditional speech processing methods.

LSTM also improved large-vocabulary speech recognition and synthesis and was used in. In 2015, Google's speech recognition reportedly experienced a dramatic performance jump of 49% through CTC-trained LSTM, which was used by.

LSTM broke records for improved, and Multilingual Language Processing. LSTM combined with s (CNNs) improved.

Architectures
RNNs come in many variants.

Fully recurrent
Basic RNNs are a network of nodes organized into successive "layers." Each node in a given layer is connected with a to every other node in the next successive layer. Each node (neuron) has a time-varying real-valued activation. Each connection (synapse) has a modifiable real-valued. Nodes are either input nodes (receiving data from outside the network), output nodes (yielding results), or hidden nodes (that modify the data en route from input to output).

For in discrete time settings, sequences of real-valued input vectors arrive at the input nodes, one vector at a time. At any given time step, each non-input unit computes its current activation (result) as a nonlinear function of the weighted sum of the activations of all units that connect to it. Supervisor-given target activations can be supplied for some output units at certain time steps. For example, if the input sequence is a speech signal corresponding to a spoken digit, the final target output at the end of the sequence may be a label classifying the digit.

In settings, no teacher provides target signals. Instead a or  is occasionally used to evaluate the RNN's performance, which influences its input stream through output units connected to actuators that affect the environment. This might be used to play a game in which progress is measured with the number of points won.

Each sequence produces an error as the sum of the deviations of all target signals from the corresponding activations computed by the network. For a training set of numerous sequences, the total error is the sum of the errors of all individual sequences.

Elman networks and Jordan networks
An network is a three-layer network (arranged horizontally as x, y, and z in the illustration) with the addition of a set of "context units" (u in the illustration). The middle (hidden) layer is connected to these context units fixed with a weight of one. At each time step, the input is fed forward and a learning rule is applied. The fixed back-connections save a copy of the previous values of the hidden units in the context units (since they propagate over the connections before the learning rule is applied). Thus the network can maintain a sort of state, allowing it to perform such tasks as sequence-prediction that are beyond the power of a standard.

networks are similar to Elman networks. The context units are fed from the output layer instead of the hidden layer. The context units in a Jordan network are also referred to as the state layer. They have a recurrent connection to themselves.

Elman and Jordan networks are also known as "simple recurrent networks" (SRN).


 * Elman network

\begin{align} h_t &= \sigma_h(W_{h} x_t + U_{h} h_{t-1} + b_h) \\ y_t &= \sigma_y(W_{y} h_t + b_y) \end{align} $$
 * Jordan network

\begin{align} h_t &= \sigma_h(W_{h} x_t + U_{h} y_{t-1} + b_h) \\ y_t &= \sigma_y(W_{y} h_t + b_y) \end{align} $$

Variables and functions
 * $$x_t$$: input vector
 * $$h_t$$: hidden layer vector
 * $$y_t$$: output vector
 * $$W$$, $$U$$ and $$b$$: parameter matrices and vector
 * $$\sigma_h$$ and $$\sigma_y$$: s

Hopfield
The is an RNN in which all connections are symmetric. It requires inputs and is thus not a general RNN, as it does not process sequences of patterns. It guarantees that it will converge. If the connections are trained using then the Hopfield network can perform as , resistant to connection alteration.

Bidirectional associative memory
Introduced by Bart Kosko, a bidirectional associative memory (BAM) network is a variant of a Hopfield network that stores associative data as a vector. The bi-directionality comes from passing information through a matrix and its. Typically, bipolar encoding is preferred to binary encoding of the associative pairs. Recently, stochastic BAM models using stepping were optimized for increased network stability and relevance to real-world applications.

A BAM network has two layers, either of which can be driven as an input to recall an association and produce an output on the other layer.

Echo state
The echo state network (ESN) has a sparsely connected random hidden layer. The weights of output neurons are the only part of the network that can change (be trained). ESNs are good at reproducing certain. A variant for is known as a.

Attention-aware Bidirectional Multi-residual Recurrent Neural Network (ABMRNN)
Due to the limited memory capacity in LSTM cells, LSTM is weak in capturing long-time dependency in sequential data. To address this challenge, an Attention-aware Bidirectional Multi-residual Recurrent Neural Network (ABMRNN) has been proposed to overcome the deficiency. ABMRNN model considers both past and future information at every time step with omniscient attention based on LSTM. In addition to that, the multi-residual mechanism has been leveraged in our model which aims to model the relationship between current time step with further distant time steps instead of a just previous time step. The results of experiments show that our model achieves state-of-the-art performance in classification tasks.

Independent RNN (IndRNN)
The Independently recurrent neural network (IndRNN) addresses the gradient vanishing and exploding problems in the traditional fully connected RNN. Each neuron in one layer only receives its own past state as context information (instead of full connectivity to all other neurons in this layer) and thus neurons are independent of each other's history. The gradient backpropagation can be regulated to avoid gradient vanishing and exploding in order to keep long or short-term memory. The cross-neuron information is explored in the next layers. IndRNN can be robustly trained with the non-saturated nonlinear functions such as ReLU. Using skip connections, deep networks can be trained.

Recursive
A is created by applying the same set of weights  over a differentiable graph-like structure by traversing the structure in. Such networks are typically also trained by the reverse mode of. They can process s of structure, such as. A special case of recursive neural networks is the RNN whose structure corresponds to a linear chain. Recursive neural networks have been applied to. The Recursive Neural Tensor Network uses a -based composition function for all nodes in the tree.

Neural history compressor
The neural history compressor is an unsupervised stack of RNNs. At the input level, it learns to predict its next input from the previous inputs. Only unpredictable inputs of some RNN in the hierarchy become inputs to the next higher level RNN, which therefore recomputes its internal state only rarely. Each higher level RNN thus studies a compressed representation of the information in the RNN below. This is done such that the input sequence can be precisely reconstructed from the representation at the highest level.

The system effectively minimises the description length or the negative of the probability of the data. Given a lot of learnable predictability in the incoming data sequence, the highest level RNN can use supervised learning to easily classify even deep sequences with long intervals between important events.

It is possible to distill the RNN hierarchy into two RNNs: the "conscious" chunker (higher level) and the "subconscious" automatizer (lower level). Once the chunker has learned to predict and compress inputs that are unpredictable by the automatizer, then the automatizer can be forced in the next learning phase to predict or imitate through additional units the hidden units of the more slowly changing chunker. This makes it easy for the automatizer to learn appropriate, rarely changing memories across long intervals. In turn this helps the automatizer to make many of its once unpredictable inputs predictable, such that the chunker can focus on the remaining unpredictable events.

A partially overcame the  of  or  in neural networks in 1992. In 1993, such a system solved a "Very Deep Learning" task that required more than 1000 subsequent layers in an RNN unfolded in time.

Second order RNNs
Second order RNNs use higher order weights $$w{}_{ijk}$$ instead of the standard $$w{}_{ij}$$ weights, and states can be a product. This allows a direct mapping to a both in training, stability, and representation. Long short-term memory is an example of this but has no such formal mappings or proof of stability.

Long short-term memory
Long short-term memory (LSTM) is a system that avoids the. LSTM is normally augmented by recurrent gates called "forget" gates. LSTM prevents backpropagated errors from vanishing or exploding. Instead, errors can flow backwards through unlimited numbers of virtual layers unfolded in space. That is, LSTM can learn tasks that require memories of events that happened thousands or even millions of discrete time steps earlier. Problem-specific LSTM-like topologies can be evolved. LSTM works even given long delays between significant events and can handle signals that mix low and high frequency components.

Many applications use stacks of LSTM RNNs and train them by to find an RNN weight matrix that maximizes the probability of the label sequences in a training set, given the corresponding input sequences. CTC achieves both alignment and recognition.

LSTM can learn to recognize unlike previous models based on s (HMM) and similar concepts.

Gated recurrent unit
Gated recurrent units (GRUs) are a gating mechanism in introduced in 2014. They are used in the full form and several simplified variants. Their performance on polyphonic music modeling and speech signal modeling was found to be similar to that of long short-term memory. They have fewer parameters than LSTM, as they lack an output gate.

Bi-directional
Bi-directional RNNs use a finite sequence to predict or label each element of the sequence based on the element's past and future contexts. This is done by concatenating the outputs of two RNNs, one processing the sequence from left to right, the other one from right to left. The combined outputs are the predictions of the teacher-given target signals. This technique proved to be especially useful when combined with LSTM RNNs.

Continuous-time
A continuous time recurrent neural network (CTRNN) uses a system of to model the effects on a neuron of the incoming spike train.

For a neuron $$i$$ in the network with $$y_{i}$$, the rate of change of activation is given by:
 * $$\tau_{i}\dot{y}_{i}=-y_{i}+\sum_{j=1}^{n}w_{ji}\sigma(y_{j}-\Theta_{j})+I_{i}(t)$$

Where:
 * $$\tau_{i}$$ : Time constant of node
 * $$y_{i}$$ : Activation of postsynaptic node
 * $$\dot{y}_{i}$$ : Rate of change of activation of postsynaptic node
 * $$w{}_{ji}$$ : Weight of connection from pre to postsynaptic node
 * $$\sigma(x)$$ : Sigmoid of x e.g. $$\sigma(x) = 1/(1+e^{-x})$$.
 * $$y_{j}$$ : Activation of presynaptic node
 * $$\Theta_{j}$$ : Bias of presynaptic node
 * $$I_{i}(t)$$ : Input (if any) to node

CTRNNs have been applied to where they have been used to address vision, co-operation, and minimal cognitive behaviour.

Note that, by the, discrete time recurrent neural networks can be viewed as continuous-time recurrent neural networks where the differential equations have transformed into equivalent s. This transformation can be thought of as occurring after the post-synaptic node activation functions $$y_i(t)$$ have been low-pass filtered but prior to sampling.

Hierarchical
Hierarchical RNNs connect their neurons in various ways to decompose hierarchical behavior into useful subprograms.

Recurrent multilayer perceptron network
Generally, a Recurrent Multi-Layer Perceptron (RMLP) network consists of cascaded subnetworks, each of which contains multiple layers of nodes. Each of these subnetworks is feed-forward except for the last layer, which can have feedback connections. Each of these subnets is connected only by feed forward connections.

Multiple timescales model
A multiple timescales recurrent neural network (MTRNN) is a neural-based computational model that can simulate the functional hierarchy of the brain through self-organization that depends on spatial connection between neurons and on distinct types of neuron activities, each with distinct time properties. With such varied neuronal activities, continuous sequences of any set of behaviors are segmented into reusable primitives, which in turn are flexibly integrated into diverse sequential behaviors. The biological approval of such a type of hierarchy was discussed in the theory of brain function by  in his book .

Neural Turing machines
Neural Turing machines (NTMs) are a method of extending recurrent neural networks by coupling them to external resources which they can interact with by es. The combined system is analogous to a or  but is  end-to-end, allowing it to be efficiently trained with.

Differentiable neural computer
Differentiable neural computers (DNCs) are an extension of Neural Turing machines, allowing for usage of fuzzy amounts of each memory address and a record of chronology.

Neural network pushdown automata
Neural network pushdown automata (NNPDA) are similar to NTMs, but tapes are replaced by analogue stacks that are differentiable and that are trained. In this way, they are similar in complexity to recognizers of s (CFGs).

Memristive Networks
Greg Snider of describes a system of cortical computing with memristive nanodevices. The (memory resistors) are implemented by thin film materials in which the resistance is electrically tuned via the transport of ions or oxygen vacancies within the film. 's has funded IBM Research and HP Labs, in collaboration with the Boston University Department of Cognitive and Neural Systems (CNS), to develop neuromorphic architectures which may be based on memristive systems. Memristive networks are a particular type of that have very similar properties to (Little-)Hopfield networks, as they have a continuous dynamics, have a limited memory capacity and they natural relax via the minimization of a function which is asymptotic to the Ising model. In this sense, the dynamics of a memristive circuit has the advantage compared to a Resistor-Capacitor network to have a more interesting non-linear behavior. From this point of view, engineering an analog memristive networks accounts to a peculiar type of in which the device behavior depends on the circuit wiring, or topology.

Gradient descent
Gradient descent is a    for finding the minimum of a function. In neural networks, it can be used to minimize the error term by changing each weight in proportion to the derivative of the error with respect to that weight, provided the non-linear s are. Various methods for doing so were developed in the 1980s and early 1990s by, , , , , Pearlmutter and others.

The standard method is called "" or BPTT, and is a generalization of for feed-forward networks. Like that method, it is an instance of in the reverse accumulation mode of. A more computationally expensive online variant is called "Real-Time Recurrent Learning" or RTRL, which is an instance of in the forward accumulation mode with stacked tangent vectors. Unlike BPTT, this algorithm is local in time but not local in space.

In this context, local in space means that a unit's weight vector can be updated using only information stored in the connected units and the unit itself such that update complexity of a single unit is linear in the dimensionality of the weight vector. Local in time means that the updates take place continually (on-line) and depend only on the most recent time step rather than on multiple time steps within a given time horizon as in BPTT. Biological neural networks appear to be local with respect to both time and space.

For recursively computing the partial derivatives, RTRL has a time-complexity of O(number of hidden x number of weights) per time step for computing the, while BPTT only takes O(number of weights) per time step, at the cost of storing all forward activations within the given time horizon. An online hybrid between BPTT and RTRL with intermediate complexity exists, along with variants for continuous time.

A major problem with gradient descent for standard RNN architectures is that exponentially quickly with the size of the time lag between important events. LSTM combined with a BPTT/RTRL hybrid learning method attempts to overcome these problems. This problem is also solved in the independently recurrent neural network (IndRNN) by reducing the context of a neuron to its own past state and the cross-neuron information can then be explored in the following layers. Memories of different range including long-term memory can be learned without the gradient vanishing and exploding problem.

The on-line algorithm called causal recursive backpropagation (CRBP), implements and combines BPTT and RTRL paradigms for locally recurrent networks. It works with the most general locally recurrent networks. The CRBP algorithm can minimize the global error term. This fact improves stability of the algorithm, providing a unifying view on gradient calculation techniques for recurrent networks with local feedback.

One approach to the computation of gradient information in RNNs with arbitrary architectures is based on signal-flow graphs diagrammatic derivation. It uses the BPTT batch algorithm, based on Lee's theorem for network sensitivity calculations. It was proposed by Wan and Beaufays, while its fast online version was proposed by Campolucci, Uncini and Piazza.

Global optimization methods
Training the weights in a neural network can be modeled as a non-linear problem. A target function can be formed to evaluate the fitness or error of a particular weight vector as follows: First, the weights in the network are set according to the weight vector. Next, the network is evaluated against the training sequence. Typically, the sum-squared-difference between the predictions and the target values specified in the training sequence is used to represent the error of the current weight vector. Arbitrary global optimization techniques may then be used to minimize this target function.

The most common global optimization method for training RNNs is s, especially in unstructured networks.

Initially, the genetic algorithm is encoded with the neural network weights in a predefined manner where one in the  represents one weight link. The whole network is represented as a single chromosome. The fitness function is evaluated as follows:

Many chromosomes make up the population; therefore, many different neural networks are evolved until a stopping criterion is satisfied. A common stopping scheme is: The stopping criterion is evaluated by the fitness function as it gets the reciprocal of the mean-squared-error from each network during training. Therefore, the goal of the genetic algorithm is to maximize the fitness function, reducing the mean-squared-error.
 * Each weight encoded in the chromosome is assigned to the respective weight link of the network.
 * The training set is presented to the network which propagates the input signals forward.
 * The mean-squared-error is returned to the fitness function.
 * This function drives the genetic selection process.
 * When the neural network has learnt a certain percentage of the training data or
 * When the minimum value of the mean-squared-error is satisfied or
 * When the maximum number of training generations has been reached.

Other global (and/or evolutionary) optimization techniques may be used to seek a good set of weights, such as or.

Related fields and models
RNNs may behave. In such cases, may be used for analysis.

They are in fact s with a particular structure: that of a linear chain. Whereas recursive neural networks operate on any hierarchical structure, combining child representations into parent representations, recurrent neural networks operate on the linear progression of time, combining the previous time step and a hidden representation into the representation for the current time step.

In particular, RNNs can appear as nonlinear versions of and  filters and also as a  (NARX).

Libraries

 * : Created by the Berkeley Vision and Learning Center (BVLC). It supports both CPU and GPU. Developed in, and has and  wrappers.
 * : The first stable deep learning library that supports dynamic, define-by-run neural networks. Fully in Python, production support for CPU, GPU, distributed training.
 * : Deep learning in and  on multi-GPU-enabled . A general-purpose deep learning library for the  production stack running on a C++ scientific computing engine. Allows the creation of custom layers. Integrates with  and.
 * Dynet: The Dynamic Neural Networks toolkit.
 * Flux: includes inferfaces for RNNs, including GRUs and LSTMs, written in.
 * : High-level, easy to use API, providing a wrapper to many other deep learning libraries.
 * : a modern open-source deep learning framework used to train and deploy deep neural networks.
 * : Tensors and Dynamic neural networks in Python with strong GPU acceleration.
 * : Apache 2.0-licensed Theano-like library with support for CPU, GPU and Google's proprietary, mobile
 * : The reference deep-learning library for Python with an API largely compatible with the popular library. Allows user to write symbolic mathematical expressions, then automatically generates their derivatives, saving the user from having to code gradients or backpropagation. These symbolic expressions are automatically compiled to CUDA code for a fast, on-the-GPU implementation.
 * (www.torch.ch): A scientific computing framework with wide support for machine learning algorithms, written in and . The main author is Ronan Collobert, and it is now used at Facebook AI Research and Twitter.
 * : The reference deep-learning library for Python with an API largely compatible with the popular library. Allows user to write symbolic mathematical expressions, then automatically generates their derivatives, saving the user from having to code gradients or backpropagation. These symbolic expressions are automatically compiled to CUDA code for a fast, on-the-GPU implementation.
 * (www.torch.ch): A scientific computing framework with wide support for machine learning algorithms, written in and . The main author is Ronan Collobert, and it is now used at Facebook AI Research and Twitter.

Applications
Applications of Recurrent Neural Networks include:
 * Time series anomaly detection
 * Rhythm learning
 * Music composition
 * Grammar learning
 * Human action recognition
 * Protein Homology Detection
 * Predicting subcellular localization of proteins
 * Several prediction tasks in the area of business process management
 * Prediction in medical care pathways
 * Human action recognition
 * Protein Homology Detection
 * Predicting subcellular localization of proteins
 * Several prediction tasks in the area of business process management
 * Prediction in medical care pathways
 * Prediction in medical care pathways