Feedforward neural network

src: www.pyimagesearch.com

A feedforward nerve network is a neural network where connections between nodes do not form cycles. Thus, it is different from recurrent neural networks.

The feedforward nerve network is the first and simplest type of artificial neural network. In this network, information moves only one direction, forward, from the input node, through the hidden node (if any) and to the output node. There are no cycles or loops in the network.

Video Feedforward neural network

Perceptron single-layer

The simplest type of neural network is a single-layer perceptron network, which consists of one layer of output node; the input is given directly to the output through a series of weights. In this way it can be considered the simplest type of feed-forward network. The number of products of weights and inputs is calculated at each node, and if its value is above some threshold (usually 0) the neuron illuminates and retrieves the activated value (usually 1); otherwise it needs a disabled value (usually -1). Neurons with this kind of activation function are also called artificial neurons or linear threshold units. In the literature, the term often refers to a network consisting of only one unit. A similar neuron was described by Warren McCulloch and Walter Pitts in the 1940s.

Perceptron can be created using any value for enabled and disabled status as long as the threshold value lies between the two.

Perceptrons can be trained by simple learning algorithms that are usually called delta rules . It calculates the error between the calculated output and the sample output data, and uses this to make adjustments to the weights, thus applying the descent gradient form.

Perceptrons single units can only learn linearly separable patterns; in 1969 in the famous monograph entitled Perceptrons, Marvin Minsky and Seymour Papert showed that it is impossible for a single-layer perceptron network to study the XOR function (although it is known that multi-layer perceptrons are capable of producing boolean functions which may).

Although a single threshold unit is fairly limited in computing power, it has been shown that the parallel threshold unit network can approach the continuous function of the concise interval of the real number to the interval [-1.1]. These results can be found in Peter Auer, Harald Burgsteiner and Wolfgang Maass "A learning rule for very simple universal approximators consisting of a single layer of perceptrons".

Sebuah jaringan syaraf multi-layer dapat menghitung output yang kontinu daripada fungsi langkah. Pilihan umum adalah apa yang disebut fungsi logistik:

${\ displaystyle f (x) = {\ frac {1} {1 e ^ {- x}}}}  ÂÂ$

With this option, single-layer networks are identical to logistic regression models, widely used in statistical modeling. Logistics functions are also known as sigmoid functions. It has a continuous derivative, which allows for use in backpropagation. This function is also preferred because the derivatives are easily calculated: $xmlns\; =\; "http://www.w3.org/1998/Math/MathML"\; alttext\; =\; "\{\backslash \; displaystyle\; f\; \text{'}(x)\; =\; f\; (x)\; (\; 1-f\; (x))\}\; ">\; Â\; ÂÂ\; Â\; Â\; ÂÂ\; Â\; Â\; Â\; Â{\displaystyle Â\; Â\; Â\; Â\; Â\; Â\; Â\; Âf^{?}()\; Â\; Â\; Â\; Â\; Â\; Â\; Â\; ÂxÂ\; Â\; Â\; Â\; Â\; Â\; Â\; Â)Â\; Â\; Â\; Â\; Â\; Â\; Â\; Â=Â\; Â\; Â\; Â\; Â\; Â\; Â\; Âf()\; Â\; Â\; Â\; Â\; Â\; Â\; Â\; ÂxÂ\; Â\; Â\; Â\; Â\; Â\; Â\; Â)()\; Â\; Â\; Â\; Â\; Â\; Â\; Â\; Â1Â\; Â\; Â\; Â\; Â\; Â\; Â\; Â-Â\; Â\; Â\; Â\; Â\; Â\; Â\; Âf()\; Â\; Â\; Â\; Â\; Â\; Â\; Â\; ÂxÂ\; Â\; Â\; Â\; Â\; Â\; Â\; Â)Â\; Â\; Â\; Â\; Â\; Â\; Â\; Â)Â\; Â\; Â\; Â\; Â\; Â}Â\; Â\; Â\; ÂAnnotation\; encoding\; =\; "application/x-tex">\; \{\backslash \; displaystyle\; f\; \text{'}(x)\; =\; f\; (x)\; (1-f\; (x))\}\; Â\; Â$ .

(The fact that satisfying the above differential equation can be easily demonstrated by applying Chain Rules.)

Maps Feedforward neural network

Multi-layer Perceptron

This network class consists of several layers of computing units, usually interconnected by way of feed-forward. Each neuron in one layer has directed the connection to the neurons of the next layer. In many applications, these network units implement the sigmoid function as an activation function.

The universal approximation theorem for neural networks states that any continuous function that maps real-time intervals to multiple real-time output intervals can be approached arbitrarily close by multi-layer perceptron with just one hidden layer. This result applies to various activation functions, e.g. for sigmoidal function.

Multi-layer networks use various learning techniques, the most popular being backward propagation . Here, the output value is compared with the correct answer to calculate the value of some predefined error function. With various techniques, the error is then fed back through the network. Using this information, the algorithm adjusts the weights of each connection to reduce the value of the error function by a small amount. After repeating this process for a considerable number of training cycles, the network will usually merge into a state where the error is small. In this case, people will say that the network has studied a specific target function. To adjust the weights correctly, one applies a general method for non-linear optimization called gradient descent. For this, the network calculates the derivative of the error function with respect to the network weights, and changes the weights in such a way that the error decreases (so it will decrease on the surface of the error function). For this reason, back propagation can only be applied to networks with differentiated activation functions.

In general, the problem of network teaching to perform well, even on samples that are not used as training samples, is a pretty subtle problem requiring additional techniques. This is particularly important for cases where only a very limited number of training samples are available. The danger is that network overfits training data and fails to capture the correct statistical process of generating data. Computational learning theory deals with training classifiers on a limited number of data. In the context of neural networks, simple heuristics, called initial cessation, often ensure that the network will generalize well to the examples that are not in the training device.

Another typical problem of back propagation algorithms is the speed of convergence and the possibility of ending up in the local minimum of the error function. There are currently practical methods that make propagation back in multi-layer perceptrons of choice tools for many of the machine learning tasks.

One can also use a series of independent neural networks that are moderated by intermediaries, similar behaviors that occur in the brain. These neurons can work separately and handle large tasks, and the results can eventually be combined.

src: www.pyimagesearch.com

See also

• Hopfield network
• Convolutional neural network
• Feed-forward
• Backpropagation
• Rprop

src: blog.boxx.com

References

src: i.ytimg.com

External links

• feedforward neural network tutorial
• Feedforward Nerve Network: Example
• Feedforward Nerve Network: An Introduction

Source of the article : Wikipedia