Generative Adversarial Network

GAN has two parts: a generator and a discriminator. In the case of numerical recognition, the generator takes in random numbers and returns an image. Then the images, either generated or given, are fed to the discriminator where the network makes a probabilistic decision on whether the images are authentic. The discriminator and generators are trained hand-in-hand: discriminator makes better decision while the generator learns by making more realistic images. Compared to VAEs, the images tend to be less blurry.

See Skymind.ai’s wiki on GAN

Liquide State Machine

https://polychord.io/neuralnetchart/img/LSM.png

LSMs are a type of spiking neural network. LSMs are sparse neural networks with activations replaced as threshold levels. The LSM models the behavior of the following: an input layer with randomly connected reservoir of neurons that add activations over the timesteps and fire after a threshold. The pattern involved with time varying inputs is called a spatio-temporal pattern (spiking neurons). Lastly, an output layer interprets the pattern and uses it for classification. LSMs are used for speech recognition or computer vision.

Extreme Learning Machine

Extreme Learning Machine

Extreme Learning Machines uses randomly chosen hidden nodes and analytically determines the output weights. This algorithm provides a good generalisation performance with extremely fast learning speeds. ELMs may seem haphazard, but it aims to resolve the key hindrances evident in FFNNs: slow gradient-based learning and iterative parameter tuning.

See this publication for more information about ELM

Echo State Network

Echo State Network

Echo State Networks are an instance of a general concept of Reservoir Computing. By incorporating the benefits of RNNs and not dealing with vanishing gradient problems, ESNs (1) assign random weights once initially, (2) feed training inputs, (3) linear regression comparison between target outputs and captured reservoir, (4) output weights are used as novel inputs. The theory behind is that random connections allow previous states to “echo” so that similar inputs will start to follow a predicted activation trajectory.

Deep Residual Network

Deep Residual Network

DRNs are small networks that aim to repeat the process to increase the depth of the network. Moreover, it also aims to give an intuitive insight in adding hidden layers. Therefore, new hidden layers learn something new by being fed inputs from previous hidden layers, essentially driving the new layer to learn something different. Fundamentally, DRNs are LSTM nodes but without gates. DRN works really well with image classification.

See Michael Dietz’s website on DRN
See Michael Dietz’s website on DRN

Kohonen Network

Kohonen Network

Kohonen Networks are used to produce a low-dimensional representation of the input. KNs are trained in an unsupervised learning manner. Every node is chosen at random from the set of training data and every node is examined to calculate which one’s weights are most like the input vector. The winning node is called the Best Matching Unit (BMU). The neighbouring node’s weights are adjusted to make them like the input vector, hence finishing training. Self organizing models are used pervasively in Image Browsing Systems, Medical Diagnosis, and Environmental Modelling.

See this website for more information about Kohonen Networks

Support Vector Machine

Support Vector Machine

SVMs are discriminative classifier defined by a separating hyperplane. Given labeled training data, the algorithm will find the optimal output to categorize the inputs.The tuning parameters include kernel, regularization, and gamma. Kernels do transformations and calculate separation lines, regularisation tells the algorithm the extent of misclassifying inputs, and the extent of how much input data is considered.

See Savan Patel’s explanation of SVM on Medium

Neural Turing Machine

Neural Turing Machine

NTMs are similar to a LSTM network, but instead of the memory stored in hidden state, the memory is stored somewhere external. The controller is the actual neural network, the memory is a matrix with memory size and item size as its axis, the read heads that “looks up” memory data, and a write head that decides where to write.

See Jonathan Hui’s explanation of NTM on Medium