Simplified spiking neural network architecture and STDP learning algorithm applied to image classification
© Iakymchuk et al.; licensee Springer. 2015
Received: 30 April 2014
Accepted: 28 January 2015
Published: 19 February 2015
Spiking neural networks (SNN) have gained popularity in embedded applications such as robotics and computer vision. The main advantages of SNN are the temporal plasticity, ease of use in neural interface circuits and reduced computation complexity. SNN have been successfully used for image classification. They provide a model for the mammalian visual cortex, image segmentation and pattern recognition. Different spiking neuron mathematical models exist, but their computational complexity makes them ill-suited for hardware implementation. In this paper, a novel, simplified and computationally efficient model of spike response model (SRM) neuron with spike-time dependent plasticity (STDP) learning is presented. Frequency spike coding based on receptive fields is used for data representation; images are encoded by the network and processed in a similar manner as the primary layers in visual cortex. The network output can be used as a primary feature extractor for further refined recognition or as a simple object classifier. Results show that the model can successfully learn and classify black and white images with added noise or partially obscured samples with up to ×20 computing speed-up at an equivalent classification ratio when compared to classic SRM neuron membrane models. The proposed solution combines spike encoding, network topology, neuron membrane model and STDP learning.
KeywordsSpiking neural networks - SNN STDP Visual receptive fields Spike coding Embedded system Artificial neuron Image classification
In the last years, the popularity of spiking neural networks (SNN) and spiking models has increased. SNN are suitable for a wide range of applications such as pattern recognition and clustering, among others.There are examples of intelligent systems, converting data directly from sensors [1,2], controlling manipulators  and robots , doing recognition or detection tasks [5,6], tactile sensing  or processing neuromedical data . Different neuron models exist  but their computational complexity and memory requirements are high, limiting their use in robotics, embedded systems and real-time or mobile applications in general.
Existing simplified bio-inspired neural models [10,11] are focused on spike train generation and real neuron modeling. These models are rarely applied in practical tasks. Some of the neuronal models are applied only for linearly separable classes  and focus on small network simulation.
Concerning hardware implementation, dedicated ASIC solutions exist such as SpiNNaker , BrainScaleS , SyNAPSE  or others , but they are targeted for large-scale simulations rather than portable, low-power and real-time embedded applications. The model we propose is mainly oriented for applications requiring low-power, small and efficient hardware systems. It can also be used for computer simulations with up to ×20 speed-up compared to classic SRM neuron membrane model. Nowadays, due to a continuous decrease in price and increase in computation capabilities, combined with the progress in high-level hardware description language (HDL) synthesis tools, configurable devices such as FPGA can be used as efficient hardware accelerators for neuromorphic systems. A proposal was made by Schrauwen and Van Campenhout  using serial arithmetic to reduce hardware resource consumption, but no training or weight adaptation was possible. Other solution, presented by Rice et al.  used full-scale Izhikevich neurons with very high resource consumption (25 neurons occupy 79% of logic resources in a Virtex4 FPGA device), without on-line training.
Computation methods used for FPGA dramatically differ from classic methods used in Von Neumann PCs or even SIMD processing units like GPUs or DSPs. Thus, the required SNN hardware architecture must be different for reconfigurable devices, opening new possibilities for computation optimization. FPGA are optimal for massive parallel and relatively simple processing units rather than large universal computational blocks as is in case of SNN, including lots of multiply-add arithmetic blocks and vast quantities of distributed block RAM . This work describes computation algorithms properly modeling the SNN and its training algorithm, specifically targeted to benefit from reconfigurable hardware blocks. The proposed solution combines spike encoding, topology, neuron membrane model and spike-time dependent plasticity (STDP) learning.
2 Spiking neural networks model
Spiking neural networks are considered to be the third generation of artificial neural networks (ANN). While classic ANN operate with real or integer-valued inputs, SNN process data in form of series of spikes called spike trains, which, in terms of computation means that a single bit line toggling between logical levels ‘0’ and ‘1’ is required. SNN are able to process temporal patterns, not only spatial, and SNN are more computationally powerful than ANN . Classic machine learning methods perform poorly for spike coded data, being unsuitable for SNN. As a consequence, different training and network topology optimization algorithms must be used [9,21].
The SNN model used in this work is the feed-forward network, each neuron is connected to all the neurons in the next layer by a weighted connection, which means that the output signal of a neuron has a different weighted potential contribution . Input neurons require spike trains and input signals (stimuli) need to be encoded into spikes (typically, spike trains) to further feed the SNN.
An approximation to the functionality of a neuron is given by electrical models which reproduce the functionality of neuronal cells. One of the most common models is the spike response model (SRM) due to the close approximation to a real biological neuron [23,24]; the SRM is a generalization of the ‘integrate and fire’ model . The main characteristic of a spiking neuron is the membrane potential, the transmission of a single spike from one neuron to another is mediated by synapses at the point where neurons interact. In neuroscience, a transmitting neuron is defined as a presynaptic neuron and a receiving neuron as a postsynaptic neuron. With no activity, neurons have a small negative electrical charge of −70 mV, which is called resting potential; when a single spike arrives into a postsynaptic neuron, it generates a post synaptic potential (PSP) which is excitatory when the membrane potential is increasing and inhibitory when decreasing. The membrane potential at an instant is calculated as the sum of all present PSP at the neuron inputs. When the membrane potential is above a critical threshold value, a postsynaptic spike is generated, entering the neuron into a refractory period when the membrane remains overpolarized, preventing neurons from generating new spikes temporarily. After a refractory period, the neuron potential returns to its resting value and is ready to fire a new spike if membrane potential is above the threshold.
These equations define the SRM, which can be modeled by analog circuits since the PSP function can be seen as a charging and discharging RC circuit. However, this model is computationally complex when used in digital systems. We propose to use a simplified model with linear membrane potential degradation with similar performance and learning capabilities as the classic SRM.
3 Simplified spiking neural model
Thus, instead of an initial postsynaptic potential ramp in the spike response model, the instant change of membrane potential allows a neuron to fire immediately in the next clock cycle after the spike arrives.
3.1 Spike-time dependent plasticity learning
Since unsupervised learning requires competition, lateral inhibition was introduced and thus, the weights of the winner neurons (first spiking neurons) are increased while other neurons suffer a small weight reduction value. Tests showed that depressing the weights of the non firing neurons decrease the amount of noise in the network. The depression of synapses that do not fire at all was added in order to eliminate ‘mute’ synapses (inactive synapses), reducing the network size and improving robustness against noise. This training causes a side effect since, for weight increase, spike-intense patterns require a higher membrane threshold, avoiding the patterns with low spike intensity to be recognized by the network. This is solved by introducing negative weights, preventing neurons from reacting on every pattern and increasing the specificity of classifier.
4 Visual receptive fields
The visual cortex is one of the best studied parts of the brain. The receptive field (RF) of a visual neuron is an area of the image affecting the neural input. The size and shape of receptive fields vary depending on the neuron position and neuron task. A variety of tasks can be done with RFs: edge detection, sharpening, blurring, line decomposition, etc. In each subsequent layer of the visual cortex, receptive fields of the neurons cover bigger and bigger regions, convolving the outputs of the previous layer.
Mammalian retinal ganglion cells located at the center of vision, in the fovea, have the smallest receptive fields, and those located in the visual periphery have the largest receptive fields . The large receptive field size of neurons in the visual periphery explains the poor spatial resolution of human vision outside the point of fixation, together with photoreceptor density and optical aberrations. Only a few cortical receptive fields resemble the structure of thalamic receptive fields, some fields have elongated subregions responding to dark or light spots, while others do not respond to spots at all. In addition, the implementation of a receptive field is a first stage of sparse coding  where the neurons are reacting to shapes, not single pixels. The receptive field model proposed here shows a good approximation to the real behavior of primary visual cortex.
4.1 Receptive field neuron response
5 Software simulation and results
5.1 Image encoding
5.2 Network architecture
The proposed SNN consists of 2 layers, an encoding layer of 256 neurons with an on-centered 5 × 5 pixel RF and second layer of 16 neurons using the simplified SRM. Experimental testing showed that, for proper competitiveness in the network, the number of neurons should be at least 20% greater than the number of classes and thus, 16 neurons were implemented. If the number of neurons is insufficient, only the most spike-intensive patterns are learnt. Each sample was presented to the network during 200 time units (TUs). With a refractory period of encoding neurons of 30 TUs, the maximum possible amount of spikes is 200/30 = 6. STDP parameters for learning were A +=0.6,A −=0.3,τ +=8,τ −=5. The maximum weight change rate σ was fixed to 0.25∗max(STDP)=0.25∗0.25=0.0625.
For the classic SRM algorithm, a table-based PSP function of 30 points was used (simplified model uses constant decrease as PSP and does not require table-based functions). For both SRM and simplified models, STDP function was also table-based with 30 positive and 30 negative values. All algorithms (classic and simplified model) were written using atomic operations without the usage of Matlab vector and matrix arithmetic. Such coding style provides more accurate results in performance tests when modeling hardware implementation.
Simulation speed of classic and simplified networks
5 classes, 8 neurons, 15,000 time units
6 classes, 9 neurons, 15,000 time units
6 classes, 16 neurons, 15,000 time units
6 classes, 100 neurons, 15,000 time units
12 classes, 50 neurons, 96,000 time units
In this paper, we describe a simplified spiking neuron architecture optimized for embedded systems implementation, proving the learning capabilities of the design. The network preserves its learning and classification properties while computational and memory complexity is reduced dramatically - by eliminating the PSP table in each neuron. Learning is stable and robust, the trained network can recognize noisy patterns. A simple, yet effective visual input encoding was implemented for this network. The simplification is beneficial for reconfigurable hardware systems, keeping generality and accuracy. Furthermore, slight modifications would allow to be used with Address-Event Representation (AER) data protocol for frameless vision . The proposed system could be further implemented in FPGAs for low-power embedded neural computation.
- Lovelace JJ, Rickard JT, Cios KJ. A spiking neural network alternative for the analog to digital converter. In: Neural Networks (IJCNN), The 2010 International Joint Conference On. New Jersey, USA: Institute of Electrical and Electronics Engineers-IEEE: 2010. p. 1–8.Google Scholar
- Ambard M, Guo B, Martinez D, Bermak A. A spiking neural network for gas discrimination using a tin oxide sensor array. In: Electronic Design, Test and Applications, 2008. DELTA 2008. 4th IEEE International Symposium On. New Jersey, USA: Institute of Electrical and Electronics Engineers-IEEE: 2008. p. 394–397.Google Scholar
- Bouganis A, Shanahan M. Training a spiking neural network to control a 4-dof robotic arm based on spike timing-dependent plasticity. In: Neural Networks (IJCNN), The 2010 International Joint Conference On. New Jersey, USA: Institute of Electrical and Electronics Engineers-IEEE: 2010. p. 1–8.Google Scholar
- Alnajjar F, Murase K. Sensor-fusion in spiking neural network that generates autonomous behavior in real mobile robot. In: Neural Networks, 2008. IJCNN 2008. (IEEE World Congress on Computational Intelligence). IEEE International Joint Conference On. New Jersey, USA: Institute of Electrical and Electronics Engineers-IEEE: 2008. p. 2200–2206.Google Scholar
- Perez-Carrasco JA, Acha B, Serrano C, Camunas-Mesa L, Serrano-Gotarredona T, Linares-Barranco B. Fast vision through frameless event-based sensing and convolutional processing: Application to texture recognition. Neural Networks IEEE Trans. 2010; 21(4):609–620.View ArticleGoogle Scholar
- Botzheim J, Obo T, Kubota N. Human gesture recognition for robot partners by spiking neural network and classification learning. In: Soft Computing and Intelligent Systems (SCIS) and 13th International Symposium on Advanced Intelligent Systems (ISIS), 2012 Joint 6th International Conference On. New Jersey, USA: Institute of Electrical and Electronics Engineers-IEEE: 2012. p. 1954–1958.Google Scholar
- Ratnasingam S, McGinnity TM. A spiking neural network for tactile form based object recognition. In: The 2011 International Joint Conference on Neural Networks (IJCNN). New Jersey, USA: Institute of Electrical and Electronics Engineers-IEEE: 2011. p. 880–885.Google Scholar
- Fang H, Wang Y, He J. Spiking neural networks for cortical neuronal spike train decoding. Neural Comput. 2009; 22(4):1060–1085.View ArticleGoogle Scholar
- Gerstner W, Kistler WM. Spiking Neuron Models: Single Neurons, Populations, Plasticity. Cambridge, United Kingdom: Cambridge University Press; 2002, p. 494.View ArticleGoogle Scholar
- Arguello E, Silva R, Castillo C, Huerta M. The neuroid: A novel and simplified neuron-model. In: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). New Jersey, USA: Institute of Electrical and Electronics Engineers-IEEE: 2012. p. 1234–1237.Google Scholar
- Ishikawa Y, Fukai S. A neuron mos variable logic circuit with the simplified circuit structure. In: Proceedings of 2004 IEEE Asia-Pacific Conference on Advanced System Integrated Circuits 2004. New Jersey, USA: Institute of Electrical and Electronics Engineers-IEEE: 2004. p. 436–437.Google Scholar
- Lorenzo R, Riccardo R, Antonio C. A new unsupervised neural network for pattern recognition with spiking neurons. In: International Joint Conference on Neural Networks, 2006. IJCNN 06. New Jersey, USA: Institute of Electrical and Electronics Engineers-IEEE: 2006. p. 3903–3910.Google Scholar
- Painkras E, Plana LA, Garside J, Temple S, Galluppi F, Patterson C, Lester DR, Brown AD, Furber SB. Spinnaker: A 1-w 18-core system-on-chip for massively-parallel neural network simulation. IEEE J. Solid-State Circuits. 2013; 48(8):1943–1953.View ArticleGoogle Scholar
- Schemmel J, Grubl A, Hartmann S, Kononov A, Mayr C, Meier K, Millner S, Partzsch J, Schiefer S, Scholze S, et al.Live demonstration: A scaled-down version of the brainscales wafer-scale neuromorphic system. In: 2012 IEEE International Symposium on Circuits and Systems (ISCAS). New Jersey, USA: Institute of Electrical and Electronics Engineers-IEEE: 2012. p. 702–702.Google Scholar
- Hylton T. 2008. Systems of neuromorphic adaptive plastic scalable electronics. http://www.scribd.com/doc/76634068/Darpa-Baa-Synapse.Google Scholar
- Schoenauer T, Atasoy S, Mehrtash N, Klar H. Neuropipe-chip: a digital neuro-processor for spiking neural networks. Neural Networks, IEEE Trans. 2002; 13(1):205–213.View ArticleGoogle Scholar
- Schrauwen B, Campenhout JV. Parallel hardware implementation of a broad class of spiking neurons using serial arithmetic. In: Proceedings of the 14th European Symposium on Artificial Neural Networks. Evere, Belgium: D-side conference services: 2006. p. 623–628.Google Scholar
- Rice KL, Bhuiyan MA, Taha TM, Vutsinas CN, Smith MC. Fpga implementation of izhikevich spiking neural networks for character recognition. In: International Conference on Reconfigurable Computing and FPGAs, 2009. ReConFig 09. New Jersey, USA: Institute of Electrical and Electronics Engineers-IEEE: 2009. p. 451–456.Google Scholar
- Xilinx. Spartan-6 family overview. Technical Report DS160, Xilinx, Inc. October 2011. http://www.xilinx.com/support/documentation/data_sheets/ds160.pdf.
- Maass W. Networks of spiking neurons: The third generation of neural network models. Neural Networks. 1997; 10(9):1659–1671.View ArticleGoogle Scholar
- MCWV Rossum, Bi GQ, Turrigiano GG. Stable hebbian learning from spike timing-dependent plasticity. J. Neurosci. 2000; 20(23):8812–8821.Google Scholar
- Pham DT, Packianather MS, Charles EYA. A self-organising spiking neural network trained using delay adaptation. In: Industrial Electronics, 2007. ISIE 2007. IEEE International Symposium On. New Jersey, USA: Institute of Electrical and Electronics Engineers-IEEE: 2007. p. 3441–3446.Google Scholar
- Paugam-Moisy H, SM Bohte. Computing with Spiking Neuron Networks In: G Rozenberg, JK T Back, editors. Handbook of Natural Computing. Heidelberg, Germany: Springer: 2009.Google Scholar
- Booij O. Temporal pattern classification using spiking neural networks. (August 2004). Available from http://obooij.home.xs4all.nl/study/download/booij04Temporal.pdf.
- Bi G-Q, Poo M-M. Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci. 1998; 18(24):10464.Google Scholar
- Martinez LM, Alonso J-M. Complex receptive fields in primary visual cortex. Neuroscientist: Rev. J Bringing Neurobiology, Neurology Psychiatry. 2003; 9(5):317–331. PMID: 14580117.View ArticleGoogle Scholar
- Foldiak P, Young MP. The Handbook of Brain Theory and Neural Networks. Cambridge, MA, USA: MIT Press; 1998, pp. 895–898. http://dl.acm.org/citation.cfm?id=303568.303958.Google Scholar
- Repository UML. Semeion Handwritten Digit Dataset. 2014. http://archive.ics.uci.edu/ml/datasets/Semeion+Handwritten+Digit Accessed 2014-10-30.
- Perez-Carrasco JA, Zhao B, Serrano C, Acha B, Serrano-Gotarredona T, Chen S, Linares-Barranco B. Mapping from frame-driven to frame-free event-driven vision systems by low-rate rate coding and coincidence processing–application to feedforward convnets. Pattern Anal. Machine Intelligence, IEEE Trans. 2013; 35(11):2706–2719.View ArticleGoogle Scholar
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.