DS4NN: Direct training of deep spiking neural networks with single spike-based temporal coding

Document Type : Research Article

Authors

1 Department of Electrical Engineering, Amirkabir University of Technology, Tehran, Iran

2 Faculty of Mathematical sciences, Shahid Beheshti University, Tehran, Iran

Abstract

Backpropagation is the foremost prevalent and common algorithm for training conventional neural networks with deep construction. Here we propose DS4NN, temporal backpropagation for deep spiking neural networks with one spike per neuron. We consider a convolutional spiking neural network consisting of simple non-leaky integrate-and-fire (IF) neurons, and a form of coding named time-to-first-spike temporal coding in which, neurons are allowed to fire at most once in a specific time interval, which corresponds to simulation duration here. These features together improve the cost and the speed of network computation. We use a surrogate gradient at firing times to solve the non-differentiability of spike times concerning the membrane potential of spiking neurons, and to prevent the emergence of dead neurons in deep layers, we propose a relative encoding scheme for determining desired firing times. Evaluations on two classification tasks of MNIST and Fashion-MNIST datasets confirm the capability of DS4NN on the deep structure of SNNs. It achieves the accuracy of 99.3% (99.8%) and 91.6% (95.3%) on testing samples (training samples) of respectively MNIST and Fashion-MNIST datasets with the mean required number of 1126 and 1863 spikes in the whole network. This shows that the proposed approach can make fast decisions with low-cost computation and high accuracy.

Keywords

Main Subjects

References

[1] I. Goodfellow, Y. Bengio, A. Courville, Deep learning, MIT press, 2016.
[2] B. Rueckauer, S.C. Liu, Conversion of analog to spiking neural networks using sparse temporal coding, in:  2018 IEEE International Symposium on Circuits and Systems (ISCAS), 2018, pp. 1-5.
[3] N. Rathi, G. Srinivasan, P. Panda, K. Roy, Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation, arXiv preprint arXiv:2005.01807,  (2020).
[4] A. Sengupta, Y. Ye, R. Wang, C. Liu, K. Roy, Going Deeper in Spiking Neural Networks: VGG and Residual Architectures, Frontiers in Neuroscience, 13 (2019).
[5] C. Lee, S.S. Sarwar, P. Panda, G. Srinivasan, K. Roy, Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures, Frontiers in Neuroscience, 14 (2020).
[6] S. Deng, S. Gu, Optimal conversion of conventional artificial neural networks to spiking neural networks, arXiv preprint arXiv:2103.00476,  (2021).
[7] J. Allred, K. Roy, L4-Norm Weight Adjustments for Converted Spiking Neural Networks, arXiv preprint arXiv:2111.09446,  (2021).
[8] J. Wu, Y. Chua, M. Zhang, G. Li, H. Li, K.C. Tan, A Tandem Learning Rule for Effective Training and Rapid Inference of Deep Spiking Neural Networks, IEEE Transactions on Neural Networks and Learning Systems,  (2021) 1-15.
[9] J. Wu, C. Xu, X. Han, D. Zhou, M. Zhang, H. Li, K.C. Tan, Progressive Tandem Learning for Pattern Recognition With Deep Spiking Neural Networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11) (2022) 7824-7840.
[10] S.R. Kheradpisheh, M. Mirsadeghi, T. Masquelier, Spiking Neural Networks Trained via Proxy, IEEE Access, 10 (2022) 70769-70778.
[11] M. Zhang, J. Wang, J. Wu, A. Belatreche, B. Amornpaisannon, Z. Zhang, V.P.K. Miriyala, H. Qu, Y. Chua, T.E. Carlson, H. Li, Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks, IEEE Transactions on Neural Networks and Learning Systems, 33(5) (2022) 1947-1958.
[12] S.R. Kheradpisheh, T. Masquelier, Temporal backpropagation for spiking neural networks with one spike per neuron, International Journal of Neural Systems, 30(06) (2020) 2050027.
[13] M. Mirsadeghi, M. Shalchian, S.R. Kheradpisheh, T. Masquelier, STiDi-BP: Spike time displacement based error backpropagation in multilayer spiking neural networks, Neurocomputing, 427 (2021) 131-140.
[14] M. Mirsadeghi, M. Shalchian, S.R. Kheradpisheh, T. Masquelier, Spike time displacement based error backpropagation in convolutional spiking neural networks, arXiv preprint arXiv:2108.13621,  (2021).
[15] S.R. Kheradpisheh, M. Ganjtabesh, S.J. Thorpe, T. Masquelier, STDP-based spiking deep convolutional neural networks for object recognition, Neural Networks, 99 (2018) 56-67.
[16] M. Mozafari, S.R. Kheradpisheh, T. Masquelier, A. Nowzari-Dalini, M. Ganjtabesh, First-Spike-Based Visual Categorization Using Reward-Modulated STDP, IEEE Transactions on Neural Networks and Learning Systems, 29(12) (2018) 6178-6190.
[17] T. Masquelier, S.J. Thorpe, Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity, PLOS Computational Biology, 3(2) (2007) e31.
[18] M. Mozafari, M. Ganjtabesh, A. Nowzari-Dalini, S.J. Thorpe, T. Masquelier, Bio-inspired digit recognition using reward-modulated spike-timing-dependent plasticity in deep convolutional networks, Pattern Recognition, 94 (2019) 87-95.
[19] M. Mozafari, M. Ganjtabesh, A. Nowzari-Dalini, T. Masquelier, SpykeTorch: Efficient Simulation of Convolutional Spiking Neural Networks With at Most One Spike per Neuron, Frontiers in Neuroscience, 13 (2019).
[20] S.R. Kheradpisheh, M. Ganjtabesh, T. Masquelier, Bio-inspired unsupervised learning of visual features leads to robust invariant object recognition, Neurocomputing, 205 (2016) 382-392.
[21] J. Göltz, L. Kriener, A. Baumbach, S. Billaudelle, O. Breitwieser, B. Cramer, D. Dold, A.F. Kungl, W. Senn, J. Schemmel, Fast and energy-efficient neuromorphic deep learning with first-spike times, Nature machine intelligence, 3(9) (2021) 823-835.
[22] R. Vaila, J. Chiasson, V. Saxena, Feature extraction using spiking convolutional neural networks, in:  Proceedings of the International Conference on Neuromorphic Systems, 2019, pp. 1-8.
[23] S.M. Bohte, J.N. Kok, H. La Poutré, Error-backpropagation in temporally encoded networks of spiking neurons, Neurocomputing, 48(1) (2002) 17-37.
[24] H. Mostafa, Supervised Learning Based on Temporal Coding in Spiking Neural Networks, IEEE Transactions on Neural Networks and Learning Systems, 29(7) (2018) 3227-3235.
[25] I.M. Comsa, K. Potempa, L. Versari, T. Fischbacher, A. Gesmundo, J. Alakuijala, Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function, in:  ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 8529-8533.
[26] S.B.a.O. Shrestha, Garrick, SLAYER: Spike Layer Error Reassignment in Time, in:  Advances in Neural Information Processing Systems, Curran Associates, Inc., 2018.
[27] Y. Wu, L. Deng, G. Li, J. Zhu, L. Shi, Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks, Frontiers in Neuroscience, 12 (2018).
[28] D.a.S. Huh, Terrence J, Gradient Descent for Spiking Neural Networks, in: Garnett (Ed.) Advances in Neural Information Processing Systems, Curran Associates, Inc., 2018.
[29] L. Zuo, F. Xu, C. Zhang, T. Xiahou, Y. Liu, A multi-layer spiking neural network-based approach to bearing fault diagnosis, Reliability Engineering & System Safety, 225 (2022) 108561.
[30] Q. Meng, M. Xiao, S. Yan, Y. Wang, Z. Lin, Z.-Q. Luo, Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation, in, 2022, pp. arXiv:2205.00459.
[31] B. Gardner, A. Grüning, Supervised Learning With First-to-Spike Decoding in Multilayer Spiking Neural Networks, Frontiers in Computational Neuroscience, 15 (2021).
[32] D.P. Kingma, J. Ba, Adam: A Method for Stochastic Optimization, in, 2014, pp. arXiv:1412.6980.
[33] E.O. Neftci, H. Mostafa, F. Zenke, Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks, IEEE Signal Processing Magazine, 36(6) (2019) 51-63.
[34] G. Bellec, D. Salaj, A. Subramoney, R. Legenstein, W. Maass, Long short-term memory and learning-to-learn in networks of spiking neurons, Advances in neural information processing systems, 31 (2018).
[35] F. Zenke, S. Ganguli, SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks, Neural Computation, 30(6) (2018) 1514-1541.
[36] J. Brownlee, A gentle introduction to object recognition with deep learning, Machine Learning Mastery, 5 (2019).
[37] H. Xiao, K. Rasul, R. Vollgraf, Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms, in, 2017, pp. arXiv:1708.07747.
[38] S.R. Kheradpisheh, M. Mirsadeghi, T. Masquelier, BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning, Neural Processing Letters, 54(2) (2022) 1255-1273.
AUT Journal of Electrical Engineering
Volume 55, Issue 2
2023
Pages 179-190

Files

Share

How to cite

Statistics