Обзор методов оптимизации бинарных нейронных сетей
https://doi.org/10.17586/0021-3454-2023-66-11-926-935
Аннотация
Развертывание моделей сверточных нейронных сетей (СНС) во встраиваемых системах ос ложнено множеством проблем, связанных с вычислительной мощностью, энергопотреблением и объемом памя ти. Для решения этих проблем в 2016 г. создан многообещающий тип нейронных сетей, использующих 1-битную активацию и веса, — бинарные нейронные сети (БНС). Такие сети потребляют меньше энергии и вычислитель ных мощностей, так как заменяют сложную операцию тяжелой свертки простыми побитовыми операциями. Одна ко квантование с 32-разрядной плавающей запятой до 1 бита приводит к потере точности и снижению производи тельности, особенно при больших наборах данных. Представлен обзор ключевых методов оптимизации, которые повлияли на производительность БНС и привели к повышению репрезентативности их моделей, также представ лены обзор способов применения БНС в задачах обнаружения объектов и сравнительный анализ их производи тельности с реальным значением.
Об авторе
А. Шаккуф
Университет ИТМО
Россия
Россия
Али Шаккуф, аспирант; факультет систем управления и робототехники
Санкт-Петербург
Список литературы
1. Basha S. S., Dubey S. R., Pulabaigari V. and Mukherjee S. Impact of fully connected layers on performance of convolutional neural networks for image classification // Neurocomputing. 2020. Vol. 378. P. 112—119.
2. Zhou W., Wang H. and Wan Z. Ore image classification based on improved CNN // Computers and Electrical Engineering. 2022. Vol. 99, art. N 107819.
3. Gao X., Xing G., Roy S., and LiuH. RAMP-CNN: A novel neural network for enhanced automotive radar object recognition // IEEE Sensors J. 2021. Vol. 21, N 4. P. 5119—5132.
4. Ashiq F., Asif M., Ahmad M. B., Zafar S., Masood K., Mahmood T., Mahmood M. T. and LeeI. H. CNN-based object recognition and tracking system to assist visually impaired people // IEEE Access. 2022. Vol. 10. P. 14819—14834.
5. Abdelhamid A. A., El-Kenawy E.-S.-M., Alotaibi B., Amer G. M., Abdelkader M. Y., Ibrahim A. and Eid M. M. Robust speech emotion recognition using CNN+LSTM based on stochastic fractal search optimization algorithm // IEEE Access. 2022. Vol. 10. P. 49265—49284.
6. Kwon S. MLT-DNet: Speech emotion recognition using 1D dilated CNN based on multi-learning trick approach // Expert Systems with Applications. 2021. Vol. 167, art. N 114177.
7. Zhang N., Wei X., Chen H. and Liu W. FPGA implementation for CNN-based optical remote sensing object detection // Electronics. 2021. Vol. 10, N 3. P. 282.
8. Lopac N., Hrzic F., Vuksanovic I. P. and Lerga J. Detection of non-stationary GW signals in high noise from Cohen’s class of time–frequency representations using deep learning // IEEE Access. 2022. Vol. 10. P. 2408—2428.
9. Horowitz M. Computing’s Energy Problem (and what we can do about it) // IEEE Intern. Solid State Circuits Conf. 2014. P. 10—14.
10. Courbariaux M. and BengioY. BinaryNet: Training deep neural networks with weights and activations constrained to +1 or -1 // ArXiv Journal. 2016. preprint arXiv: 1602.02830.
11. Wang E., Davis J. J., Moro D., Zielinski P., Lim J. J., Coelho C., Chatterjee S., Cheung P. Y., Constantinides G. A. Enabling binary neural network training on the edge // 5th Intern. Workshop on Embedded and Mobile Deep Learning. 2021. June. P. 37—38.
12. Sohoni N. S., Aberger C. R., Leszczynski M., Zhang J. and CRé. Low-memory neural network training: A technical report // ArXiv journal. 2019. preprint arXiv:1904.10631.
13. Courbariaux M., Bengio Y. and Jean-Pierre D. Binaryconnect: Training deep neural networks with binary weights during propagations // ArXiv e-prints. 2015. abs/1511.00363.
14. Hinton G. Neural networks for machine learning // Coursera (video lectures). 2012.
15. Bengio Yo. Estimating or propagating gradients through stochastic neurons // Technical Report arXiv:1305.2982. Universite de Montreal. 2013.
16. Darabi S., Belbahri M., Courbariaux M. and Nia V. P. BNN+: Improved binary network training. 2019. https://openreview.net/pdf?id=SJfHg2A5tQ.
17. Shen M., Han K., Xu C., Wang Y. Searching for accurate binary neural architectures // IEEE/CVF Intern. Conf. on Computer Vision Workshops. 2019.
18. Wang Z., Lu J., Tao C., Zhou J., Tian Q. Learning channel-wise interactions for binary convolutional neural networks // IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 2019. P. 568.
19. Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network // NIPS. 2017. P. 344—352.
20. Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm // arXiv preprint. 2018. arXiv:1808.00278.
21. Shen M., Liu X., Gong R., Han K. Balanced binary neural networks with gated residual // IEEE Intern. Conf. on Acoustics, Speech and Signal Processing (ICASSP) 2020. 4197.
22. Jianhao Zhang, Yingwei Pan, Ting Yao, He Zhao and Tao Mei. dabnn: A super fast inference framework for binary neural networks on arm devices // arXiv preprint. 2019 arXiv:1908.05858.
23. Cai Z.; He X.; Sun J. and Vasconcelos N. Deep learning with low precision by half-wave gaussian quantization // IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2017.
24. Ioffe S. and Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift // Proc. of the 32nd Intern. Conf. on Machine Learning, ICML. 2015. P. 448—456.
25. Wang P., He X., Li G., Zhao T., Cheng J. Sparsity-inducing binarized neural networks // AAAI Conf. on Artificial Intelligence. 2020. N 34. P. 12192.
26. Yang J., Shen X., Xing J., Tian X., Li H., Deng B., Huang J. and Hua X.-s. Quantization networks // IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2019.
27. Zhu S., Dong X. and Su H. Binary ensemble neural network: More bits per network or more networks per bit? // IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2019.
28. Anderson A. G. and Berg C. P. The High-Dimensional Geometry of Binary Neural Networks // ArXiv Journal. 2017. abs/1705.07199.
29. Qin Haotong, Gong Ruihao, Liu Xianglong, Bai Xiao, Song Jingkuan, Sebe Nicu. Binary Neural Networks: A Survey // Pattern Recognition. 2020. N 105. 107281. DOI: 10.1016/ j.patcog.107281.
30. Liang T., Glossner J., Wang L., Shi S. and Zhang X. Pruning and quantization for deep neural network acceleration: A survey // Neurocomputing. 2021. Vol. 461. P. 370—403.
31. Xu Sheng, Liu Chang, Zhang Baochang, Lu Jinhu, Guo Guodong, Doermann D. BiRe-ID: Binary Neural Network for Efficient Person Re-ID // ACM Trans. on Multimedia Computing, Communications, and Applications. 2022. N 18. DOI: 10.1145/3473340.
32. Zhang K., Yin M. and Wang Y. Why Quantization Improves Generalization: NTK of Binary Weight Neural Networks // ArXiv Journal. 2022. abs/2206.05916.
33. Darabi S., Belbahri M., Courbariaux M. and Nia V. P. Regularized binary network training // ArXiv Journal. 2018, 1812.11800.
34. Wang S., Zhang C., Su D., Wang L., Jiang H. High-precision binary object detector based on a bsf-xnor convolutional layer // IEEE Access 9. 2021. P. 106169.
35. Rastegari M., Ordonez V., Redmon J. and Farhadi A. Xnor-net: Imagenet classification using binary convolutional neural networks // ECCV. 2016. P. 525— 542.
36. Barry D., Shah M., Keijsers M., Khan H. and Hopman B. XYOLO: A model for real-time object detection in humanoid soccer on low-end hardware // ArXiv Journal: 1910.03159. 2019.
37. Chen Hanlin, Zhuo Li'an, Zhang Baochang, Zheng Xiawu, Liu Jianzhuang, Ji Rongrong, Doermann D., Guo Guodong. Binarized Neural Architecture Search for Efficient Object Recognition // ArXiv Journal. 2020.
38. Kung Jaeha, Zhang David, van der Wal G. Chai, Sek Mukhopadhyay S. Efficient Object Detection Using Embedded Binarized Neural Networks // Journal of Signal Processing Systems. 2018. N 90. P. 1—14. DOI: 10.1007/s11265 017-1255-5.
39. Wang Xingang, Siyang Sun, Yin Yingjie, Xu De, Wu Wenqi, Gu Qingyi. Fast Object Detection Based on Binary Deep Convolution Neural Networks // CAAI Trans. on Intelligence Technology. 2018. N 3. DOI: 10.1049/trit.1026.
40. Kim H., Choi K. The implementation of a power efficient BCNN based object detection acceleration on a Xilinx FPGA-SoC // Intern. Conf. Internet Things (iThings) IEEE Green Comput. Commun. (GreenCom) IEEE Cyber, Phys. Social Comput. (CPSCom) IEEE Smart Data (SmartData). 2019. P. 240—243.
41. Peng H., Chen S. BDNN: Binary convolution neural networks for fast object detection // Pattern Recognition Lett. 2019. Vol. 125. P. 91—97.
42. Ojeda F. C., Bisulco A., Kepple D., Isler V. and Lee D. D. On-device event filtering with binary neural networks for pedestrian detection using neuromorphic vision sensors // IEEE Intern. Conf. Image Process. (ICIP) 2020. P. 3084—3088.
43. Wang Z., Wu Z., Lu J. and Zhou J. BiDet: An efficient binarized object detector // IEEE/CVF Conf. Comput. Vis. Pattern Recognition. 2020. P. 2049—2058.
44. Xu S., Zhao J., Lu J., Zhang B., Han S. and Doermann D. Layer-wise searching for 1-bit detectors // IEEE/CVF Conf. Comput. Vis. Pattern Recognition. (CVPR). 2021. P. 5682—5691.
45. Zhao J., Xu S., Wang R., Zhang B., Guo G., Doermann D. and Sun D. Data-adaptive binary neural networks for efficient object detection and recognition // Pattern Recognition. Lett. 2022. Vol. 153. P. 239—245.
46. Mani V. R. S., Saravanaselvan A. and Arumugam N. Performance comparison of CNN, QNN and BNN deep neural networks for real-time object detection using Zynq FPGA node // Microelectron. 2022. Vol. 119, art. N 105319.
47. Pérez-Hernández F., Tabik S., Lamas A., Olmos R., Fujita H., Herrera F. Object detection binary classifiers methodology based on deep learning to identify small objects handled similarly: application in video surveillance // Knowl. Base Syst. 2020. N 194 P. 105590.
48. Frickenstein A., Vemparala M.-R., Mayr J., Nagaraja N.-S., Unger C., TombariF. and StecheleW. Binary DAD-Net: Binarized driveable area detection network for autonomous driving // IEEE Intern. Conf. Robot. Autom (ICRA). 2020. P. 2295—2301.
49. Ajay B. S., MRao. Binary neural network based real time emotion detection on an edge computing device to detect passenger anomaly // 34th Intern. Conf. VLSI Design, 20th Intern. Conf. Embedded Syst. (VLSID). 2021. P. 175—180.
50. Zhuang B., Shen C., Tan M., Liu L. and Reid I. Structured binary neural networks for accurate image classification and semantic segmentation // IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). 2019. P. 413—422.
Рецензия
Для цитирования:
Шаккуф А. Обзор методов оптимизации бинарных нейронных сетей. Известия высших учебных заведений. Приборостроение. 2023;66(11):926-935. https://doi.org/10.17586/0021-3454-2023-66-11-926-935
For citation:
Shakkouf A. Review on Optimization Techniques of Binary Neural Networks. Journal of Instrument Engineering. 2023;66(11):926-935. https://doi.org/10.17586/0021-3454-2023-66-11-926-935
Просмотров:
25
Послать статью по эл. почте 