In the past decade, machine learning technology based on deep neural networks has made great progress, thanks to the continuous development of HPC hardware and software and practical applications. There are already a number of organizations and enterprises offering services to the public based on machine learning systems, such as face and speech recognition, photo optimization, and so on. Deep neural networks also require different computational power, so the demand for distributed neural network systems is also increasing. This paper mainly studies the collaborative optimization method of distributed systems based on deep learning. In this paper, a new distributed deep learning training system is designed and implemented by effectively combining cluster resource scheduling and distributed training based on the advantages of Pytorch in rapid neural network construction and parallel computing and cluster resource scheduling.

Shaam Daparvar. Collaborative Optimization Method for Distributed Systems based on Deep Learning. Distributed Processing System (2020), Vol. 1, Issue 1: 34-41. https://doi.org/10.38007/DPS.2020.010105.

[1] Chagraoui H, Soula M. Multidisciplinary collaborative optimization based on relaxation method for solving complex problems:. Concurrent Engineering, 2020, 28(4):280-289. https://doi.org/10.1177/1063293X20958921

[2] Ines A, Joao X. DJAM: distributed Jacobi asynchronous method for learning personal models. IEEE Signal Processing Letters, 2018, PP(9):1-1.

[3] Mao W, Feng W, Liang X. A novel deep output kernel learning method for bearing fault structural diagnosis. Mechanical Systems and Signal Processing, 2018, 117(FEB.15):293-318. https://doi.org/10.1016/j.ymssp.2018.07.034

[4] Sattler F, Muller K R, Samek W. Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints. IEEE Transactions on Neural Networks and Learning Systems, 2020, PP(99):1-13.

[5] Rabhi S, Semchedine F, Mbarek N. An Improved Method for Distributed Localization in WSNs Based on Fruit Fly Optimization Algorithm. Automatic Control and Computer Sciences, 2020, 55(3):1-16. https://doi.org/10.3103/S0146411621030081

[6] Kavousi-Fard A, Dabbaghjamanesh M, Jin T, et al. An Evolutionary Deep Learning-Based Anomaly Detection Model for Securing Vehicles. IEEE Transactions on Intelligent Transportation Systems, 2020, PP(99):1-9.

[7] Saravanan G, Yuvaraj N. Cloud resource optimization based on poisson linear deep gradient learning for mobile cloud computing. Journal of Intelligent and Fuzzy Systems, 2020, 40(1):1-11. https://doi.org/10.3233/JIFS-200799

[8] Hamdia K M, Ghasemi H, Bazi Y, et al. A novel deep learning based method for the computational material design of flexoelectric nanostructures with topology optimization. Finite Elements in Analysis and Design, 2019, 165(Nov.):21-30.

[9] Ko H, Pack S. Distributed Device-to-Device Offloading System: Design and Performance Optimization. IEEE Transactions on Mobile Computing, 2020, PP(99):1-1.

[10] Samala R K, Kotapuri M R. Distributed Generation Allocation in Distribution System using Particle Swarm Optimization based Ant-Lion Optimization. International Journal of Control and Automation, 2020, 13(1):414-426.

[11] Tabasi M, Asgharian P. Optimal operation of energy storage units in distributed system using social spider optimization algorithm. AIMS Electronics and Electrical Engineering, 2019, 3(4):309-327.

[12] Sameti M, Haghighat F. Optimization of 4th generation distributed district heating system: Design and planning of combined heat and power. Renewable Energy, 2018, 130(JAN.):371-387. https://doi.org/10.1016/j.renene.2018.06.068

[13] Giri S, Mondal A K, Bera P. Design of PIDD controller for Hybrid Distributed Generation System using Social Spider Optimization Algorithm. Indian Science Cruiser, 2019, 33(3):27. https://doi.org/10.24906/isc/2019/v33/i3/185420

[14] Altsybeyev V, Kozynchenko V. Development of the distributed information system for the cooperative work under the design and optimizationof charged particle accelerators. Cybernetics and Physics, 2019(Volume 8, 2019, Number 4):195-198.

[15] Mughees M, Awan F G, Mughees A. Volt/VAr Optimization of Distribution System with Integrated Distributed Generation. Mehran University Research Journal of Engineering and Technology, 2017, 36(1):117-128. https://doi.org/10.22581/muet1982.1701.11

[16] Tabasi M, Asgharian P. Optimal operation of energy storage units in distributed system using social spider optimization algorithm. International Journal on Electrical Engineering and Informatics, 2019, 11(3):564-579. https://doi.org/10.15676/ijeei.2019.11.3.8

[17] Baran S, Nima S, Reza A, et al. Optimization of synchronized frequency and voltage control for a distributed generation system using the Black Widow Optimization algorithm. Clean Energy, 2020(1):1.

[18] Nawaz M A, Raheem A, Shakoor R, et al. feasibility and optimization of standalone pv-biogas hybrid distributed renewable system for rural electrification: a case study of a cholistan community. Mehran University Research Journal of Engineering and Technology, 2019, 38(2):453-462. https://doi.org/10.22581/muet1982.1902.19