- Open Access
Research on multi-robot scheduling algorithms based on machine vision
© The Author(s). 2018
- Received: 26 July 2018
- Accepted: 9 October 2018
- Published: 4 December 2018
In the multi-robot system, how to achieve effective and reasonable task coordination between multi-robots is an important problem;, multi-robot task scheduling is the term used for the coordination of the key technologies. Therefore, in this paper we combined the pilot scheduling method with the following method and the behavior method of the robot based on task scheduling, and we then studied how to improve the traditional robot scheduling effect, which is a deep-learning algorithm that is applied to multi-robot scheduling to formulate an action selection strategy. We thus proved the effectiveness of this idea experimentally. Based on the above research foundation, this paper continues to build a simple simulation experiment platform, which simply sets up three obstacles and completes the task of robot scheduling on the platform.
- Deep learning
- Convolutional neural network
Since the 1980s, researchers have been working on transforming classical scheduling theory into practical production and application. Artificial intelligence, computer intelligence and other intelligent scheduling methods have been applied to solve practical production-scheduling problems. Based on the existing technological conditions and research results, we can see that the intelligent scheduling method is currently one of the best and most promising ways to solve practical production-scheduling problems. With the development of related research and its practical application, the intelligent scheduling method to solve the actual production problem will continue to make progress. Advanced manufacturing systems often use modern industrial-robot computer control for each station asked to handle materials. The scheduling problem of the multi-robot is the extension of the classical job shop problem; it is the traditional job shop scheduling problem but more complex because it should not only consider the workpiece processing arrangements, but also consider the order handling and handling-robot task allocation.
Because the visual sensor has the characteristics of acquiring large amounts of information rapidly with information integrity, the visual sensor has become an important part of robot intelligence control. With the visual technology used in robot technology, the robot can not only have greatly improved ability to adapt to the surrounding environment, but also the visual sensor has other sensor-incomparable advantages due to the amount of information obtained being very large, so the robot can, within the external environment, enable objects and people to work together to complete more complex tasks. As with human vision, machine vision means that the robot has this most important sense, but the key problem in the field of machine vision is the image-processing speed, but the field also has been plagued by visual technology development, the key to solving this problem. On the other hand, with the development of robot technology, people’s demand for robots is no longer limited to single robots, but more attention is being transferred to multiple robot systems. This is not only because the work is more than a single robot can manage and, for some dynamic and complex tasks, the development of a single robot rather than the development of a multi-robot system is complicated and expensive, but the system used, a single complex machine can beer complete the task than a simple robot.
The optimization of production scheduling is the core of advanced manufacturing technology and modern management technology. Many industrial fields, such as electronic products and metallurgy, have scheduling-optimization problems. The multi-robot scheduling problem is essentially a resource-allocation, optimization, path-selection problem, i.e., how fast can it be achieved? The problem is of maximizing the resources in an effective timeframe, including assigning tasks, under the constraints of the process route and delivery deadline, and achieving multi-objective planning problems in an optimal manner. Small-scale, multi-variety, production orders, with considerable operation process information are complex, and production planning and operation planning are difficult to balance. It is also difficult to guarantee delivery deadlines and quality, which directly affects the economic benefits of enterprises that need to use job-scheduling optimization methods to achieve increased productivity and reduced costs.
In nature, organisms tend to acquire more advanced functions in groups, and multi-robot systems are based on the simulation of populations and biotas. These studies are gradually forming collaborative robotics, which is based on the concept of multi-agents and the issue of cooperation between multiple robots is receiving increasing attention. To organize and control multiple robots based on the characteristics of multiple agents enables them to perform complex tasks that a single robot cannot perform. The intelligent behavior of a system composed of multiple robots will be greatly improved compared with a single robot, mainly because the information exchange between the robots improves the ability of the system to deal with incomplete and uncertain knowledge. The cooperation between the robots will mean that a system with a certain intelligent behavior can use object-oriented technology and modular ideas to organize the system. Therefore, research on multi-robot systems is an inevitable trend in the development of multi-robot technology, which will bring about tremendous changes in robotics and other related technologies.
At present, there is no unified multi-robot system definition, which is usually defined as an organic dynamic whole composed of several members for completing one or more tasks through certain communication methods, organizational structures and cooperation mechanisms. Designing a multi-functional robot is a complex project, but designing a multi-robot system, consisting of multiple limited-function robots to accomplish the same task, is relatively easy. The multi-robot system consists of a relatively simple or single robot that, through coordination with other members, accomplishes tasks that were previously impossible. Because the task is divided into several parts and executed by the robot separately, real parallel execution can be realized, so in some specific types of tasks, the multi-robot system can complete the task more efficiently, which has great application potential in the fields of production, daily life, the Military, etc. Although multi-robot systems have unique advantages, the addition of multiple robots poses additional problems, such as the difficulty of configuring or deploying these systems, and further research on new issues, such as collaboration, is required. Although research on multi-robot systems has attracted widespread attention over the past decade, research in this area is still far from mature. The more successful work of multi-robot research includes cell robot systems or configurable robot systems [1–5]; multi-robot motion planning [6–10]; and multi-robot collaborative architecture [11–15].
The research on the multi-robot system in China began in the 1990s, and the development of the system was also restricted. However, in recent years, more and more experts have attached importance to research into multi-robot systems in China, and many universities have set up laboratories, invested a lot of research funds, and began to study multi-robot technology. A multi-robot cooperative assembly system has been established in Shenyang Automation Research of Chinese Academy of Sciences. The system is based on industrial multi-robot assembly and has become a major platform for multi-robot collaborative research in the industrial field, and has been greatly welcomed. Shanghai Jiaotong University robot soccer team is also a successful multi-robot platform. It has made outstanding achievements in recent years in important competitions both at home and abroad, and has won many championships. In addition, Harbin Institute of Technology has successfully developed a robot soccer game platform based on a multi-agent system. Based on that, a small robot force model has been built, and the robot team made by it has achieved good results in robot soccer matches at home and abroad.
In multi-robot systems, how to effectively and reasonably coordinate tasks among multiple robots is an important problem. The scheduling of multiple robots is one of the key technologies of this task coordination. In the process of scheduling, how to quickly and efficiently avoid obstacles and select the proper formation is the core problem of multi-robot scheduling. In order to solve this critical problem, many thermal intelligence algorithms have been introduced in the research [16–20]. With the development of deep-learning technology, today’s deep learning has been successfully applied to many research fields [21–25]. The question is: Can you use the advantages of deep learning to solve multi-robot scheduling coordination problems?
Establishing a multi-robot scheduling algorithm model based on deep learning and giving the best model parameters
Establishing a multi-robot simulation platform, which can be used to perform multi-robot simulation experiments
2.1 Obstacle avoidance and formation transformation of multi-robot scheduling
According to the different requirements of a multi-robot scheduling task, the more successful multi-robot scheduling control methods are based on the behavior method, the navigation-following method, the virtual structure method and so on. The combination of the behavior method and navigation-following method is a common strategy in multi-robot scheduling. The advantages of the leader-follower method are simple in principle, but the disadvantage is that the leader-follower robot is relatively independent, and if the leader is too fast, the robot may not follow the exit track, but the leader is too dependent on the exit track. The behavior-based method is intuitive and has some elastic space in the formulation of behavior, but it does not have good mathematical theory support, so the stability of the formation is hard to analyze.
Firstly, we select an individual in the multi-robot system to be the leader of the formation, then the other robots in the system will be regarded as its followers. The followers are required to track the leader at a fixed distance interval. Secondly, each robot may act as to the decomposition of target movement, robot collision avoidance, obstacle avoidance, maintain formation; the robot makes corresponding responses according to the environmental information sensor obtained by reaction of output as the robot takes the next step (e.g., along a direction or at a speed). This architecture requires a robot behavior selection module, that is to say, establishing some rules to synthesize the output response of each sub-behavior to determine the real action response of the robot, which is the so-called obstacle avoidance and formation transformation strategy.
The operation mode of a multi-robot system can be generally divided into two types: coordination and cooperation. Sometimes people do not distinguish between them which is called “collaboration.” The first problem faced when a multi-robot system is given a task is how to organize the multiple robots to complete the task. The problem to be solved here is how to effectively cooperate with multiple robots. After determining the respective tasks and relationships for a certain mechanism, the problem becomes how to maintain the coordination between the robots; that is, multi-robot coordination. Therefore, multi-robot coordination and multi-robot cooperation are two different and related concepts in multi-robot system research. The former research focuses on the specific motion-control problems after the multi-robot cooperation relationship has been determined, while the latter is the high-level organization and operation mechanism. The focus is on the flexible control mechanism that means that the system can be quickly organized and reconstructed. The system structure is the most important part of the system. The cooperation mechanism between multiple robots is embodied by it and it determines the operating mechanism of the multi-robot system in the process of task decomposition, allocation, planning, decision-making and execution. The role of the individual robot is related to the efficiency of collaboration. The kind of control model that the robot uses is often determined according to the specific working environment, and it is also treated differently according to different types of robots. Planning includes moving from global task level to local, individual, behavioral-action-level planning, which should be dynamic between autonomous individuals to adapt to dynamically changing systems and environments. As a distributed information and decision system, the multi-robot system is an important structural feature of the network structure. Communication is the exchange of individual information, and negotiation is a purposeful information interaction. Perception means that the robot acquires information through its sensing system and uses it. Perception as an interaction between the robot and the outside world and is a more direct and effective way of obtaining information than communication. In order for the system to obtain the control parameters of the ideal collaborative behavior and adapt it to a dynamically changing environment, learning mechanisms are often introduced int the system.
2.2 Deep learning
In this paper, based on deep-learning scheduling, the multi-robot requires pre-planning for obstacle avoidance, formation maintenance and transformation, target tracking locking and so on, and then regards the output of the multi-perception deep neural network as the parameter input of the model. The entire model for the complex has not yet discovered the law of the environment and there will be a lot of environmental information which is constantly updated, and a mathematical model of the information changes is well summarized, then we use the deep-learning algorithm to create the robot learning strategy. We will use some mature models combined with the characterization of perception neural network decision-making or prediction, then characterize the perception neural networks that will become the input parameters of the existing mature model, which can make decision- making or forecasting more accurate. So, in this topic, we combine the strategy formulation of the deep-learning obstacle with the robot-scheduling algorithm.
Firstly, through supervised learning, we use a form of off-line sampling when we supervise and train a certain sample to develop the strategy using a method of training a network through deep learning, the network can then learn the environment according to on-line obstacle recognition. When the robot receives the information about the environment, it can make the decision accurately according to the trained network, so it can avoid the obstacles and better carry out the scheduling.
The concept of deep learning stems from the study of artificial neural networks. A multi-layer perceptron with its multiple hidden layers is a deep-learning structure. Deep learning combines low-level features to form more abstract, high-level, representation attribute categories or features to discover distributed feature representations of data. The concept of deep learning was proposed by Hinton et al. in 2006. Based on the Deep Trusted Network (DBN), an unsupervised, greedy, layer-by-layer training algorithm is proposed with the hope of solving the deep structure-related optimization problems. Then, the deep structure of the multi-layer automatic encoder is proposed. In addition, the convolutional neural network proposed by Lecun et al. is the first true multi-layer structure learning algorithm to use spatial relations to reduce the number of parameters in order to improve training performance. Deep learning is a method based on the representation of data in machine learning. Observations (e.g., an image) can be represented in a variety of ways, such as a vector of each pixel-intensity value, or more abstractly represented as a series of edges, regions of a particular shape, and the like. It is easier to learn tasks from instances (e.g., face recognition or facial expression recognition) using some specific representation methods. The advantage of deep learning is to replace the manual acquisition features with unsupervised or semi-supervised feature learning and hierarchical feature-extraction-efficient algorithms. Deep learning is a new field in machine-learning research. Its motivation is to establish and simulate a neural network for human brain analysis and learning, which mimics the mechanism of the human brain to interpret data. Like the machine-learning method, the deep-machine-learning method also distinguishes between supervised learning and unsupervised learning. The learning models established under different learning frameworks are very different. For example, Convolutional Neural Networks (CNNs) is a deep-machine-learning model under supervised learning, and Deep Belief Nets (DBNs) is a machine-learning model under unsupervised learning.
3.1 Deep-learning environment
Through the following virtual physical environment, we verify the feasibility of the autonomous decision-making method based on the deep-learning network. In the experiment, supervised learning is still needed, and the early stage operations, such as sample formulation and classification, are also needed. These operations are basically similar to those in the third chapter. The samples were divided into 11 categories, including 6400 pictures, of which 4000 were used for training, and 2400 were used for testing. The 11 types of samples includes five types of robot left turn, five types of robot right turn and formation transformation.
Parameter table of convolution neural network
Number of images Size
Convolution kernel size
Fully connected layer
Fully connected layer
From Fig. 7, it can be seen that robots can handle and make the right choices when facing various environments. When the robot has made effective recognition of the environmental information, it can respond quickly according to the decision information and complete the scheduling task.
This paper chooses the obstacle problem of the strategy target-recognition algorithm based on the deep-learning scheduling problem, robot motion strategy is in order to achieve better, faster and more efficient completion of the multi-robot task scheduling, which can then be applied in practical application.
3.2 Construction of robot simulation platform
m_distancel = 15; //length of robot interval
m_distancew = 15; ///width of robot spacer
m_velocityavedistance = 10; ///average speed of robot
m_robotscope = 3; //the value of the scope of the robot, that is the size of the robot
m_distancelimitl = 6; //enter the detection range
m_distancelimits = 2; //enter the dead zone
The fact is that the robot can detect obstacles around here, simulated obstacles are only the last chapter do the experiment of the sample pictures. According to the simulation experiment parameters mentioned above, the following experiments are done to verify whether the robot can complete the scheduling task in various environments.
4.1 Robot scheduling simulation in a simple environment
4.2 Robot scheduling simulation in complex environment
4.3 Robot scheduling simulation in the experimental environment
Multi-robot scheduling has always been a difficult issue in robot scheduling. How to implement efficient multi-robot scheduling and design an appropriate scheduling strategy is a key issue. This paper combines the pilot scheduling method with the following method and the behavior method of the robot based on task scheduling, and then studies how to improve the traditional robot scheduling effect. The deep-learning algorithm is then applied to the scheduling to formulate the strategy of the robot action selection mechanism, and proves the effectiveness of this experimental idea. Based on the above research foundation, in this paper we build a simple simulation experiment platform, which simply sets up three obstacles and completes the task of robot scheduling on the platform. Based on the comparison of different strategies and the roadblock scene analysis, the results show that the method used in this paper can select the path more effectively and can complete the obstacles in the shortest time.
The authors thank the editor and anonymous reviewers for their helpful comments and valuable suggestions.
This work was supported by Hebei Province Natural Science Foundation (No. E2016202341) and the Research Project of Science and Technology for Hebei Province Higher Education Institutions (No. BJ2014013).
Availability of data and materials
Please contact author for data requests.
All authors take part in the discussion of the work described in this paper. Author JL wrote the first version of the paper and did some of the experiments mentioned in the paper. Author FY revised the paper and its previous versions. Both authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- Lim H J, Lee M H, Seo T, et al. Verification of automated cell culture robot system, BioIncubot, using human adipose derived-stem cell[J]. 2017; 한국생물공학회 학술대회.Google Scholar
- L. Yang, Y.U. Bentong, Z. Zhang, et al., in Acta Academiae Medicinae Weifang. Da Vinci robot system versus video-assisted thoracoscopy for the patients with radical surgery of non-small cell lung cancer: a case control study[J] (2018)Google Scholar
- Kamiya K, Saito W, Nagaike K. Cell culture container, automatic cell culture apparatus, liquid container, robot hand, and robot system: US20170369833[P]. 2017;Google Scholar
- A. Shakoor, T. Luo, S. Chen, et al., A high-precision robot-aided single-cell biopsy system[C]// IEEE International Conference on Robotics and Automation. IEEE, 5397–5402 (2017)Google Scholar
- M. Saha, P. Isto, Multi-robot motion planning by incremental coordination[C]// Ieee/rsj International Conference on Intelligent Robots and Systems. IEEE, 5960–5963 (2006)Google Scholar
- K. Solovey, O. Salzman, H. Dan, Finding a needle in an exponential haystack: discrete RRT for exploration of implicit roadmaps in multi-robot motion planning[J], vol 107 (2014), pp. 591–607Google Scholar
- K. Solovey, H. Dan, k-color multi-robot motion planning[J]. International Journal of Robotics Research 33(1), 82–97 (2014)View ArticleGoogle Scholar
- K. Solovey, H. Dan, On the hardness of unlabeled multi-robot motion planning[J]. Computer Science 35(14) (2015)Google Scholar
- C.M. Clark, Probabilistic road map sampling strategies for multi-robot motion planning[J]. Robotics & Autonomous Systems 53(3), 244–264 (2005)View ArticleGoogle Scholar
- J.T. Shepard, C.A. Kitts, A multirobot control architecture for collaborative missions comprised of tightly coupled, interconnected tasks[J]. IEEE Systems Journal 99, 1–12 (2016)Google Scholar
- F. Mizoguchi, H. Nishiyama, H. Ohwada, et al., Smart office robot collaboration based on multi-agent programming[J]. Artif. Intell. 114(1–2), 57–94 (1999)View ArticleGoogle Scholar
- M.A. Rahman, M.S. Miah, W. Gueaieb, et al., SENORA: a P2P service-oriented framework for collaborative multirobot sensor networks[J]. IEEE Sensors J. 7(5), 658–666 (2007)View ArticleGoogle Scholar
- J. Zhao, J. Yan, H. Cai, Architecture of Internet-based multi-operator-multi-robot teleoperation system[J]. High Technol. Lett. 24(5), 459–463 (2003)Google Scholar
- D. Fox, W. Burgard, H. Kruppa, et al., A probabilistic approach to collaborative multi-robot localization[J]. Autonomous Robots 8(3), 325–344 (2000)View ArticleGoogle Scholar
- E. Spaho, K. Umezaki, L. Barolli, et al., A fuzzy-based reliability system for knowledge sharing between robots in P2P JXTA-overlay platform[J]. Cluster Computing 16(4), 933–945 (2013)View ArticleGoogle Scholar
- Q. Zhang, Intelligent computing for 2D spatial information interpolation in GIS[J]. Journal of the Acoustical Society of America 95(2), 1085–1099 (2008)Google Scholar
- P.M. Wong, M. Nikravesh, Introduction: field applications of intelligent computing techniques[j]. Journal of Petroleum Geology 24(4), 381–387 (2010)View ArticleGoogle Scholar
- C.Ö. Karacan, Forecasting gob gas venthole production performances using intelligent computing methods for optimum methane control in longwall coal mines[J]. International Journal of Coal Geology 79(4), 131–144 (2009)View ArticleGoogle Scholar
- S.K. Das, N. Sivakugan, Discussion of intelligent computing for modeling axial capacity of pile foundations[J]. Tetrahedron Letters 51(13), 1770–1773 (2010)View ArticleGoogle Scholar
- J. Guice, Controversy and the state: Lord ARPA and intelligent computing[J]. Social Studies of Science 28(1), 103–138 (1998)View ArticleGoogle Scholar
- G. Litjens, T. Kooi, B.E. Bejnordi, et al., A survey on deep learning in medical image analysis[J]. Medical Image Analysis 42(9), 60–88 (2017)View ArticleGoogle Scholar
- H. Greenspan, B.V. Ginneken, R.M. Summers, Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique[J]. IEEE Transactions on Medical Imaging 35(5), 1153–1159 (2016)View ArticleGoogle Scholar
- M.M. Bronstein, J. Bruna, Y. Lecun, et al., Geometric deep learning: going beyond Euclidean data[J]. IEEE Signal Processing Magazine 34(4), 18–42 (2017)View ArticleGoogle Scholar
- Y. Guo, A. Oerlemans, A. Oerlemans, et al., Deep learning for visual understanding[J]. Neurocomputing 187(C), 27–48 (2016)View ArticleGoogle Scholar
- K. Sirinukunwattana, E.A.R. Shan, Y.W. Tsang, et al., Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images[J]. IEEE Transactions on Medical Imaging 35(5), 1196–1206 (2016)View ArticleGoogle Scholar