1. INTRODUCTION

In the military parade, the synchronization is a major factor for the performance. Shimohagi, Ishihara, and Okugawa [1–3] conducted scientific analysis for the synchronization of the parade; however, it is difficult to measure the synchronization level of the parade because it requires a large number of posture data at same time. We do not have any quantitative evaluation methods for the synchronization level of the parade. In this research, we propose the evaluation method for the synchronization level with OpenPose. OpenPose [4] can measure many kinds of postures from images simultaneously, therefore, it is much easier to get a large number of posture data than conventional motion analysis with motion capture [5,6], and it is suitable for analyzing cooperative operation of multiple people such as parade (Figure 1).

Figure 1

Military Parade with OpenPose ( https://www.youtube.com/watch?v=zaviumhF7aA ).

As a first step of this research, for measuring the synchronization level of the whole of the parade, we focus on the arm swing of two persons in “Marking Time”. Marking Time is a kind of parade style, in which soldiers is moving their legs as in marching without step forward. In this paper, we analyze the synchronization level of the marking time, between two persons with OpenPose. When we evaluate the synchronization level of the two persons appropriately, we can evaluate the synchronization level of the parade.

This paper is organized as follows. After introducing OpenPose in the next section, we describe the data processing method with OpenPose. In Section 4, we propose the evaluation method for the synchronization level. After explaining the two experiments with 3D CG and real behavior of the marking time by two persons, we show our results from the experiments. In addition, this paper is updated with the results of reviews from the proceeding paper in ICAROB [7].

2. OpenPose

OpenPose represents simultaneous posture recognition of multiple people on single images. The left side of Figure 2 shows an example of pose estimation with OpenPose. If the movements of the two are synchronized, we can expect that we get the same trend data.

Figure 2

Multi-person pose estimation with OpenPose.

The right side of Figure 2 shows the points of posture data.

As the three-dimensional data of [x, y, likelihood] on each point, we can obtain 75 data for one piece of posture information. It is difficult to measure the synchronization because the values and the accuracy of the posture data are different depending on the position and body size. Therefore, it is necessary to use the trend of data which is not influenced by position or height as an index. We can get the posture data of each person in same frame with OpenPose, but the data is not related to the posture data in the before and the after frames. In order to obtain the time series posture data of each person, we did cluster analysis based on the location in each frame.

3. PROPOSED METHOD

In this section, we propose the measurement method using indicators focusing on inflection point of movement. In the following, we explain the proposed method in order.

3.1. Focus Points

Focusing on the synchronized arm swing of the left hand, we obtain the data of each shoulder (point 5 in Figure 3), elbow (point 6 in Figure 3) and palm (point 7 in Figure 3). Each data, which are points 5–7, are one-unit set. After we get time series data of each member, we converted the data into four angles: φ, ψ, λ, θ (Figure 3).

Figure 3

Focus points of OpenPose.

3.3. Smoothing

In order to read the inflection point of the time series data of arm swing angle accurately, we use Simple moving average [Equation (2)] for smoothing out short-term fluctuations of the angle data. Simple moving average is the unweighted mean of the n data. When we have a series of the angle data (Ang_t, Ang_t+1, …, Ang_t+n), then the simple moving average Ang_sma is

Angsma=Angt+Angt+1+⋯+Angt+nn (2)

In this method, we set n = 8 for reading the inflection points accurately.

Figure 4 shows an outline of the data before and after the smoothing process. It can be seen that the outline is smooth, and the inflection points are easy to read.

Figure 4

Data smoothing before and after.

4. EXPERIMENT

In this chapter, we conducted two kinds of experiments. In Experiment 1, we examined the measurement accuracy and the evaluation index of our method using 3D CG models. After confirming the effectiveness of the evaluation method, in Experiment 2, we evaluate the synchronization level r in real marking time behavior with our method.

4.1. Experiment 1: Accuracy of Our Method

For measuring the effectiveness of our method, we analyzed the 3D CG models created by Unity. Since we can adjust the synchronization level between two models, it can be an ideal analysis target. 3D CG is appropriate for evaluating the synchronization level of the parade because we can completely control human-like model.

In the 3D CG, the two models are only doing the marking time (Figure 5).

Figure 5

An example of 3D CG models with OpenPose.

The arm swing angle of the 3D CG models was implemented according to the simple vibration [Equation (5)]. In this section, the phase difference of the arm swing angle is expressed by changing the initial condition of θ₀ as 0, π/8, π/4, π/2 and π.

x=Asin(ωt+θ0)           (ω=2π) (5)

Figure 6 shows the synchronization level r of time series angle data of two models with four different angles. Each angle is represented different figure such as φ is represented as square. As the initial value θ_0 shifts from 0 to π, the value r decreases in four angles. Also, when the initial value θ_0 is π, that is, each arm swing angle is in opposite phase, it approaches −1 in four angles. The synchronization level of four angles are almost same in Figure 6.

Figure 6

The synchronization level r of 3D CG.

From the result, the synchronization level r is monotonically decreasing according to the value θ₀. Therefore, it was confirmed that our evaluation method is appreciable for measuring the synchronization level in the 3D CG marking time movie.

4.2. Experiment 2: Evaluation of Marking Time

Since we can show our evaluation method is appreciable, we evaluated the synchronization of the real marking time. As in Experiment 1, we analyzed the marking time of two persons. We experimented with 6 pairs of two cadets in the National Defense Academy, which have parade training every day (Figure 7).

Figure 7

A capture of real marking time with OpenPose.

For Experiment 2, we have three types of arm swing: Phase Synchronization, Phase Difference and Phase Opposition. Figure 7 shows a snapshot of Phase Opposition. Both cadet arm swing is opposite in Figure 7.

Figure 8 shows the average of the synchronization level r of marking time. As shown in Figure 8, we can confirm that the synchronization level r decreases as the difference of the arm swing angle shifts.

Figure 8

The synchronization level r of marking time.

Figure 9 shows the synchronization level r in each angle with standard deviation. In Phase Difference case, we did not set the different level of the arm swing quantitatively like as Experiment 1. Therefore, the standard deviation of Phase Difference is wider than other cases. However, the results of Experiment 2 show the same trend as the result of Experiment 1, and it is our evaluation method is appreciable for measuring the synchronization level in the real Parade. In addition, it was suggested that measuring the synchronization level with armpit angle (φ) has good sensitivity for evaluation comparing to other angles.

Figure 9

The synchronization level r in each angle with standard deviation.

5. CONCLUSION

We propose the evaluation method for the synchronization level of the marking time with OpenPose. We focus on the arm swing angle and analyze the marking time for two persons. Through the experiment using 3D CG, it was confirmed that our method based on time series arm swing angle is appreciable for measuring the synchronization level. Moreover, through the experiment of real marking time of six pairs of two cadets, our evaluation method is appreciable for real environment. In addition, it was suggested that measuring the synchronization level with armpit angle has good sensitivity for evaluation.

Authors Introduction

Mr. Yohei Okugawa

He is the LTJG of Japan Maritime Self Defense Force. He currently belongs to the Department of Computer Science at the National Defense Academy of Japan as a graduate student. He graduated from the College of Media Arts, Science and Technology at the University of Tsukuba in 2014.

Dr. Masao Kubo

He is an Associate Professor of Dep. of Computer Science at National Defense Academy in Japan. He graduated from the Precision Engineering Department, Hokkaido University, in 1991. He received his PhD degree in Computer Science from Hokkaido University in 1996.

Dr. Hiroshi Sato

He graduated Doctor course at Dep. of Computational intelligence and System Science in Tokyo Institute of Technology in Japan. He is an Associate Professor of Department of Computer Science at National Defense Academy in Japan. His research interests include SICE, IEEJ, IEICE, JSAI member.

Mr. Bui Duc Viet

He received his bachelor’s degree from the Department of Computer Science, National Defense Academy, Japan in 2019. He is currently a Master course student in National Defense Academy, Japan.