1. INTRODUCTION

Until now, Computer Graphics (CG) has only been to create and view works. On the other hand, CG in the future will be able to create more realistic images. Furthermore, we will be able to enter in the images, and touch, smell, and taste the images, and will become more familiar. Therefore, CG has become more and more important to cooperate with many fields such as physics, chemistry, earth science, biology, life science, medicine, psychology, cognitive science, electrical and electronic engineering, information and communication technology, computer, civil engineering, architectural engineering, mechanical engineering, robotics, marine engineering, space engineering, art, entertainment, and so on. And, the bridge between them is the concept of artificial reality, Virtual Reality (VR), and tele-existence. In particular, VR is a technology that enables humans to experience a virtual environment other than the one in which they actually exist, as if it were a real environment, and to act in that virtual world. Artificial reality has the same meaning.

Virtual reality has been studied since the 1960s. However, since 2016, the commercialization of more inexpensive and high-performance VR equipment has rapidly increased, so 2016 is called the “first year of VR”, and VR technology has been studied and developed all over the world and has made remarkable progress. Various companies have begun to develop and sell VR equipment, and although it is a little expensive, even ordinary people can easily purchase it. VR technology has become widespread, as PCs and other devices that can use VR devices have become affordable even for individuals. As a result, VR technology is expected to develop in various fields such as medical, welfare, education, training, business, entertainment, and so forth. However, there are some major challenges in the development of VR technology.

We paid attention to the troublesomeness of wearing in this. When using VR equipment, it is often necessary to set up cables and sensors in advance. Also, when using a VR device, it is necessary to have a certain amount of space around the user. This is to ensure the safety of the surroundings and the user himself/herself because the controller is always used when operating/moving in the world inside VR. Therefore, there is a problem that the place where the VR device can be used is limited. It tends to be more expensive than models that require cables, but recently, integrated (standalone) models that do not require cables have also been released. Therefore, the problem of setting up the cable or the like can be solved. However, the spatial problem when using VR equipment has not been solved.

Therefore, to try to solve the spatial problem, the operation by brain waves—Brain–Machine Interface (BMI) is used. Since the frequency band of human brain waves, such as “α wave” and “β wave”, fluctuates depending on the activity state, the purpose is to use them for controlling CG characters and robots by reading them with a device and discriminating them. It is a study. I think that it may be possible to solve the spatial problem when using VR equipment by using this method [1–4].

2. METHOD OF STUDY

2.2. Used Equipment

Equipment is sold by NeuroSky (see Figure 1). In this study, we use this device to measure brain waves.

Figure 1

Mind wave mobile.

2.6. EEG Processing Method

2.6.1. Fast Fourier Transform

It is a type of Discrete Fourier Transform (DFT) and is an algorithm devised so that DFT can be executed at high speed on a computer [see Equation (1)]. When using this analysis method, the MathNet. Numerics library was used [5].

Xk=∑n=0N−1xne−i2πN (1)

2.6.2. EEG feature discrimination method

For EEG characterization, Support Vector Machine (SVM), which is generally used for EEG characterization, was used. SVM is one of the pattern recognition models that uses supervised learning [see Equation (2)]. SVM has high discrimination performance for untrained data. In this study, a Gaussian kernel [see Equations (3) and (4)] was used as the kernel function. In addition, five libraries of MathNet.Numerics, Accord, Accord.MachinerLearning, Accord.Math, and Accord.Statistics were used to use SVM.

y(x)=sgn{∑n=SVwnK(xn,x)+b} (2)

={0    x∈class A1    x∈class B (3)

K(x1,x2)=exp(−‖x1−x2‖22σ2) (4)

2.6.3. Visual Studio 2019

Visual Studio 2019 processes the brain wave data read by Mind Wave Mobile and performs feature detection. At this time, FFT is used to detect the feature amount of the electroencephalogram data. Then, using SVM, the processed brain wave data is discriminated so that 0 is output when the brain wave is relaxed and 1 is output when the brain wave is conscious of moving.

2.6.4. Unity 2019.2.18f

We created CG characters and programs that act according to the characteristics of the read brain waves. The brain wave data was processed by Visual Studio 2019, the result of discrimination by SVM was received, and the CG character was created to act according to the discrimination result. We use “Unity-Chan!” as the CG character, which is provided free of charge in Unity’s Asset Store (see Figure 2).

Figure 2

The CG character which used in Unity.

3. EXPERIMENTAL METHOD

We have the subject sit in a chair, wear Mind Wave Mobile on the head, and then start measuring brain waves. During the experiment, the subjects are made aware of relaxing and moving, and evaluate how the CG character in Unity moves accordingly. In this study, there are only two types of CG character movements: stop when you are relaxed and keep moving forward while you are thinking of moving your body.

4. EVALUATION EXPERIMENT

In this experiment, three university students will be the subjects. An evaluation experiment was conducted and a questionnaire was conducted. The evaluation contents were “good points”, “bad points”, and “others” in a free description format.

5. EVALUATION RESULT

The results of the questionnaire are as follows.

In “Good points,” “It is fresh and interesting to move just by thinking” was mentioned. On the other hand, “Bad points” are “I was worried because there was a little time from thinking until the operation was reflected in the CG character,” “Mind Wave Mobile is uncomfortable to wear,” and “The action that the CG character is thinking There were times when it did the opposite of what it was.” Moreover, “Others” included “Can I do anything other than move forward?” and “Can I use running properly when walking?” “Other” were that “is it not compatible with other movements?” and “is there no other CG character?”, etc.

6. CONSIDERATION

From the “good points” of the evaluation results, it was found that the control of the CG character by brain waves, which is the purpose of this study, can obtain a certain result in distinguishing only two patterns of forward and stop.

From the “bad points”, it was found that there was a problem with the accuracy and speed of EEG discrimination, and it was found that it was necessary to improve the EEG feature detection method and discrimination method, and to improve the performance of the processing device. It was also found that there was a problem with the installation of Mind Wave Mobile, but it is thought that the problem with this is that the size of the product used this time cannot be adjusted. From the above “bad points”, it was found that there are points that should be further improved in this study.

From the opinion of “Other”, it was found that actions other than advancing and stopping are necessary in this study.

Moreover, we would like to increase the number of subjects of the experiment in the future.

7. CONCLUSION

In this study, we tried to solve the space problem around the user, among the “troublesomeness of wearing” which has become an issue in the development of VR technology. The brain wave data was measured by Mind Wave Mobile, which is superior to conventional equipment concerning the compactness and the ease of handling. The brain wave data from Mind Wave Mobile was received and processed by Visual Studio 2019, and the characteristics of the brain wave data processed by SVM were determined. We sent the discrimination result to Unity and verified the method in the field called BMI, which manipulates the characters in Unity based on the discrimination result.

From the results of the evaluation experiment, it was found that although the operation of the CG character by brain waves can obtain certain results, the accuracy is not sufficient and there are many points to be improved. Future tasks include improving the accuracy of brain wave feature detection and brain wave feature discrimination, increasing character movements by discriminating the ratio of brain wave frequency bands, and improving the uncomfortable wearing of Mind Wave Mobile. It is thought that there is, and we plan to work on it in the future.

If this study progresses, it is expected that it will play a major role in the medical field. By using this technology for people with physical disabilities who are physically handicapped or who are using artificial limbs, it will be possible to operate the body only with brain waves, which will be much more convenient than it is now. In addition, the field of e-sports is currently developing in the world, and VR technology is attracting attention in that field as well, so it is expected that if this study progresses, it will be able to play an active role in the field of e-sports.

CONFLICTS OF INTEREST

The authors declare they have no conflicts of interest.

AUTHORS INTRODUCTION

Prof. Makoto Sakamoto

He received the PhD degree in computer science and systems engineering from Yamaguchi University. He is presently an Associate Professor in the Faculty of Engineering, University of Miyazaki. He is a theoretical computer scientist, and his current main research interests are automata theory, languages and computation. He is also interested in digital geometry, digital image processing, computer vision, computer graphics, virtual reality, augmented reality, entertainment computing, complex systems, data science, and so forth.

Mr. Kenji Sakoma

He is a master student at Department of Computer Science and System Engineering, University of Miyazaki. His current research interests are computer graphics, image processing, and VR technology.

Mr. Taketo Kamasaka

He is a master student at Department of Computer Science and System Engineering, University of Miyazaki. His current research interests are computer graphics, entertainment computing, and so on.

Mr. Kodai Miyamoto

He is a master student at Department of Computer Science and System Engineering, University of Miyazaki. His current research topic is a fundamental study on educational support using VR technology.

Prof. Amane Takei

He is working as Associate Professor for Department of Electrical and Systems Engineering, University of Miyazaki, Japan. His research interest includes high performance computing for computational electromagnetism, iterative methods for the solution of sparse linear systems, domain decomposition methods for large-scale problems. He is a member of IEEE, an expert advisor of The Institute of Electronics, Information and Communication Engineers (IEICE), a delegate of the Kyushu branch of Institute of Electrical Engineers of Japan (IEEJ), a director of Japan Society for Simulation Technology (JSST).

Prof. Tsutomu Ito

He is an Assistant Professor in Department of Business Administration, Ube National college of technology. He has published many papers in refereed journals and proceedings, particularly in the area of industrial management, and computer science. His current research interests include internet of things (IoT), mechanical engineering, artificial intelligence (AI), automata theory, quantitative analysis of Japanese Keiretsu. He was one of the winners of the Best Paper Award in the International Conference on Artificial Life and Robotics (ICAROB) in 2015 and 2016, respectively. He earned his Doctor degree of Engineering from Hiroshima University, Japan in 2018.

Prof. Takao Ito

He is Professor of Management of Technology (MOT) in Graduate School of Advanced Science and Engineering at Hiroshima University. He is serving concurrently as Professor of Harbin Institute of Technology (Weihai) China. He has published numerous papers in refereed journals and proceedings, particularly in the area of management science, and computer science. He has published more than eight academic books including a book on Network Organizations and Information (Japanese Edition). His current research interests include automata theory, artificial intelligence, systems control, quantitative analysis of inter-firm relationships using graph theory, and engineering approach of organizational structures using complex systems theory.