Toward Smart Tomato Greenhouse: The Fourth Tomato Harvesting Robot Competition

Takayuki Matsuo; Takashi Sonoda; Yasunori Takemura; Masanori Sato; Kazuo Ishii

doi:10.2991/jrnal.k.190829.002

<Previous Article In Issue

Download article (PDF)

Volume 6, Issue 2, September 2019, Pages 138 - 142

Toward Smart Tomato Greenhouse: The Fourth Tomato Harvesting Robot Competition

Authors

Takayuki Matsuo¹^{, *}, Takashi Sonoda², Yasunori Takemura², Masanori Sato³, Kazuo Ishii⁴

¹Department of Creative Engineering, National Institute of Technology, Kitakyushu College, 2-20-1 Shii, Kokuraminami, Kitakyushu, Fukuoka 802-0985, Japan

²Department of Integrated System Engineering, Nishinippon Institute of Technology, 1-11 Aratsu, Kanda, Miyako, Fukuoka 800-0394, Japan

³Department of Applied Information Technology, Nagasaki Institute of Applied Science, 536 Aba, Nagasaki, Nagasaki 851-0193, Japan

⁴Graduate School of Life Science and System Engineering, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan

^*Corresponding author. Email: matsuo@kct.ac.jp

Corresponding Author

Takayuki Matsuo

Received 8 November 2018, Accepted 31 December 2018, Available Online 24 September 2019.

DOI: 10.2991/jrnal.k.190829.002 How to use a DOI?
Keywords: smart greenhouse; agriculture robot; tomato harvesting robot
Abstract: Agriculture, one of the most important industries for human life, is faced with serious problems, the shortage of workers, the falling birthrate and the aging population, global warming and natural disasters, etc. Tomato is one of the most important fruit vegetables, and most tomatoes in Japan are cultivated in the greenhouses, or large scale farms as the solution for effective production toward smart greenhouses, whereas the high temperature and high humidity, construction and management costs are big problems and technologies such as environmental control system, factory automation technology, AI, robotics are required. We had held Tomato Harvesting Robot competition from 2014 toward realization of smart tomato greenhouse aiming at promote the automated tomato harvesting to reduce the working time of harvesting. In this paper, we report on the tomato harvesting robot competition and the results mainly from the fourth competition.
Copyright: © 2019 The Authors. Published by Atlantis Press SARL.
Open Access: This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION

According to the statistical information of the Ministry of Agriculture, Forestry and Fisheries in Japan, Japan’s food self-sufficiency rate (calorie base) is about 40%, which is the lowest level among major developed countries. In the agricultural field, the aging and depopulation of producers is progressing, and labor shortage due to lack of successors are serious problems. “Smart agriculture” is proposed as a future agriculture production that solves these problems. One of the key technologies for smart greenhouse is robotics, therefore, agricultural robots are expected to be introduced to the agricultural field as the solution tools.

Tomato is one of the important fruit vegetables and most tomatoes are produced in the greenhouses, or large-scale farms, where the high temperature and humidity, and long harvest age force the farmer heavy works. To develop the tomato harvesting robot, many research issues exist such as manipulator design, end-effector design, collaborative behavior, artificial intelligence, motor control, image processing, target recognition and so on. Kawamura et al. have developed a tomato harvesting robot equipped with a 5-degree-of-freedom (DOF) manipulator on a mobile platform. It shows that the target can be grasped with sufficient accuracy for the commanded position [1]. Tomato recognition is based on the luminance signal and red signal from the camera, and the three-dimensional position estimation of the tomato fruit by stereo vision [2,3]. It points out that the recognition rate declines due to changes in the lighting environment and that it is difficult to recognize stems. As the next step, speeding up tomato harvesting is desired, and research on feed-forward control has been developing. Kondo [4] experimentally investigated the natural frequency of tomato tufts to introduce a model of tomato tufts, and introduced them into manipulator trajectory planning. The harvesting mechanism (end-effector) for harvesting tomatoes is also an important research subject [4,5], and methods such as cutting from stems, breaking, and suction are proposed. Taking advantage of the recent development of computers, real-time image processing technology has also been developed. Tomato fruit gripping experiments are being carried out in an environment where the direction of easy gripping and sunshine conditions change [6]. Xu et al. developed a methodology for automated strawberry grading system based on image processing. Al-Hiary et al. [7] proposed and experimentally evaluate a software solution for automatic detection and classification of plant leaf diseases. Research is actively progressing in recent years. Wang et al. [8] developed a tomato harvesting robot used in greenhouse which consists of a four-wheel independent steering system, a 5-DOF harvesting system, a navigation system and a binocular stereo vision system steering system. Due to the rapid development of image processing technology in recent years, agricultural robots using Deep Learning are attracting attention. Wang Xinshao and Cheng [9] proposed weed seeds classification based on Principal Component Analysis (PCA) Network Deep learning and the PCA Network variant method obtains good classification result and improves the recognition accuracy. And, Jeon et al. [10] developed plant lead recognition system using a Convolution Neural Network (CNN) and proposed a method to classify leaves using the CNN model.

We have also organized Tomato Harvesting Robot competition from 2014 aiming at smart tomato greenhouse as the tomato harvesting time engages in the whole tomato production [11]. In this paper, we will report on the result of the fourth tomato harvesting robot competition.

2. COMPETITION REGULATIONS

The tomato harvesting robot competition consists of two leagues which are the Senior League and the Junior League. The target competitors for Senior League are supposed to the team with automated and remote-controlled robots, and the Junior League are for high school or junior high school students who build robot using LEGO Mindstorm. In this section, competition regulations of both reargues will be introduced.

2.1. Senior League

The Senior League supposes that teams are not restricted to university nor company, just each team should have own tomato harvesting robot(s). Two kind of competition field are prepared, one is the rail-style area (Figure 1a) and the other is the free-style (Figure 1b). The rail-style area is designed to have the similar environment with the large-scale tomato factory with heat pipes for warming greenhouse. Free-style area is for the robots of open-field culture environment. Tomatoes are hanged on height from 800 to 1200 mm referring to the tomatoes arrangement of Hibikinada Saien Co., Ltd.

The required specification of tomato harvesting robot is shown in Table 1. Regarding the sizes of robot, the projected area of the robot to the ground is within the 800 × 800 mm² and no height limitation. For safety reason, the robots should have an emergency stop switch on the easy-to-find position. As the recommendations, the weight of the robot is <50 kg and the electric power of motor is <70 W/each.

Size	W: 800 mm, D: 800 mm (H: No limitation)
Equipment	Emergency stop switch
Weight	Max. 50 kg (recommend)
Motor power	Max. 70 W (recommend)

Table 1

Specifications of a robot

The first stage is intended as the inspection of basic functions needed for tomato harvesting, so that single tomatoes are suspended as shown in Figure 2a. The team succeeded to touch the tomato, moves to the second stage. In second stage, some sets of bunches of tomatoes are suspended as shown in Figure 2b. The five high score teams go up to the final stage. The score is calculated based on the number of successfully harvested tomatoes and damages to tomatoes. Also, the unripe tomatoes are counted as the damaged tomatoes. Final score of second stage is the score multiplied the basic score and the coefficient decided by a combination of choice between the control method which are remotely and autonomous control and the areas which are the rail area and the free style area. In final stage, the robots harvest tomato from plant body as shown in Figure 2c.

The robots are classified mainly into two types, manual control and autonomous control, and the former robots are classified by whether the operator observes tomato directly or indirectly using cameras mounted on robots, and by robot locomotion whether the robot uses rail or not. Totally, the robots are categorized into six types depending on operation and locomotion method as shown in Table 2. The success points of one-tomato-harvesting change depending on the robot category, e.g., the point of one-tomato-harvesting for T-1 is 2 × 1 = 2 points, and that of category T-6 is 2 × 8 = 16 points.

Robot control	Manual (x1)				Autonomous (x8)

Operator manipulation	Direct observation of tomato (x1)		Indirect observation of tomato (x2)		Start and stop commands
Mode of locomotion	Rail-type (x1)	Free-type (x2)	Rail-type (x1)	Free-type (x2)	Rail-type (x1)	Free-type (x2)
Total ratio for one tomato	x1	x2	x2	x4	x8	x16
Category	T-1	T-2	T-3	T-4	T-5	T-6

Table 2

Categories of tomato robots

2.2. Junior League

In Junior League, the subject is to carry small size tomatoes to assigned positions. Students should develop the robot using LEGO Mindstorm with the functions such as line trace, color recognition, end-effector with mechanism design and motors with control, and their programming [11].

In Line Trace Challenge, robots should detect white line in the competition area and move along the line using a color sensor. On the way of the course, the tomato-box-harvesting field (harvest field) exists, where tomato boxes are arranged. The robot must move to the harvest field in order to get the tomato boxes. In Color Identifying Challenge, robots should explore and recognize color signs of the same color of three boxed tomatoes (tomato box) on the course. As guidance to harvest field, red, yellow and blue lines are drawn in the field. The robots should detect guidance line by color sensor and carry the tomato boxes to the same color positions.

In Mechanism Design and Control Challenge, robot should manipulate a tomato box or boxes using a manipulator and carry to the storage location. To pick up the tomato box, a manipulator equipment should be made by each team. The robot is required to store, transport and relocation depending on tomato box colors. After picking up the box, the robot should return to the course. In Object Detection Challenge, robots should detect a battery charging station and stop there.

The basic specification for robot is that the size of the robot is within 300 × 300 mm² on ground. Figure 3 shows overview of Junior League.

3. SUMMARY OF THE FOURTH TOMATO HARVESTING ROBOT COMPETITION

Five finalists are: Hibikino Tom’s R (Category T-5, Kyushu Institute of Technology), Nagasaki GANBARANBA (Category T-4, Nagasaki Institute of Applied Science), KPC2017A, KPC2017B (Category T-5, Kyushu Polytechnic college) and HAYASHI-LAB (Category T-5, Kyushu Institute of Technology) as shown in Table 3. Overall winner and Overall runner-up are shown in Table 4. In Junior League, number of the participated teams were 18 teams and the result is Table 5.

Team	Affiliation	Category
HAYASHI-LAB	Kyushu Institute of Technology	T-5
Hibikino-Tom’s R	Kyushu Institute of Technology	T-5
Nagasaki GANBARANBA	Nagasaki Institute of Applied Science	T-4
KPC2017A	Kyushu Polytechnic college	T-5
KPC2017B	Kyushu Polytechnic college	T-5

Table 3

The list of finalists

Ranking	Team	Score
Overall winner	HAYASHI-LAB	120
Overall runner-up	Hibikino-Tom’s R	68

Table 4

Final result of the Senior League

Ranking	Team
1st place	Ohnishi lions (Fukuoka Technical High School)
2nd place	Ooita kogyo C (Oita Technical High School)
3rd place	NiAScience (High school attached Nagasaki Institute of Applied Science)

Table 5

Result of Junior League

4. DISCUSSION

The score evaluation of the senior league of the fourth tomato harvesting robot competition are shown in Equation (1)

(1)P=C(2α+β)-2(γ+δ)-ε

where P is the final score, C: the category coefficient in Table 2, α: the number of obtained tomatoes without damage, β: the number of damaged tomatoes, γ: the number of dropped tomatoes, δ: the number of damaged tomatoes, ε: deducted point when robots damage to tomato plants (the final stage). Table 6 shows the result of the second stage. Score 1 is final score using Equation (1). Team A was the first place in the second stage, which could be 110 score. However, Team A dropped nine tomatoes, the worst number in the seven teams of Table 6. In the case of using Equation (1), Total score P is almost not influenced from γ, δ and ε at selecting high class such as T-5 and T-6. This means that the positive harvesting makes bigger score than the risk of dropping. Therefore, some teams dropped many tomatoes intentionally for the purpose to avoid difficult tomatoes. That is, the tomatoes are not handled carefully. In actual tomato factory, damaged tomatoes and drop tomatoes have no commercial value. Therefore, we considered that concept of carefully dealing with tomatoes is employed. We proposed new equation as shown in Equation (2).

(2)P=ηC(2α+β)-2(γ+δ)-ε

(3)η=αα+β+γ+δ

Equation (2) used “harvest success rate η” expressed by Equation (2) and if robot damage and drop tomatoes magnification C is decreased. Score 2 of Table 6 shows the result of changing scores after applying Equation (2). Number of dropped tomatoes influences the score and Team A becomes the second place.

Team	Category	C	α	β	γ	η	Score 1	Score 2
A	T-5	8	7	2	9	0.50	110	31.78
B	T-5	8	4	3	1	0.39	86	42.00
C	T-4	4	4	2	1	0.57	38	20.86
D	T-5	8	1	1	0	0.50	24	12.00
E	T-5	8	1	0	0	1.00	16	16.00
F	T-3	2	1	1	0	0.50	6	3.00
G	T-1	1	1	0	0	1.00	2	2.00

Table 6

Results of second stage (Score 1) and score after applying Equation (2) (Score 2)

5. CONCLUSION

In this paper, tomato harvesting robot competition is introduced toward Smart Tomato Greenhouse. The tomato harvesting robot competition has two leagues which are Senior and Junior League. In Junior League, number of the participated teams were 18 teams. In Senior League, number of participated teams were nine teams. In next competition, concept of carefully dealing with tomatoes will be used and we proposed harvest rate η and if robot damage and drop tomatoes, the score of team will be decreased.

ACKNOWLEDGMENTS

This works were supported by the Center for Socio-Robotics Synthesis, Kyushu Institute of Technology, The University of Kitakyushu, Nippon Bunri University, Nishinippon Institute of Technology [This support included The Ministry of Education, Culture, Sports, Science and Technology (MEXT), Center of Community.], Kitakyushu College of Technology, Hibikinada-Saien, City of Kitakyushu, The Kitakyushu Chamber of Commerce and Industry, Institute of Systems, Information Technologies and Nanotechnologies (ISIT), Japan Chapter of IEEE Society on Systems, Man, and Cybernetics, The Japan Society of Mechanical Engineers (JSME), The Robotics Society of Japan (RSJ), Kitakyushu Foundation for the Advancement of Industry Science and Technology (FAIS), IT and Mechatronics Chapter and Kyushu Chapter of Japanese Society of Agricultural Machinery and Food Engineers, The MathWorks Co. Ltd., and Ministry of Agriculture, Forestry and Fisheries (MAFF).

Authors Introduction

Dr. Takayuki Matsuo

He is an Associate Professor at Department Creative Engineering, National Institute of Technology (KOSEN), Kitakyushu College, Japan. His research area is underwater robots and biomimetic robots.

Dr. Takashi Sonoda

He is an Associate Professor at Department Integrated System Engineering, Nishinippon Institute of Technology, Fukuoka, Japan. His research are underwater robotics and robot manipulator systems.

Dr. Yasunori Takemura

He is an Associate Professor at Department Integrated System Engineering, Nishi-Nippon Institute of Technology, Fukuoka, Japan. His research area is about machine learning, data mining and Robotics.

Dr. Masanori Sato

He is an Associate Professor in the Faculty of Applied Information Technology in Nagasaki Institute of Applied Science, Japan. His main interests are agricultural robots, underwater robots, and life support robots.

Prof. Kazuo Ishii

He is a Professor at Graduate School of Life Science and System Engineering, Kyushu Institute of Technology, Japan. His research area is about field robots and intelligent robot systems

REFERENCES

[1]Kawamura et al., Study on agricultural robot (1st report), J. JASM, Vol. 46, 1984, pp. 353-358. (in Japanese).

[2]Kawamura et al., Study on agricultural robot (2nd report), J. JASM, Vol. 47, 1985, pp. 177-182. (in Japanese).

[3]Kawamura et al., Fruit harvesting robot, J. JASM, Vol. 47, 1985, pp. 237-241. (in Japanese).

[4]N Kondo, K Yamamoto, H Shimizu, K Yata, M Kurita, T Shiigi, et al., A machine vision system for tomato cluster harvesting robot, Eng. Agric. Environ. Food, Vol. 2, 2009, pp. 60-65.

[5]N Kondo, K Yata, M Lida, T Shiigi, M Monta, M Kurita, et al., Development of an end-effector for a tomato cluster harvesting robot, Eng. Agric. Environ. Food, Vol. 3, 2010, pp. 20-24.

[6]T Ota, T Yamashita, S Hayashi, and T Komeda, Development of a tomato harvesting robot with a vision system using specular reflection (part 1), J. JSAM, Vol. 72, 2010, pp. 587-594.

[7]H Al-Hiary, S Bani-Ahmad, M Reyalat, M Braik, and Z Alrahamneh, Fast and accurate detection and classification of plant diseases, Int. J. Comput. Appl., Vol. 17, 2011, pp. 31-38.

[8]LL Wang, B Zhao, JW Fan, XA Hu, S Wei, YS Li, et al., Development of a tomato harvesting robot used in greenhouse, Int. J. Agric. Biol. Eng., Vol. 10, 2017, pp. 140-149.

[9]W Xinshao and C Cheng, Weed seeds classification based on PCANet deep learning baseline, in 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), IEEE (Hong Kong, China, 2015), pp. 408-415.

[10]W-S Jeon and S-Y Rhee, Plant leaf recognition using a convolution neural network, Int. J. Fuzzy Log. Intell. Syst., Vol. 17, 2017, pp. 26-34.

[11]K Ishii, Y Takemura, T Matsuo, and T Sonoda, Tomato harvesting robot competition, in 2016 Joint 8th International Conference on Soft Computing and Intelligent Systems (SCIS) and 17th International Symposium on Advanced Intelligent Systems (ISIS), IEEE (Sapporo, Japan, 2016), pp. 537-542.

<Previous Article In Issue

Download article (PDF)

Journal: Journal of Robotics, Networking and Artificial Life
Volume-Issue: 6 - 2
Pages: 138 - 142
Publication Date: 2019/09/24
ISSN (Online): 2352-6386
ISSN (Print): 2405-9021
DOI: 10.2991/jrnal.k.190829.002 How to use a DOI?
Open Access: This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

ris enw bib

TY  - JOUR
AU  - Takayuki Matsuo
AU  - Takashi Sonoda
AU  - Yasunori Takemura
AU  - Masanori Sato
AU  - Kazuo Ishii
PY  - 2019
DA  - 2019/09/24
TI  - Toward Smart Tomato Greenhouse: The Fourth Tomato Harvesting Robot Competition
JO  - Journal of Robotics, Networking and Artificial Life
SP  - 138
EP  - 142
VL  - 6
IS  - 2
SN  - 2352-6386
UR  - https://doi.org/10.2991/jrnal.k.190829.002
DO  - 10.2991/jrnal.k.190829.002
ID  - Matsuo2019
ER  -

download .riscopy to clipboard