Using Importance Ranks to Derive Suitable Timing of Visual Sensing in Manipulation Task Containing Error Recovery

Akira Nakamura; Kazuyuki Nagata; Kensuke Harada; Yukiyasu Domae

doi:10.2991/jrnal.k.190402.002

<Previous Article In Issue

Download article (PDF)

Volume 5, Issue 4, March 2019, Pages 283 - 288

Using Importance Ranks to Derive Suitable Timing of Visual Sensing in Manipulation Task Containing Error Recovery

Authors

Akira Nakamura¹^{, *}, Kazuyuki Nagata¹, Kensuke Harada², Yukiyasu Domae¹

¹Automation Research Team, Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST), Koto-ku, Tokyo 135-0064 Japan

²Robotic Manipulation Research Group, Systems Innovation Department, Graduate School of Engineering Science, Osaka University, Toyonaka 560-8531, Japan

^*Corresponding author. Email: a-nakamura@aist.go.jp

Corresponding Author

Akira Nakamura

Received 12 October 2018, Accepted 14 November 2018, Available Online 30 March 2019.

DOI: 10.2991/jrnal.k.190402.002 How to use a DOI?
Keywords: Manipulation skill; importance rank; error recovery; error classification; task stratification
Abstract: In general, a manipulation task can be composed of many skill primitives. Therefore, it is desirable to carry out plural visual sensing in most skill primitives; however, performing sensing all the time is difficult. In this paper, we propose the addition of importance ranks to the attribute of skill primitives in order to derive a suitable timing for performing sensing. Furthermore, we show that the skill primitives distinguished by their high importance ranks considerably correlates with error recovery.
Copyright: © 2019 The Authors. Published by Atlantis Press SARL.
Open Access: This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION

A robot came to play an active role in many fields from this century. In the robot manipulator, studies came to be performed for the manipulation robot which played an active part in various categories, similarly. Therefore, the use of robot manipulator worked in several domains such as power plants for maintenance and manufacturing factories for production (Figure 1) has been researched upon considerably [1–3].

These manipulation tasks tend to be complex. We have considered the technique that the complicated tasks could be carried out as easy as possible by using the concept called the skills [1–3]. The “skills” are technique to resolve one task into the sequence of motion primitives by making reference to person’s operational execution. If each motion primitive is feasible using a manipulator, it is possible to carry out the task using the robot in many cases.

The robot causes failure as well as a person, too. In late years practice of the difficult task performed by a robot is often demanded, and studies of error recovery have been also proposed [4–7]. However, most are studies of recovery which corrects difference in trajectory of manipulator. The study of error recovery in which restoration from failure is performed is hardly done.

We studied the technique of recovery for various error classes from a small error to big failure [8,9]. About small-scale trajectory adjustment, error recovery is carried out by a forward correction process. However, it is difficult to correct big error with the method of the trajectory adjustment. In the big failure, error recovery is performed by a backward correction process which returns to the previous step.

A task performed by manipulator consists of several skill primitives, and it is desirable that visual sensing is performed in most of skill primitives. However, performing sensing all the time is difficult because of some restrictions. In this paper, we propose to distinguish skill primitives according to their importance ranks to select the primitives with suitable timing of visual sensing. Furthermore, we show that the skill primitive distinguished according to a high rank is deeply related to the skill primitive in which error recovery is considered in advance. In addition, visual sensing is assumed to be performed precisely to stop movement before and after skill primitives are executed.

In the next section, skills performed by manipulator and the stratification of manipulation tasks are explained. The procedure of marking manipulation skill by the importance ranks is proposed in Section 3, and a concept of error recovery and a method of error classification are described in Section 4. Section 5 presents the effectiveness of applying importance ranks to skill primitives using examples of repacking tasks.

2. STRATIFICATION OF TASKS

In this section, fundamental matter about manipulation skill is explained, the details can be found in Hasegawa et al. [1].

2.1. Concept of Skills

We observed and analyzed human behavior in motion, and it was derived that movement of one task consists of sequence of plural meaningful primitives of motion. We named such primitives of movement “skills” [1]. For example, there are move-to-touch skill, rotate-to-level skill, and rotate-to-insert skill. In assembly task, a particular task can consist of these skill primitives. Therefore, these skills mean fundamental skills. There are various methods of the resolution to the sequence with respect to the task. We proposed a method using fluctuations of number of contacting points in a motion primitive of skill [2]. Furthermore, a lot of skills can be derived by observing and analyzing human behavior in motion.

2.2. Stratification of Tasks

Figure 2 shows a hierarchy of manipulation tasks [8]. If the servo layer is disregarded, the skill layer means the lowest layer marked the task⁽⁰⁾ layer. This skill layer is composed of elementary motion primitives. Further, the task⁽¹⁾ layer is a layer on one of task⁽⁰⁾ layer. About the layer above it, the task^{(i + 1)} layer is located one tier above the task⁽ⁱ⁾ layer, similarly. And, the top layer is task^(max).

3. ADDITION OF IMPORTANCE RANKS IN SKILL PRIMITIVES TO DERIVE SUITABLE TIMING FOR VISUAL SENSING

Ideally, visual sensing should be performed twice in each skill primitive to derive the exact posture of the objects before execution and confirm the last state of objects after execution. However, in actuality, it is difficult to carry out sensing at all sensing points because of the restrictions of time and hardware.

We researched the timing appropriate to visual sensing in skill sequences with error recovery in Nakamura et al. [10]. It is efficient that a visual sensing is performed by considering the difficulty in accomplishing a skill primitive, as shown in Case 4 of Section 4 of Nakamura et al. [10]. The degree of necessity of a visual sensing is derived based on the consideration of the following idea in this paper. An importance rank derived from the degree of the necessity of sensing was added to the attribute of each skill primitive. Moreover, the priority of performing visual sensing in each skill primitive can be derived by deciding the whole value of the importance rank in the total task. To decide the value of an importance rank, an operator’s teaching is allowed, although it is desirable to decide the value automatically.

Let us consider importance ranks with a transfer task (Figure 3) and grasping and gripper-open tasks (Figure 4) as an example. In the transfer task, it is designated that a skill primitive in which strict visual sensing is needed has first priority and that a skill primitive with grasping one or more objects has second priority (Table 1). In the grasp and gripper-open skills, the skill primitive in which the contact state, with the grasped object, changes has the top priority (Table 2). The timing of efficient visual sensing in a total task can be derived by considering an importance rank in each skill primitive according to Tables 1 and 2.

Important rank	Conditions
HT (1)	The target position and orientation are demanded strictly.
HT (2)	The hand is grasping one or more objects.
LT (= HT (3))	Other than the above.

HT (1) > HT (2) > LT (= HT (3)).

Table 1

Importance rank of transfer skill

Important rank	Conditions
HG (1)	The contact state with the grasped object changes.
LG (= HG (2))	Other than the above.

HG (1) > LG (= HG (2)).

Table 2

Importance rank of grasp and gripper-open skills

4. ERROR RECOVERY USING CLASSIFICATION

The real performance may differ from the ideal performance. Errors occasionally happen by various reasons in actual manipulation. Our ideas of classifying errors and a stream of error recovery process with hierarchy of task are described in this section (for more details, please refer to Nakamura and Kotoku [8]).

4.1. Classifying Errors

The reasons of failures in practice of manipulation are attributable to several types of errors. We grouped the states of errors into plural classes, namely execution, planning, modeling, and sensing, according to causes [8].

Merely remedying the origins of these errors does not always reach perfection of recovery. When the environment of robotic work changes by the error very much, it may be needed to go back to a former step.

4.2. Recovery Based on Classifying Errors

A generalized flow showing process of stratified tasks taking error recovery into consideration was proposed in Nakamura and Kotoku [8]. Figure 5 illustrates the central portion of Figure 10 in Nakamura and Kotoku [8]. Error recovery is executed through a backward correction process in this process. At the step of confirmation in each skill primitive task⁽⁰⁾_(i0), the outcome is decided as correct or error by using an automated-means or a human judgment. Error recovery is practiced based on the following classifications.

Class 1: It is judged that the error is an execution error, and task⁽¹⁾_(i1) is practiced repeatedly with no correction of the parameters.
Class 2: It is decided that the error is a planning error, and task⁽¹⁾_(i1) is carried out repeatedly with correcting planning parameters.
Class 3: It is decided that the error is a modeling error, and task⁽¹⁾_(i1) is carried out repeatedly with correction of modeling parameters.
Class T⁽¹⁾: It is decided that the error is a sensing error, and task⁽¹⁾_(i1) is carried out repeatedly using different sensing parameters.
Class T⁽²⁾: task⁽²⁾_(i2) is practiced repeatedly after the required changes are performed and goes back to the initial position one tier above layer task⁽¹⁾_(i1).
:
:
Class T^(max): task^(max)_(imax) is carried out repeatedly after necessary changes are done and goes back to the initial position at (max − 1) tier above layer task⁽¹⁾_(i1).
Class T^{(max + 1)}: When too many changes are judged to be necessary, the process being carried out is aborted.

5. IMPORTANCE RANKS IN SKILL PRIMITIVES WITH ILLUSTRATION OF REPACKING TASKS

Let us consider manipulation robots playing an active role in physical distribution. Then, take into account tasks of repacking objects from a large box into a small box at a distribution center and the processes of error recovery for the robot tasks. Here, a representative task of lifting up an object pointed out by using the parallel jaw gripper on the robot [11] is selected, and an importance rank is considered for each skill primitive according to Section 3.

The principal errors in the repacking tasks are described in this section. And the process of recovery is explained with respect to each error. Moreover, we take into account the relation between typical errors of the task and skill primitives with high importance degrees.

5.1 Sequence of Repacking Tasks

Let us consider the task of picking up and placing objects such as Plastic bottles by using a manipulation robot with a gripper. The processes are shown in Figure 6. Figure 7 shows the flow of the task sequence if there is no error. These motion primitives are as follows.

(Skill₁) Move-to-approach: Moving to the starting point of the approach motion.
(Skill₂) Pre-grasp: Opening to grasp the object.
(Skill₃) Approach: Moving at low speed until arriving at the grasping point.
(Skill₄) Grasp: Grasping the target object.
(Skill₅) Lift-up: Lifting the grasped object.
(Skill₆) Departure: Moving to a particular point in a reference frame of a box from which the object is taken.
(Skill₇) Move-between-reference-frames: Moving to a particular point in a reference frame of a box, in which the object must be placed.
(Skill₈) Move-to-destination: Moving to the destination point.
(Skill₉) Lower: Bringing down the grasped object.
(Skill₁₀) Hand-open: Opening to place the object.
(Skill₁₁) Leave: Moving to the safe area.
(Skill₁₂) Home: Going back to the initial point for the continuous task.

5.2 Importance Ranks of Skills in Repacking Tasks

Here, we take into consideration the importance ranks of skills in the repacking task. An importance rank of each skill primitive is derived (Section 3) as follows:

Skill₁: L^T,
Skill₂: L^G,
Skill₃: H^T (1),
Skill₄: H^G (1),
Skill₅: H^T (2),
Skill₆: H^T (2),
Skill₇: H^T (2),
Skill₈: H^T (2),
Skill₉: H^T (1),
Skill₁₀: H^G (1),
Skill₁₁: L^T,
Skill₁₂: L^T.

Let us suppose that the following relation (1) exists among the importance ranks.

(1)HT(1)>HG(1)>HT(2)>L (=LT=LG)

These skill primitives, in turn, can be arranged as follows based on the importance ranks with inequality (1).

First priority: Skill₃, Skill₉.
Second priority: Skill₄, Skill₁₀.
Third priority: Skill₅, Skill₆, Skill₇, Skill₈.
Fourth priority: Skill₁, Skill₂, Skill₁₁, Skill₁₂.

5.3 Candidate Errors in Repacking Tasks

Next, we will consider the representative types of errors occurred in the tasks.

(1)
Errors when grasping and lifting.
- (1a) Error about height; the robotic hand cannot attain to a bottle in Skill₃ (Figure 8a).
- (1b) Error of movement in parallel with the bottom of the box; the open gripper of hand is not put around the cap of a bottle in Skill₃ (Figure 8b).
- (1c) Error in which a bottle cannot be pulled up owing to tight packing in a box in Skill₅ (Figure 8c).
(2)
Errors when carrying an object.
- (2a) Error in which a bottle is dropped in Skill_i (i = 6 – 8) (Figure 9).
(3)
Errors when packing.
- (3a) Error of movement in parallel with the bottom of the box; a gap exists between the target position in Skill₉. No movement of another bottle is performed (Figure 10a).
- (3b) Error in which the bottle grasped by the gripper pushes the neighboring bottles in the box in Skill₉ (Figure 10b).
- (3c) Error in which both (3a) and (3b) occur simultaneously (Figure 10c). Gaps occur both between the positions of the grasped bottle and surrounding bottles.
- (3d) Error in which Skill₉ is stopped before completion (Figure 10d) because the available space is too small.

5.4 Error-recovery Processes

The recovery process from each error is explained in this section. The following numbers and letters in brackets coincide with those describing the candidate errors in the previous section. The corrections were carried out based on definite or indefinite causes derived from the error classification, and the process restarts from the corresponding step.

(1)
Cases (1a), (1b), and (3a)
- Class 1: Execution is simply repeated.
- Class 2: The process is executed repeatedly from the planning step after the planning parameters are modified from the gap.
- Class 3: The process is executed repeatedly from the modeling step after modification of the geometric models of the bottles.
- Class T⁽¹⁾: The process is executed repeatedly from the sensing step after adjustment of the coordinate system.
(2)
Cases (1c) and (3d)
- Class 1: Execution is simply repeated.
- Class 2: The process is restarted from the planning step after the revision of the planning methods such as the motion with slightly shaking in lifting or lowering.
- Class 3: The process is executed repeatedly from the modeling step after modification of the geometric models representing the bottles or box.
- Class T⁽¹⁾: [The same as (1)].
(3)
Case (2a)
In most cases, the task of picking up a bottle in the box is changed to the task of lifting the bottle that has dropped to the bottom of the transition path; Class T⁽²⁾ is chosen. The skill sequence of this task is executed from the beginning.
(4)
Cases (3b) and (3c)

The task according to scenario is finished, and the process moves to the initial position of the task sequence at one point. Next, the task in which the bottle should be moved to the correct position is executed. If necessary to move many bottles, it is desirable to investigate whether or not the paths of correction converge efficiently.

5.5 Sensing Time in Repacking Tasks

We considered typical errors in 5.3 and recovery processes of errors in 5.4. Furthermore, we introduced the importance ranks of skill primitives in 5.2 so that Skill₃ (Approach skill) and Skill₉ (Lower skill) became high priority tasks. Then, these are included in skill primitives treated in 5.3 and 5.4. That is, Skill₃ is relevant to (1a) and (1b), and Skill₉ is relevant to (3a)–(3d). Thus, the skill primitive in which error recovery is almost considered has a deep relation with the skill primitive in which visual sensing is needed. Moreover, skill primitives that are important alongside the above-mentioned skills are grasp and gripper-open skills, i.e., Skill₄ and Skill₁₀, respectively, and transfer skills: Skill₅–Skill₈. Especially, Skill₅ and Skill₇ are skill primitives with high importance ranks, as in Cases (1) and (2) in 5.3, respectively.

When we take into account error recovery processes in the task composed of several skill primitives, the total number of skill primitives in which our error-recovery procedure [8,9] can be applied becomes very larger. It is difficult to consider those all the recovery processes, since great effort is necessary for planning of these recovery processes. When using the importance ranks proposed in Section 3, it becomes relatively easy to choice skill primitives, for which it is suitable to consider an error-recovery procedure.

6. CONCLUSION

In general, a manipulation task can be composed of many skill primitives. It is desirable to perform plural visual sensing in most skill primitives as geometric modeling and task planning are carried out before execution, and task achievement is confirmed after execution. However, the performance of sensing at all sensing points is difficult because of the restrictions of time and hardware. In this paper, we proposed the addition of importance ranks to the attribute of skill primitives to derive suitable timing to perform visual sensing. The use of the proposed method eases the selection of skill primitives in which visual sensing should be performed. Furthermore, we showed that skill primitives distinguished according to their high importance ranks considerably correlate with those in which error recovery is considered in advance. This characteristic can be used for selecting skill primitives in which error recovery planning is carried out in advance.

In the future, we will apply our proposed procedure to sensing, modeling, planning, and execution in many kinds of tasks using robotic hands. We will attempt to apply our procedure of error recovery to tasks executed by actual robots.

CONFLICTS OF INTEREST

There is no conflict of interest.

Authors Introduction

Dr. Akira Nakamura

He received the PhD degree in Electrical Engineering from Keio University in 1991. He is currently a research staff at National Institute of Advanced Industrial Science and Technology (AIST). His research interests include robot planning, vision and control system.

Dr. Kazuyuki Nagata

He received his PhD in Engineering from Tohoku University in 1999. He is currently a Senior Research Scientist with Intelligent Systems Research Institute of Advanced Industrial Science and Technology (AIST).

Prof. Kensuke Harada

He received his Doctoral degrees in Mechanical Engineering from Kyoto University in 1997. From 2016, he has been working as a Professor at Graduate School of Engineering Science, Osaka University.

Dr. Yukiyasu Domae

He received his PhD degree from Hokkaido University in 2014. He is currently a team leader with Artificial Intelligent Research Center (AIRC), National Institute of Advanced Industrial Science and Technology (AIST).

REFERENCES

[1]T Hasegawa, T Suehiro, and K Takase, A model-based manipulation system with skill-based execution, in IEEE Transactions on Robotics and Automation, IEEE (1992), pp. 535-544.

[2]A Nakamura, T Ogasawara, T Suehiro, and H Tsukune, Skill-based backprojection for fine motion planning, in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’96), IEEE (Osaka, Japan, Japan, 1996), pp. 526-533.

[3]A Nakamura, T Ogasawara, K Kitagaki, and T Suehiro, Using robust and simplified geometric models in skill-based manipulation, in Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the Next Millennium (Cat. No. 01CH37180), IEEE (Maui, HI, USA, USA, 2001), pp. 139-145.

[4]BR Donald, Planning multi-step error detection and recovery strategies, Int. J. Robot. Res., Vol. 9, 1990, pp. 3-60.

[5]CM Baydar and K Saitou, Off-line error prediction, diagnosis and recovery using virtual assembly systems, in Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No. 01CH37164), IEEE (Seoul, South Korea, South Korea, 2001), pp. 818-823.

[6]M Scheutz and J Kramer, Reflection and reasoning mechanisms for failure detection and recovery in a distributed robotic architecture for complex robots, in Proceedings 2007 IEEE International Conference on Robotics and Automation, IEEE (Roma, Italy, 2007), pp. 3699-3704.

[7]P Pastor, M Kalakrishnan, S Chitta, E Theodorou, and S Schaal, Skill learning and task outcome prediction for manipulation, in 2011 IEEE International Conference on Robotics and Automation, IEEE (Shanghai, China, 2011), pp. 3828-3834.

[8]A Nakamura and T Kotoku, Systematization of error recovery in skill-based manipulation, Artif. Life Robot., Vol. 14, 2009, pp. 203-208.

[9]A Nakamura, K Nagata, K Harada, N Yamanobe, T Tsuji, T Foissotte, et al., Error recovery using task stratification and error classification for manipulation robots in various fields, in 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE (Tokyo, Japan, 2013), pp. 3535-3542.

[10]A Nakamura, K Nagata, K Harada, and N Yamanobe, The suitable timing of visual sensing in error recovery using task stratification and error classification, J. Robot. Netw. Artif. Life, Vol. 4, 2017, pp. 134-137.

[11]A Nakamura, K Nagata, K Harada, and N Yamanobe, Error recovery using task stratification and error classification for manipulation robots in physical distribution, Trans. Control Mech. Syst, Vol. 3, 2014, pp. 122-131.

<Previous Article In Issue

Download article (PDF)

Journal: Journal of Robotics, Networking and Artificial Life
Volume-Issue: 5 - 4
Pages: 283 - 288
Publication Date: 2019/03/30
ISSN (Online): 2352-6386
ISSN (Print): 2405-9021
DOI: 10.2991/jrnal.k.190402.002 How to use a DOI?
Open Access: This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

ris enw bib

TY  - JOUR
AU  - Akira Nakamura
AU  - Kazuyuki Nagata
AU  - Kensuke Harada
AU  - Yukiyasu Domae
PY  - 2019
DA  - 2019/03/30
TI  - Using Importance Ranks to Derive Suitable Timing of Visual Sensing in Manipulation Task Containing Error Recovery
JO  - Journal of Robotics, Networking and Artificial Life
SP  - 283
EP  - 288
VL  - 5
IS  - 4
SN  - 2352-6386
UR  - https://doi.org/10.2991/jrnal.k.190402.002
DO  - 10.2991/jrnal.k.190402.002
ID  - Nakamura2019
ER  -

download .riscopy to clipboard