2.1. Emotion Recognition System (Methodology)

In order to develop the emotional state model, an experimental setup was designed with a structured procedure to invoke the desired emotion as shown in Figure 3.

Fig. 3.

Steps in data collection for emotion modeling.

The experiment was conducted in a controlled room with 20°C. There are 21 healthy subjects randomly chosen from International Islamic University Malaysia (IIUM) Gombak and Kuantan campuses. The audio-visual materials in inducing subject’s emotion are compiled from International Affective Picture System and video sharing website [18]. The efficacy of audio-video stimuli to invoke the desired emotion has been reported in [19, 20].

Basic emotion models consist of the fundamentals human feeling namely anger, joy, sadness, surprise, boredom, disgust, and neutral [21]. Since the model is limited by a number of fundamental human emotions, the theorists had to hypothesize the mixing of the basic emotions into another type of emotion blending. The emotion blending must consist of more than one fundamental emotion as listed before. For this research, there are two fundamental emotions and only one emotion blending is being considered. The two fundamental emotions are happy (joy) and sad, while the emotion blending is nervous. Meanwhile, natural or calm emotion is being considered as a default emotion for the hybrid automata system developed. The other type of emotions is under the future study of this work. The crucial role of emotion in the control of rehabilitation system stems from the fact that current system only deals with physiological aspect of the subject [22].

The psycho-physiological aspect should make the system more adaptive and could mimic the human therapist who requires to access the emotional state of the subject before any session of rehabilitation therapy begins.

There are 3 sessions for each subject marked as ‘a’ in the Figure 3 flowchart representing 3 types of emotions under study as mentioned before. A set of electromagnetic (EM) signals from human body are recorded by using Resonant Field Imaging (RFI™) device. The efficiency of the device has been proved in [23] to classify human health condition in particular for smoker and non-smoker. In [24], there are 17 regions on human body that can be used as measuring points for determining human emotion. However, this works consider only ten of them due to the fact that the emotion recognizer is developed to be used in the robot assisted rehabilitation platform for post stroke patients specifically. These points are both left and right head, arms, palms, thighs, and forearms as denoted in Figure 4. It is then used to model the relationship between the EM signals dataset and the corresponding emotion; with calm emotion set as the default emotion for the constructed system [25].

Fig. 4.

Point of interests (POIs) for EM signal acquisition.

By utilizing Waikato Environment for Knowledge Analysis (WEKA) programming [26], the dataset is being tried on a few classifiers, for example, Attribute Selected Classifier with a precision of 70%, Decision Table (74%), Naïve Bayes (68%) and the Multilayer Perceptron with 52%. Bayes Network (BN) performed the best with the most elevated classification precision so it is chosen as the classifier with a Simple Estimator calculation in looking the contingent likelihood tables. Tenfold cross approval is chosen as the test alternative in this exploration.

The BN is fit to give reasoning under vulnerability, evaluating convictions for theory variable, and steady in joining all data from different sources [27]. The Bayesian classifier assumes the attributes to have independent distributions and the theorem for the classifier is given in Eq. (1) and Eq. (2).

• d_i is the features (10 points EM signals),

• c_j is the class (emotion),

From the experiment, the result shows that the system is able to recognize the right emotion at the rate of 90% accuracy and 80.6% precision by using Bayesian Network classifier.

The GUI provides a display and a manual input as illustrated in Figure 5. It can only provide parameters for the control system and cannot perform any automatic coding. To use the GUI, the user has to be defined and the set of 10 EM signal values have to be keyed in. The trained BN classifier is then executed to classify the emotion. The emotion s then coded into value that is later passed to the hybrid automata system to control the rehabilitation robotic platform.

Fig. 5.

Graphical User Interface (GUI) for displaying appropriate emotion.

As the last step, the new coded dataset value is customized into the chosen classifiers before the matching identical emotion and its relating hybrid automata code are shown as the yield. The hybrid automata code is the coded value that is passed from GUI to Stateflow to administer the rate or speed, and the direction of the end-effector of the rehabilitation robotic platform. For instance, Code 1 shows that the subject is in sad feeling. On the off chance that the client is in nervous feeling, Code 2 is shown while Code 3 is assigned for happy feeling. A default feeling which is calm is marked as code 0. “Reset” catch is utilized to initialize the EM signal values at all POI to be zero.

In confirming the emotion prompted during the experimental session, toward the end of every varying audio-visual session, the subject is given an arrangement of questionnaire [28]. The questionnaire is made out of 2 sections which are demographic information of the subject and a Likert Scale of 4 for every video session. Details of the questionnaire can be found in Figure 6.

Fig. 6.

Questionnaire to verify the subject’s emotion

2.2. Robotic Platform

In embedding the emotion into Human Machine Interaction applications, an upper extremity rehabilitation robot is utilized to test the system via hybrid automata as the control system. There are four major parts of the rehabilitation robotic platform system consisting of rotational axis actuator, linear guide or locking mechanism, linear guide actuator, and gripper as depicted in Figure 7 [25].

Fig. 7.

The robot assisted rehabilitation platform.

The affected subject’s arm is put on the platform then the linear actuator help the movement of the arm forward and backward repeatedly by using brushless DC motor. Impact of stroke generally shows up on one side of the body where amid the rehabilitation method of the influenced limb (i.e. arm), the unaffected one is held constraint. The system of the rehabilitation treatment through arm movement is appeared in Figure 8.

Fig. 8.

Demonstration of the affected arm’s movement during treatment.

2.3. Controller Modality: DES

Figure 9 illustrates the block diagram of emotion recognition system embodiment into the rehabilitation robotic platform via hybrid automata system [29].

Fig. 9.

Emotion embodiment into rehabilitation robotic platform

The output from the emotion recognizer is being classified and coded by GUI before the hybrid automata system use the coded emotion to give several commands to the robotic platform in controlling the speed and direction of the gripper movement. S_i/r_i represents the state and the control symbol for each mode and x_i represents the plant symbol of the system.

The controller modality for the robotic platform is Discrete Event System (DES) as elaborated in Figure 10.

Fig. 10.

Controller modality for robotic platform with emotion embodiment.

The controller developed for this project is divided into 3 major parts which are high level controller, interface, and low level controller (plant). The high level controller is designed to control the designated discrete input-output while the low level controller in plant is used to handle the continuous input-output to the robotic platform. The interface used to interchange both discrete and continuous element in the DES is the robotic platform, through its ability in changing the speed and direction of the gripper based on emotion detected.

For the developed system, the input-output discrete elements are the emotion of the subject and the control state commands via hybrid automata system whereas, the speed and position of the gripper movement is labelled as input-output continuous elements [30].

A set of the discrete inputs for the hybrid automata model is the EM signals from the POI which are eventually classified into particular emotions while the continuous input is the desired speed of the rehabilitation platform. The basic representation of the hybrid automata model of the system is shown in Figure 11 where S₁ represents the initial state. The automata can be described in 6-tuple as follows;

1. (i)

   Control states, X={S1,S2,S3,S4,S5,S6,S7,S8,S9,S10,S11}

2. (ii)

   Plant symbol, ∑={x1,x2,x3,x4,x5,x6,x7,x8,x9,x10,x11,x12,x13,x14,x15,x16,x17,x18,x19,x20,x21,x22,x23,x24,x25,x26,x27,x28,x29}

3. (iii)

   Control symbol, ∑G={r1,r2,r3,r4,r5,r6,r7,r8,r9,r10,r11}

4. (iv)

   Transition function, f={f(S1,x1)=S6,f(S6,x2)=S7,f(S7,x3)=S8,f(S8,x4)=S7,f(S7,x5)=S6,f(S8,x6)=S6,f(S6,x7)=S8,f(S6,x8)=S11,f(S7,x8)=S11,f(S8,x8)=S11,f(S9,x8)=S11,f(S11,x9)=S2,f(S2,x10)=S3,f(S3,x11)=S4,f(S4,x12)=S3,f(S3,x13)=S2,f(S2,x14)=S4,f(S4,x15)=S2,f(S2,x16)=S10,f(S3,x16)=S10,f(S4,x16)=S10,f(S5,x16)=S10,f(S10,x17)=S6,f(S4,x18)=S5,f(S5,x19)=S4,f(S8,x20)=S9,f(S9,x21)=S8,f(S7,x22)=S9,f(S9,x23)=S7,f(S6,x24)=S9,f(S9,x25)=S6,f(S5,x26)=S3,f(S3,x27)=S5,f(S5,x28)=S2,f(S2,x29)=S5}

5. (v)

   Initial state, X₀= {S₁}

6. (vi)

   Final state, X_m= {S_i}

Fig. 11

Hybrid automata model.

Two types of symbols are used in the hybrid automata which are plant symbol and control symbol. These symbols are employed to represent the task assigned to the rehabilitation platform in changing the speed and direction of the gripper based on the emotion of the subject. There are eleven control states or modes described for the rehabilitation platform. Each of the modes represents the dynamic evolution of the system (i.e. speed, direction, and subject emotion) given different desired speeds and directions to follow during the rehabilitation session. The transition between modes is allowed once the guard condition (i.e the type of emotional state) is fulfilled.

The continuous set of platform dynamics is governed by the discrete type of emotion. The intention in controlling the speed and direction of the platform movement is to mimic the human therapist in changing the rehabilitation tasks based on the emotional state of the patient. Thus, the system is able to act more naturally in autonomous form.

The input to the system is the emotion, E. The position of the gripper, X(m) is acquired from the potentiometer reading, from which speed, ν of the gripper (cm/s) and the direction, dir of the gripper movement are derived. The wanted speed that is set for the gripper to move relies on upon the automaton that is activated by the separate feeling. It is expected that the subject is at default state when he/she is “calm” with a set speed of 0.833cm/s. For the other emotional states the speed is set appropriately. For instance, the most extreme speed of the platform is set when the subject is happy, since the patient could give greatest attention regarding the rehabilitation session whereas the base speed of the gripper is set when the subject is sad. For this emotional state, the subject may give practically zero concentration throughout the rehabilitation therapy session. The moderate speed is set if the subject is nervous in which case the subject give partial attention for the rehabilitation session.

The operation of the system is described by different tasks in which the linear actuator or gripper moves at different speeds and directions when the affected arm is under rehabilitation session. In describing the hybrid automata model, the control symbols or conditions of the modes, control states or modes, and plant symbols of the system are initialized in Table 1 to Table 3.

The sets of control states which represent the patient’s emotion and direction of the gripper movement is presented in Table 1.

S₁₀ and S₁₁ denote the control state that the patient arm has reached the home and goal position in the rehabilitation trajectory. For example, in the initial state, S₁ the gripper moves backward until it reached the home position at X=20 with the speed of 1cm/s from any previously rest position. The gripper then moves forward to the goal position located at X=5 with different speed based on the emotional state detected. The speed is set to decrease whenever the gripper is nearly to the end point in order to reduce the momentum before the gripper switches direction as depicted by S₁₀ and S₁₁. The robotic platform’s operation continues until a stop command is sent to the system. Each of the control symbols, r produce a unique rehabilitation treatment that mends to symbolize the different type of task assigned as listed in Table 2. The tasks assigned are different in terms of speed of the gripper’s movement solely based on the subject emotion.

The plant symbols listed in Table 3 are generated based on the information from position sensor (potentiometer) of the robotic platform and the emotion of the subject. It represents the possible evolution of system dynamics which is speed and direction of the gripper besides the subject emotion. The plant symbols will change from one plant symbol to another plant symbol only if it fulfills the requirement of the guard condition of the hybrid automata model. It then becomes the precondition before the transition between one control state to another control state can occur [31].

3.1. Case Study 1: Offline Simulation with Simulated Inputs

In the case studied, an offline simulation is conducted to analyze the functionality of the hybrid automata. A series of emotional states is set as the input to the hybrid automata system in ‘Repeating Sequence Stair’ form as shown in Eq. (3);

The period for each emotion state is fixed to 10s as correlated to a finding which claims that emotion is very brief in duration and can change in several seconds [32]. Figure 12 shows the result from the simulation. The subject is sad from point A to point B during a particular rehabilitation therapy. Since the location of the gripper is not reaching the home position (X=20), the speed of the gripper is set to 1cm/s with the backward motion until it reaches point C. The speed is set intentionally as part of the calibration process.

Fig. 12.

Simulation of emotion, state, position, and speed profiles for Case Study 1.

Thus the emotion detected during this time is ignored (i.e. state=S₁). Point C shows that the subject has reached the home position and switches the gripper direction from backward to forward motion with calm emotion (E=0).

The speed is set to -0.883cm/s and the active state is S₉. While performing the task, it is observed that the subject has shown a change in the emotion to nervous (E=2) at point D, thus the speed is adjusted to - 0.667cm/s and S₇ becomes the active state. At point E, the goal position (X=5cm) is reached to prepare the gripper to negate its motion into backward direction. Thus the state switches to S₁₁ to adjust the minimum speed value to 0.333cm/s in order to reduce the impact of gripper’s direction changing and some ‘cushion’ on the drastic change in the direction. At point F, the emotion remains nervous (E=2). Without delaying, the state switch to S₃ and the speed changes to 0.667cm/s. Up to point G, the emotion detected is happy (E=3) and this causes the state to switch to S₄, thus increases the gripper speed to 1.0cm/s. As it reaches the home position which is at X=20cm, the state S₁₀ is activated, thus negates the direction and sets the new speed to - 0.333cm/s as can be seen at point H. When the point I is reached, the state profile is automatically changed to S₆ and the speed is set to -0.5cm/s in the forward motion (E=1).

The case study proves that the developed hybrid automata system works perfectly in the simulation environment. The control framework is flexible and effective enough to allow additional or deletion of any state without the needs to redo major reprogramming.

3.2. Offline Experimentation with Simulated Inputs

For case study 2, the same automata developed in case study 1 is tested to analyze the adaptability of the rehabilitation platform with the hybrid automata developed. The sequence of emotion is given in ‘Repeating Sequence Stair’ form as shown in Eq. (4);

The period for each emotion state is fixed to 10s. Figure 13 captures the dynamics evolution between the emotion, the system state, position and speed of the system. For the first ten seconds, which is from point A to B, the emotion of the subject is denoted as 3 to indicate happy emotion. However, regardless of emotion, since the system is initializing, the gripper is moving backward to home position at X=20 from the previous resting position, with speed of 1.0cm/s. As the gripper reaches the home position at point B for the first time, the emotion is still happy (E=3), but the state is now switched to S₈ to prepare the gripper to move forward at speed of -1.0cm/s.

Fig. 13.

Simulation of emotion, state, position, and speed profiles for Case Study 2.

At point C while moving forward, the emotion is switched to nervous (E=2), thus the state switches to S₇ and the speed is made slower to -0.667cm/s. At point D, the emotion switches to calm (E=0), prompting the state to switch to S₉ and the speed to -0.883cm/s while the gripper is still moving forward until goal position. At point E, the goal position is reached so the state switches to S₁₁ to negate the speed value. In order to cushion the drastic change in the direction, the speed for this state is set to 0.333cm/s (the minimum speed). Now the gripper is moving backward to home position. At point F, the emotion detected is still calm (E=0) and this causes the state to switch to S₅, thus changing the speed to 0.833cm/s.

The state remains the same until at point G; the emotion detected changes to sad (E=1). It causes the speed and state changes to 0.5cm/s and S₂ respectively. At point H, the speed switches to 0.667cm/s indicates the input emotion is nervous (E=2). It continues until the home position is reached. As it reaches the position, the state S₁₀ is activated, thus negates the direction and sets the new speed of -0.333cm/s as can be seen at point I. At point J, the state switches to S₇ denoting the emotion detected is still nervous (E=2) and the speed of the gripper is set to -0.667cm/s.

At point K, the emotion changes to happy (E=3). Without delaying, the state switch to S₈ and the speed changes to -1.0cm/s. At point L, the emotion detected is sad (E=1) and this causes the state to switch to S₆, thus increases the gripper speed to -0.5cm/s. At point M, the goal position is reached to prepare the gripper to move backward. So the state switches to S₁₁ to negate the speed value of 0.333cm/s. Finally at point N, S₂ is activated (E=1) and the speed of the gripper is set to 0.5cm/s.

From the graph in Figure 13, the system is able to track the change of emotion and position of the gripper, and react accordingly by changing the corresponding state and speed of the system. Figure 14 shows the speed error between the simulated and experimented case study.

Fig. 14.

Error generated by the rehabilitation platform.

It can be seen the peak errors occur at the points where the direction of gripper changes due to the mechanism of the robot in changing the gripper’s speed on time via the hybrid automata’. The experimental and simulation results verify that the hybrid automata approach is practical for the Human Machine Interaction in rehabilitation applications.

3.3. Online Simulation vs Experimentation with Real Time Inputs

In case study 3, the rehabilitation system takes the emotional state input in real time. Instead of using ‘Repeating Sequence Stair’ as an array of the input emotions, a constant block called ‘emot’ is utilized in integrating the Simulink-Stateflow with rehabilitation platform and the Graphical User Interface (GUI) to infer the emotion of the subject. The function of ‘emot’ block is to set the hybrid automata input which is directly taken from the ‘hybrid automata code’ in GUI.

In this case, the initial emotion of the subject is sad (E=1) at the beginning of the therapy. The emotion of the subject converges from sad to nervous, nervous to calm, then back to nervous again before ends with happy emotion.

When the task execution starts, S₁ is triggered since the Stateflow is in initialization mode. The gripper is in backward motion with the desired speed of 1cm/s at point A as seen in Figure 15. When the subject’s arm has reached the home position (X=20cm) at point B, S₆ activated to guide the patient arm’s movement in forward motion. Simultaneously, the desired speed is decreased to -0.5cm/s. At point C, the gripper speed is set at -0.667cm/s and the active state is S₇. As the gripper reaches the goal position (X=5cm) at point D for the first time, the emotion detected is still nervous (E=2), but the state is now switched to S₁₁ to prepare the gripper to move backward at speed of 0.333cm/s. At point E, the emotion remains at nervous (E=2), prompting the state to switch to S₃ and the speed to 0.667cm/s.

Fig. 15.

Simulation of emotion, state, position, and speed profiles for Case Study 3.

While performing the task, it is observed that the subject has shown a change in the emotion to nervous (E=0) at point F, thus the speed adapts the changes to 0.883cm/s and S₅ becomes the active state. At point G, the state switches to S₃ denoting the emotion detected is still nervous (E=2) and the speed of the gripper is set to 0.667cm/s. At point H, the speed changes to a negative value of -0.333cm/s which is in S₁₀. It symbolizes that the subject’s arm has reached the home position and the gripper direction is changed to forward motion again. At point I, the state switches to S₇ denoting the emotion detected is still nervous (E=2) thus the gripper changes the direction and the speed is set to 0.667cm/s. The subject is finally observed to be happy (E=3) in performing the task due to change of state as can be seen at point J. It activates S₈ with the speed of -1.0cm/s.