Music Recommendation System Driven by Physiological Evaluation Appearing in Fingertip Skin Temperature
- DOI
- 10.2991/jrnal.k.210521.011How to use a DOI?
- Keywords
- Music recommendation system; music therapy; peripheral skin temperature
- Abstract
In recent years, music therapy has been performed with the aim at recovering the cognitive function of elderly people. In the previously reported system, it was necessary to input the subjective evaluation of the user to determine the next recommended song. In this study, variations in fingertip skin temperature were used as input instead of subjective evaluation, focusing on evaluation of emotion expressed as physiological response through variations in fingertip skin temperature. Our system can be used with no support for users.
- Copyright
- © 2021 The Authors. Published by Atlantis Press B.V.
- Open Access
- This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).
1. INTRODUCTION
In recent years, against the backdrop of increasingly aging societies in developed countries such as Japan, activities using music for the elderly (“music therapy” [1,2]) have been used with the aim of improving recognition ability. We previously developed music recommendation systems for music therapy for older people [3,4].
However, in the previously reported music recommendation system [3], it was necessary to input the subjective evaluation of the user (“I want to listen to it again” or “I don’t want to listen to it again”) into the recommended music in order to determine the next recommended song. Therefore, in the case of a user who has difficulty in responding by himself/herself, such as a dementia patient, the support of a caregiver may be required, and the problem of securing human resources hinders the use of our previously reported system [3].
In this study, we focused on the evaluation of emotions by measuring peripheral skin temperature [2], developed a method for determining recommended songs based on the physiological evaluation of the user, and evaluated our system through the experiments.
2. RELATED STUDIES
We now briefly explain the systems used in our previous study [3] and another study [5], which we build upon to evaluate emotions in this study.
2.1. Previously Reported System for Music Recommendation
The method for music recommendation used in this study is based on a previously reported methods [3]. Similar to the previously reported music recommendation system [3], our system is a hybrid system for improving recognition ability using collaborative filtering and impression words [6]. See the paper [3] for details on the music recommendation module in our system. See the paper [7] for details on the music recommendation using impression words in our system.
2.2. Evaluation of Emotions by Measuring Peripheral Skin Temperature
In the study by Watanabe et al. [5], the subject watched a horror video and an environmental video alternately, and the peripheral skin temperatures of the subjects viewing the videos were measured. Three peripheral parts, the front of auricle, the nose, and the back of the right index finger were measured, and temperature changes were compared and examined. No significant change was observed in the skin temperature on the front of the auricle and on the nose. However, the skin temperature on the back of the fingers of the right hand decreased when viewing the horror video and increased when viewing the environmental video. From this result, the influence of autonomic nerve activity appears to be reflected strongly in the peripheral skin temperature of the fingertip as a change in blood flow due to vasoconstriction.
3. CONSTRUCTION OF A PHYSIOLOGICAL EVALUATION METHOD FOR MUSIC
A physiological evaluation method for music, which measures the peripheral skin temperatures of subjects listening to music, will be described. In this study, we selected the pad of the left index finger as the peripheral part for measuring temperature.
3.1. Changes in Peripheral Skin Temperature while Listening to Music
To evaluate emotions by measuring peripheral skin temperature in a music recommendation system, we investigated the changes in peripheral skin temperature of subjects listening to music.
When subject A (a female in her 20s) listened to her favorite song (“Canon,” composed by Johann Pachelbel) and least favorite song (“Mystery III -Heterophony for Swirl Butoh -,” composed by Akira Nishimura), the peripheral skin temperature tended to rise while listening to her favorite song and decreased while listening to her least favorite song (Figures 1 and 2) [6].
Changes in peripheral skin temperature when listening to music (Favorite song).
Changes in peripheral skin temperature when listening to music (Least favorite song).
3.2. Determination of Threshold for Changes in Peripheral Skin Temperature
Similar to the previously reported music recommendation system [3], the system used in this study determines the next recommended song by inputting an evaluation value (“1” if you want to listen to the song again, or “0” if you don’t want to listen to the song again) into a computer. In this study, we determined the threshold value for dividing “1” when “physiologically preferable” and “0” when “physiologically unfavorable” based on the change in peripheral skin temperature.
Based on the results of the preliminary experiment, 0.4°C (the temperature difference between the maximum temperature and the subsequent minimum temperature) and 180s (the elapsed time since the subject had begun listening to music) were set as the threshold values [3].
3.3. Binarization of Physiological Evaluation
Using the two thresholds described in Subsection 3.2, the music being listened to was judged to be “Physiologically favorable (1)” or “Physiologically unfavorable (0)” based on the temperature change of the peripheral skin when listening to the music. Figure 3 shows the judgment processing flow for the physiological evaluation of music. In Figure 3, the elapsed time from the start of measurement is indicated by “time,” and the measured temperature at that time is indicated by “temp.” The maximum temperature from the start of measurement to that point is indicated as “max,” and the difference between “max” and “temp” is indicated as “dif.”
Flowchart for binarizing physiological evaluation [6].
4. EXPERIMENTAL EVALUATION
4.1. Conditions
The music recommendation system [6] used in this study is an improved version of the previously reported system [3]. The improved system can support multiple music databases, and is aimed at the elderly. It was reported that elderly people tend to choose songs that are the subject of elementary school music textbooks as familiar songs (favorite songs) [8]. Therefore, we selected a CD described as an anthology of older songs enjoyed by older people with dementia, and selected 52 songs on the CD that were also included in a music textbook database for elementary schools as well as 58 popular songs [9]. There were a total of 10 subjects: four females in their 20s (Subjects A–D), five males in their 20s (Subjects E–I), and one male in his 60s (Subject J). For programming, we used Visual C++ 6.0 (Microsoft) on a PC (Dell Optiplex790, CPU: Intel Corei7-2600 3.40 GHz, main memory: 4.00 GB, and OS: Windows 7 Professional, Microsoft) for the experiment. We used a digital temperature and relative humidity sensor SHT11 [10] to measure temperature.
4.2. Results and Discussion
4.2.1. Concordance rate
Table 1 shows the results of the concordance rate between the physiological evaluation and the subjective evaluation. The concordance rate of all experimental results was relatively high, with an average of 70.9%. Individually, the average for children’s song was 67.3%, while that for popular songs was 74.5%. However, since there is a large difference in the concordance rate between subjects (Table 1), improvements such as reviewing the threshold are considered necessary. In addition, depending on the subject, the concordance rate of either the children’s song or the popular song is high, and the concordance rate of the other is low. Therefore, it would appear that there are some problems that cannot be solved only by adjusting the threshold value.
| Subject | A | B | C | D | E | F | G | H | I | J | Total | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Children’s songs | Number of recommended songs | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 150 | |||||||||||
| Evaluation value | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | |
| Physiological evaluation | 8 | 7 | 5 | 10 | 6 | 9 | 6 | 9 | 7 | 8 | 9 | 6 | 8 | 7 | 10 | 5 | 8 | 7 | 11 | 4 | 78 | 72 | |
| Subjective evaluation | 9 | 6 | 8 | 7 | 14 | 1 | 9 | 6 | 8 | 7 | 5 | 10 | 7 | 8 | 11 | 4 | 5 | 10 | 13 | 2 | 89 | 61 | |
| Number of concordances: Individual | 8 | 6 | 3 | 5 | 6 | 1 | 5 | 5 | 4 | 4 | 5 | 6 | 7 | 7 | 8 | 2 | 2 | 4 | 11 | 2 | 59 | 42 | |
| Number of concordances: All | 14 | 8 | 7 | 10 | 8 | 11 | 14 | 10 | 6 | 13 | 101 | ||||||||||||
| Concordance rate (%) | 93.3 | 53.3 | 46.7 | 66.7 | 53.3 | 73.3 | 93.3 | 66.7 | 40 | 86.7 | 67.3 | ||||||||||||
| Popular songs | Number of recommended songs | 8 | 11 | 10 | 10 | 9 | 8 | 8 | 8 | 9 | 12 | 93 | |||||||||||
| Evaluation value | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | |
| Physiological evaluation | 3 | 5 | 3 | 8 | 9 | 1 | 8 | 2 | 6 | 3 | 1 | 7 | 3 | 5 | 6 | 2 | 6 | 3 | 10 | 2 | 55 | 38 | |
| Subjective evaluation | 3 | 5 | 6 | 5 | 10 | 0 | 8 | 2 | 2 | 7 | 2 | 6 | 5 | 3 | 7 | 1 | 5 | 4 | 9 | 3 | 57 | 36 | |
| Number of concordances: Individual | 3 | 5 | 1 | 3 | 9 | 0 | 7 | 1 | 2 | 3 | 1 | 6 | 2 | 2 | 6 | 1 | 4 | 2 | 9 | 2 | 44 | 25 | |
| Number of concordances: All | 8 | 4 | 9 | 8 | 5 | 7 | 4 | 7 | 6 | 11 | 69 | ||||||||||||
| Concordance rate (%) | 100 | 36.4 | 90 | 80 | 55.6 | 87.5 | 50 | 87.5 | 66.7 | 91.7 | 74.5 | ||||||||||||
Experimental results [6]
4.2.2. Correct answer rate of music recommendation
Table 2 shows a comparison of the percentage of correct answers for the song recommendation results. The percentage of correct answers for the music recommendations was relatively low. The probable reason for this was that many of the subjects were relatively young, not elderly people who were targeted by the music database used in this study. As an example, a male in his 60s (Subject J) had a relatively high percentage of correct answers (73.3–83.3%) in both the physiological and subjective evaluations. Figure 4 shows the change in peripheral skin temperature for the song that Subject J judged to be his “favorite song” in both the physiological and subjective evaluations. The peripheral skin temperature gradually increased over time (Figure 4).
| Subject | A | B | C | D | E | F | G | H | I | J | Total | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Children’s songs | Physiological evaluation (%) | 53.3 | 33.3 | 40 | 40 | 46.7 | 60 | 53.3 | 66.7 | 53.3 | 73.3 | 52 |
| Subjective evaluation (%) | 60 | 53.3 | 93.3 | 60 | 53.3 | 33.3 | 46.7 | 73.3 | 33.3 | 86.7 | 59.3 | |
| Popular songs | Physiological evaluation (%) | 37.5 | 27.3 | 90 | 80 | 66.7 | 12.5 | 37.5 | 75 | 66.7 | 83.3 | 57.6 |
| Subjective evaluation (%) | 37.5 | 54.5 | 100 | 80 | 22.2 | 25 | 62.5 | 87.5 | 55.6 | 75 | 60 |
Recommendation accuracy [6]
Changes of subject J in peripheral skin temperature when listening to music (Favorite song).
Subject J (a male in his 60s) had a high rate of accuracy for both children’s songs and popular songs. From this, the selection of the music database is considered to be one of the factors that contributed to the low recommendation accuracy for one of the subjects who were in their 20s. In future research, we plan to conduct experiments that focus on the elderly.
5. CONCLUSION
By focusing on the evaluation of emotions by measuring the peripheral skin temperature, we have developed a music recommendation system that inputs subjective evaluations using peripheral skin temperature. This system can be used with no support for users. The concordance rate between the physiological evaluation and the subjective evaluation was relatively high, with an average of 70.9% in the present experiments. The concordance rate varied depending on the subject and the music genre. One of the factors is that the threshold is set for one subject. In addition to music, there are many factors that affect peripheral skin temperature, such as room temperature and the physical condition of the subject. In the present experiments, the 90% subjects were 20s. In future research, we plan to conduct experiments that focus on the elderly, reviewing the threshold in our system.
CONFLICTS OF INTEREST
The authors declare they have no conflicts of interest.
ACKNOWLEDGMENT
We would also like to thank the subjects of our experiments for their cooperation.
AUTHORS INTRODUCTION
Ms. Mayuka Wada
She received her B.S. degree from Kyoto Prefectural University in 2019. She works at Shimadzu System Development Corp.
Dr. Taro Asada
He received his B.S., M.S. and PhD degrees from Kyoto Prefectural University in 2002, 2004 and 2010, respectively. He works as an Associate Professor at the Graduate School of Life and Environmental Sciences of Kyoto Prefectural University. His current research interests are human interface and image processing. He is a member of HIS and IIEEJ.
Dr. Yasunari Yoshitomi
He received his B.E., M.E. and PhD degrees from Kyoto University in 1980, 1982 and 1991, respectively. He works as a Professor at the Graduate School of Life and Environmental Sciences of Kyoto Prefectural University. His specialties are applied mathematics and physics, informatics environment, intelligent informatics. He is a member of IEEE, HIS, ORSJ, IPSJ, IEICE, SSJ, JMTA and IIEEJ.
Dr. Masayoshi Tabuse
He received his M.S. and PhD degrees from Kobe University in 1985 and 1988 respectively. From June 1992 to March 2003, he had worked in Miyazaki University. Since April 2003, he has been in Kyoto Prefectural University. He works as a Professor at the Graduate School of Life and Environmental Sciences of Kyoto Prefectural University. His current research interests are machine learning, computer vision and natural language processing. He is a member of IPSJ, IEICE and RSJ.
REFERENCES
Cite this article
TY - JOUR AU - Mayuka Wada AU - Taro Asada AU - Yasunari Yoshitomi AU - Masayoshi Tabuse PY - 2021 DA - 2021/05/31 TI - Music Recommendation System Driven by Physiological Evaluation Appearing in Fingertip Skin Temperature JO - Journal of Robotics, Networking and Artificial Life SP - 47 EP - 51 VL - 8 IS - 1 SN - 2352-6386 UR - https://doi.org/10.2991/jrnal.k.210521.011 DO - 10.2991/jrnal.k.210521.011 ID - Wada2021 ER -