The Research about Editing System of Performance Information for Player Piano
- DOI
- 10.2991/jrnal.k.200512.015How to use a DOI?
- Keywords
- Automatic piano; knowledge database; computer music; music interface
- Abstract
We have developed a system that allows a piano to perform automatically. In order to play music in the manner of a live pianist, we must add expression to the piano’s performance. In the case of music, there are often 1000 or more notes in the score, requiring that an editor spend a huge amount of time to edit. Therefore, we have developed an interactive musical editing system that utilizes a database to edit music more efficiently.
- Copyright
- © 2020 The Authors. Published by Atlantis Press SARL.
- Open Access
- This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).
1. INTRODUCTION
We have developed a system that allows a piano to perform automatically [1]. In this system, 90 actuators have been installed on the keys and pedals of a grand piano. These actuators execute key strokes and pedal movements to govern the piano’s performance, e.g., “Figure 1 (Continued)” [2,3].
The automatic piano.
Our research focuses on giving player piano performance with human-like skill and expression. Playing the piano expressive by player piano, it is necessary to adjust the volume, length, and timing of music. We aimed to develop a system that, like a skilled pianist, can perform even the first musical score based on information related to previous skills and experience. In this paper, we developed a system that automatically estimates the performance expression of unedited music using edited performance data and score data. We described a phrase search using Dynamic Programming (DP) matching and a method for selecting an optimal phrase for inference, and evaluated the entire song [4].
2. EDITING SUPPORT SYSTEM
2.1. Performance Information
The automatic piano that we have developed uses a music data structure that is similar to Musical Instrument Digital Interface (MIDI). We defined performance information, dividing it into two categories: the notes and the pedals. The note information is comprised of the six parameters involved in producing a tone: “Key” (note), “Velo” (velocity), “Gate”, “Step”, “Bar” and “Time”. “Velo” is the dynamics, given by the value of 1–127. “Gate” is the duration of the note in milliseconds. “Step” is the interval of time between notes, and it also exhibits tempo. “Bar” is the vertical line placed on the staff to divide the music into measures.
The pedal information consists of four parameters. Specifically, there are four parameters: “key” (“Damper” or “Shifting”), which indicates the type of pedal, “Velo”, which indicates the amount of pedaling, “Time”, which indicates the time to pedal, and “Bar”, which indicates the bar number.
2.2. How to Make the Data for Player Piano
We show the structure of the edit system how to make the data for player piano in “Figure 2 (Continued)”. When we have the music data A that the pianist has played, and do not have the music data B that the pianist has played, we extract a characteristic of the pianist from music data A. Then we make the music data B’s musical performance by doing reasoning from characteristic of musical performance and information of musical grammar and score of music B.
The structure of the edit system.
2.3. Search System
As a result of the analysis, it was found that phrases of the same pattern existing in the same tune are performed in a similar expression (Figure 3) [5]. This time, we used DP matching to search for similar phrases.
The discovery about the relation of same tunes and similar expression.
DP matching is a technique used widely in the field of speech recognition, bioinformatics and so on. It has a feature that can calculate the similarity between two words that are different in a number of characters from each other. In Figure 4, the route of minimum cost in each point is taken, and the route with the lowest cost is assumed finally to be the optimal path. The cost at that time is defined as the distance between patterns. In this system, this distance is handled as a threshold to judge whether the phrases are similar to each other.
Result of dynamic programming matching method.
2.4. Selection System
After doing the Search system, there are a lot of similar phrases which has the same DP matching points. Similar phrases are phrases we found by doing DP matching from music data. Objective phrase is phrase we want to inference in this time. Therefore, we need to select the best phrases for objective phrases by doing the Selection system. In this year’s system, three indicators were used from the viewpoint of music theory: Dynamic symbols, Beats and Steps, and Musical forms.
2.5. Dynamic Symbols
If the Dynamic symbols are different between similar phrase and objective phrase, the performance is affected even if the phrase is the same, and the search phrase and the similar phrase that matches the dynamics on the score are preferentially selected.
2.6. Beat and Step
Based on the musical grammar, it is known that strong beats are closely related to Step [6]. In places considered to be strong beats, similar phrases are selected using the property that the Step value is larger than in other places. The position of the strong beat depends on the rhythms.
2.7. Musical Forms
If the Selection system in “Subsections 2.5 and 2.6” does not narrow down the number of similar phrases, the selection is performed according to the Musical Forms. Musical Forms is a music format such as the Rondo Forms or the Sonata Forms. For example, if different songs were in the sonata format, it was predicted that the presentation part had a presentation part feature, the development part had a development part feature, and each part had a unique feature. We decide same part has similar performance.
3. INFERENCE SYSTEM
3.1. System Overview
The Velo value that the performer is executing for the dynamic symbol on the score is examined, and Velo suitable for the optimal phrase is inferred from the search phrase.
3.2. Inference by the Strength Symbol
In order to obtain the characteristic of the change of the Velo value due to the influence of the intensity sign, the tendency was examined from the performance data of the pianist. The pianist of the surveyed performance data is Gerhard Opitz, and its title is shown in Table 1 [7].
| Title of musical compositions | |
|---|---|
| ① | Sonate fur Klavier Nr.8 c-moll Op.13-1 “Grande Sonate Pathetique” |
| ② | Sonate fur Klavier Nr.8 c-moll Op.13-2 “Grande Sonate Pathetique” |
| ③ | Sonate fur Klavier Nr.8 c-moll Op.13-3 “Grande Sonate Pathetique” |
| ④ | Sonate fur Klavier Nr.23 f-moll Op.57-1 “Appassionata” |
The studied title of musical compositions
As a result, the definition of Velo with detailed strength varied depending on the song, but the one with the high frequency of each Velo value was adopted as the strength symbol, and the results are summarized in Table 2.
| Dynamic mark | ppp | pp | p | mp | mf | f | ff |
|---|---|---|---|---|---|---|---|
| Velo | 15–20 | 20–45 | 45–65 | 65–75 | 75–80 | 80–95 | 95–115 |
The Velo value for a composite dynamic mark
Inference using Table 2 and Velo’s inference formula for the xth sound are shown in Equations (1) and (2). SVelo refers to the optimal phrase Velo.
4. INFERENCE EXPERIMENT
4.1. Experimental Method
An experiment was conducted to compare the music reproduced using the editing support system with the performance of the pianist. The target song is Beethoven Piano Sonata No.14, Op.27-2(“Moonlight Sonata”). Figure 5 shows an example of a search phrase.
Search phrase.
4.2. Experimental Result
Using graphs and correlations, we compared inferred phrases with actual performance. The Search system found 18 similar phrases. Table 3 shows the 10 data selected by Dynamics symbols. ○ indicates that the numerical value is 80% or more of the maximum value, and × indicates that the value is 40% or less, and △ indicates neither of them. From Table 3, the optimal phrase was selected as the 1st number 226. For confirmation, Table 4 shows the correlation coefficient between each similar phrase and actual performance.
Selecting method by using Dynamics symbols
The Velo value for a composite dynamic mark
4.3. Consideration
Table 4 shows that the phrase selected by the selection system is closer to the actual performance than other similar phrases (Figure 6). From Figure 7, it was confirmed that the Velo was closer to the actual performance. Thus, the selection system and inference system newly added this year are considered to be effective.
Comparing select phrase to pianist’s performance data.
Comparing select phrase to inferred select phrase to pianist’s performance data.
5. CONCLUSION
This year, we introduced a selection system that selects the most suitable phrases and an inference system that infers the Velo values that are likely to be performed by the pianist, to the editing support system. The inference experiment showed that the performance expression of the unedited piece could be inferred. This selection system is based on concepts that are faithful to music theory. In reality, however, not all pianists play faithfully to the strength and weakness symbols on the score.
Therefore, the current system of inference based on indicators such as strength and weakness symbols cannot be used to infer the case of performers who do not have strength and weakness symbols in their scores, such as Bach. In order to solve this problem, we need to consider human playing from a new perspective. After that, we will develop a new system that can select the best phrases for classical music of all era.
CONFLICTS OF INTEREST
The authors declare they have no conflicts of interest.
AUTHORS INTRODUCTION
Ms. Mami Ezaki
She is a Master course student in at the Department of Intelligent and Control Systems Kyushu Institute of Technology, Japan. She received her Bachelor’s degree from the Department of Mechanical Information Science and Technology Kyushu Institute of Technology, Japan in 2019.
Prof. Eiji Hayashi
He is a Professor in the Department of Intelligent and Control Systems at Kyushu Institute of Technology. He received the PhD (Dr. Eng.) degree from Waseda University in 1996. His research interests include Intelligent mechanics, Mechanical systems and Perceptual information processing. He is a member of The Institute of Electrical and Electronics Engineers (IEEE) and The Japan Society of Mechanical Engineers (JSME).
REFERENCES
Cite this article
TY - JOUR AU - Mami Ezaki AU - Eiji Hayashi PY - 2020 DA - 2020/05/20 TI - The Research about Editing System of Performance Information for Player Piano JO - Journal of Robotics, Networking and Artificial Life SP - 73 EP - 76 VL - 7 IS - 1 SN - 2352-6386 UR - https://doi.org/10.2991/jrnal.k.200512.015 DO - 10.2991/jrnal.k.200512.015 ID - Ezaki2020 ER -