Automatic Music Transcription Using Fourier Transform for Monophonic and Polyphonic Audio File

The music industry has become popular and continues to evolve across the world. Even in difficult situations (like Covid-19 Pandemic), the industry implements different approaches using the available technology to interact with the consumers [1]. Prior to digitalization, distribution of music is even boarder [2]. Besides can be enjoyed through the audio, music can be represented in a sheet of a paper. Musical sheet is an important tool for musicians that enables musicians to communicate with each other [3]. With musical sheet, anyone can learn a composition of a song, but not every song has the musical sheet or it may require payment to get the sheet.

Music Transcription is the process of music information retrieval to produce musical sheet. By definition, Music Transcription is listening to the song and write it down in musical notation [4]. To perform Music Transcription manually, a person must have certain knowledge and it requires a lot of time to translate a song into notation. While Music Transcription process done manually, Automatic Music Transcription (AMT) is the Music Transcription process using computational technique.

The proposed method in this study wants to use Fourier Transform to perform pitch detection system by separating audio file into windows systematically using the duration of the notes. The paper will break down the whole step and experiment in a clear and visible state. The system is based on Note Value Detection and Pitch Detection. In this study, Note Value Detection can be obtained from the perceived pulse of the signal to estimate the duration of each note (note value / note length) in the audio file. Note Value Detection algorithm needs to be performed because the result of the algorithm can be used to split the signal into multiple windows, so the system can have windows contained signals ordered by time. Pitch Detection in this research is using Fast Fourier Transform on each window to obtain the frequencies and store it to the data collection. The final step of the method is translating the data collection into musical notation as the presentation.

Evaluation of the system will be performed using the pair of musical sheet and audio file. The musical sheet will act as the ground truth in the array form. The audio file will act as the input of the system that produce the list of note name and note value. The evaluation value will be obtained by comparing the ground truth and the result.

3.1 Music frequency

Technically, music consists of pitches at specific frequencies [17]. Music can be created based on vibrations that can be expressed in Hertz (Hz). Frequency de-scribes the number of waves (or vibrations) at a particular time, so 1 Hz means 1 wave per second. The number of waves in second represents how high or low a sound. High tones correspond to high frequencies and low tones correspond to low frequencies as shown in Table 1.

Table 1. Note frequencies for standard tuning notes

The frequencies of the notes are actually mathematically related and are determined around the center note A₄ [18]; 440 Hz. The association between these frequencies is strongly related to compound interest. The frequency will double every 12 notes, when the key is the same but in a different octave. Therefore, the relation between the frequencies, can be written in Eq. (1) using the half steps away from A₄ denoted by n [17].

Note frequency $(\mathrm{Hz})=440 \times 2^{\frac{n}{12}}$     (1)

3.2 Onset detection

Onset detection is the process of determining the beginning of each musical notes in the audio file [19]. The onset of the note is the time when there is a sudden increase of energy at the beginning of the note as shown in Figure 1. First, this method computes the threshold spectral flow operation on the audio spectrogram and give a list that represents the amount of spectral energy that increases at each frame. The method continuous to select peak positions from the onset strength curve and give a list of onsets location.

13._fig1.png

Figure 1. Onset of a single note

3.3 Fourier transform

Fourier Transform is a method to decompose or divide the signal into a sinusoidal sum of different frequencies [20]. Fourier transform is able to show the amplitude and frequency of each of the determined sinusoids. The frequency domain of the Fourier Trans-form contains exactly the same information as the original function.

Discrete Fourier Transform offers a potential method of analysis in the digital computer. However, the trans-formation shows that if there are N data in the time function and the amplitude of the N individual sinusoid is calculated, then calculation time is proportional to N². Even with a high-speed computer, it requires extra machine time to compute Discrete Fourier Trans-form of a large N. Therefore, Discrete Fourier Trans-form cannot be used for audio analysis, because audio contains a very large size of N.

In 1965 Cooley and Tukey published their mathematical algorithm called Fast Fourier Transform. Fast Fourier Transform is the advancement of method using the Discrete Fourier Transform. Fast Fourier Transform re-duces the calculation time to Nlog₂N. This increase in calculation speed has completely revolutionized many aspects of scientific analysis.

Brigham explained that Fast Fourier Transform algorithm separates Discrete Fourier Transform into two parts as shown in Eq. (2); Even Part (n=2m) and Odd Part (n=2m+1) [20]. Then, using factorization, the equation can be written in Eq. (3). Later, the Even Part represented by the Eq. (4) and the Odd Part represented by the Eq. (5). Therefore, the Eq. (3) can be written in Eq. (6).

$D(k)=\sum_{m=0}^{m=N / 2-1} d(2 m) e^{-i \frac{2 \pi k(2 m)}{N}}+$$\sum_{m=0}^{m=N / 2-1} d(2 m+1) e^{-i \frac{2 \pi k(2 m+1)}{N}}$       (2)

$D(k)=\sum_{m=0}^{m=N / 2-1} d(2 m) e^{-i \frac{2 \pi k(2 m)}{N}}+$$e^{-i \frac{2 \pi k}{N}} \sum_{m=0}^{m=N / 2-1} d(2 m+1) e^{-i \frac{2 \pi k(2 m)}{N}}$         (3)

$E(k)=\sum_{m=0}^{m=N / 2-1} d(2 m) e^{-i \frac{2 \pi k(2 m)}{N}}$      (4)

$O(k)=\sum_{m=0}^{m=N / 2-1} d(2 m+1) e^{-i \frac{2 \pi k(2 m)}{N}}$    (5)

$D(k)=E(k)+e^{-i \frac{2 \pi k}{N}} O(k)$        (6)

To optimize the algorithm, Brigham explained that for index k+N/2 can be calculated separately using Eq. (7). Using the calculation of Euler exponential, the equation for index k+N/2 can be written in the Eq. (8). In the Eq. (8), we can find the equation of Even Part (Eq. (4)) and Odd Part (Eq. (5)). Therefore, the equation can be written in Eq. (9).

$D\left(k+\frac{N}{2}\right)=\sum_{m=0}^{m=N / 2-1} d(2 m) e^{-i \frac{2 \pi\left(k+\frac{N}{2}\right)(2 m)}{N}}+$$e^{-i \frac{2 \pi\left(k+\frac{N}{2}\right)}{N}} \sum_{m=0}^{m=N / 2-1} d(2 m+1) e^{-i \frac{2 \pi\left(k+\frac{N}{2}\right)(2 m)}{N}}$        (7)

$D\left(k+\frac{N}{2}\right)=\sum_{m=0}^{m=N / 2-1} d(2 m) e^{-i \frac{2 \pi k(2 m)}{N}}-$$e^{-i \frac{2 \pi k}{N}} \sum_{m=0}^{m=N / 2-1} d(2 m+1) e^{-i \frac{2 \pi k(2 m)}{N}}$        (8)

$D\left(k+\frac{N}{2}\right)=E(k)-e^{-i \frac{2 \pi k}{N}} O(k)$      (9)

5.1 Evaluation metrics

The model of this research will be evaluated using quantitative measurement. The evaluation method will be performed through the experiments with the music file. The evaluation metrics will be performed to calculate the accuracy for each music file using the modification of the evaluation metrics of Nakamura et al. [21]. The purpose of the evaluation is to see how well the system transcribe the audio into the digital format of music. There are three evaluation metrics that can be used to obtain the accuracy of the proposed system by comparing the original musical information with generated musical information in linear manner (window by window). All of the metrics are quantitative measurements that will give a percentage error rate; the smaller the error, the more accurate the system. There are three variables can be used for the accuracy of music transcription; note values, note accuracies, and noises. Note value and note accuracies are important in music transcription, different duration of the notes can change the whole song. Noise detection in music transcription is important too, because the extra note can cause confusion to the listener.

Note value can be evaluated by comparing note values (note lengths) of the original musical information with the generated musical information. The comparation of note values will be evaluated by calculating the number of incorrect note values of the generated musical sheet. Then the result of the number of incorrect note values will be divided the size of original musical sheet. The formula to evaluate note value can be written in Eq. (10).

Note value error rate $=$$\frac{\text { number of incorrect note values }}{\text { size of musical sheet }}\quad \times 100 \%$     (10)

Note accuracy can be evaluated by comparing the window of the original musical sheet with the window of the generated musical sheet one by one. First, the metric will calculate number of undetected pitches in the generated window then divided with total pitches in the original window. The calculation will be performed to each window and will be added up together and then will be divided by the size of original musical sheet. The formula to evaluate note accuracy can be written in Eq. (11).

Pitch error rate $=$$\frac{\sum \frac{\text { number of undetected notes }}{\text { number of original notes }}}{\text { size of musical sheet }}\quad \quad\times 100 \%$ (11)

Noises accuracy can be evaluated by comparing the window of the original musical sheet with the window of the generated musical sheet one by one. First, the metric will calculate number of detected noises in the generated window then divided with total pitches in the generated window. The calculation will be performed to each window and will be added up together and then will be divided by the size of original musical sheet. The formula to evaluate noises can be written in Eq. (12).

Extra note error rate $=$$\frac{\sum \frac{\text { number of extra notes }}{\text { number of detected notes }}}{\text { size of musical sheet }} \quad\quad\times 100 \%$    (12)

5.2 Evaluation file

Evaluation of the system will be performed using 5 pairs of musical sheet and audio file. The list of the songs can be seen in Table 2. These songs are selected for the experiments because all of the songs are common, so the examiners can understand the result better by comparing with the original audio file. All of the songs have two version of file; monophonic and polyphonic. So, in total there will be 10 musical sheets and audio files.

Table 2. Songs for evaluation

The data that will be stored in the array for evaluation is the conversion of the musical sheet. For the example of monophonic song, the conversion of the musical sheet as shown in Figure 3 can be represented in a list as shown in Table 3. Since monophonic music is only containing one note at a time, the filter variable will be assigned to find the dominant note at a time.

Evaluation of polyphonic music files will be performed using 5 just like the monophonic music, the evaluation is also using the songs in Table 2. but this time the songs will be polyphonic. Like the monophonic music evaluation, the data that will be stored in the array for evaluation is the conversion of the musical sheet. For the example, the conversion of the musical sheet as shown in Figure 4 can be represented in a list as shown in Table 4.

13.fig3_.jpg

Figure 3. London bridge is falling down monophonic musical sheet

13._fig4.png.jpg

Figure 4. London bridge is falling down polyphonic musical sheet

Table 3. London bridge is falling down monophonic list representation

Table 4. London bridge is falling down polyphonic list representation

5.3 Evaluation result

Table 5 shows the error rates for the experiments using 10 audio files (monophonic songs and polyphonic songs). Note Value Error Rate evaluates the accuracy of note value. Pitch Error Rate evaluates the accuracy of the pitch, if the error rate equals to 0% then all the pitches are detected and vice versa. Extra Note Error Rate evaluates the accuracy of the pitch too, if the error rate equals to 0% then there are no extra notes detected in the system, but if the error rate is greater than 0% then there are unwanted pitches detected in the system.

Table 5. Evaluation result

The result shows that Note Value Error Rates give a relatively small percentage error. The incorrectly detected note value in the songs appears in the end note, because the system can’t determine when the note ended. Pitch Error Rates still occurred in the relatively small percentage error, it could happen because the system failed to detect the lower pitch notes in “Happy Birthday to You” polyphonic audio file, and in “Ode to the Joy” polyphonic audio file happened because the octave error. Extra Note Error Rate occurred in every polyphonic file; the extra notes happened a lot because there is the same note that happened in other octaves.