Mood Based Music Categorization System for Bollywood Music

Abstract

Music shares a very special relation with human emotions. People often choose to listen to a song or music which best fits their mood at that instant. In spite of this strong correlation, most of the music applications today are devoid of providing the facility of mood-aware playlist generation. The authors wish to contribute to automatic identification of mood in audio songs by utilizing their spectral and temporal audio features. Their current work involves analysis of various features in order to learn, train and test the model representing the moods of the audio songs. The focus of their work is on the Indian popular music pieces and continues to analyse, develop and improve the algorithms to produce a system to recognize the mood category of the audio files automatically.

Headnote

Abstract

Music shares a very special relation with human emotions. We often choose to listen to a song or music which best fits our mood at that instant. In spite of this strong correlation, most of the music applications today are devoid of providing the facility of mood-aware playlist generation. We wish to contribute to automatic identification of mood in audio songs by utilizing their spectral and temporal audio features. Our current work involves analysis of various features in order to learn, train and test the model representing the moods of the audio songs. Focus of our work is on the Indian popular music pieces and our work continues to analyse, develop and improve the algorithms to produce a system to recognize the mood category of the audio files automatically.

Keywords

Music; Mood; Classification; mood detection.

(ProQuest: ... denotes formulae omitted.)

1. Introduction

In terms of music, we have our interests, favourite artists, albums and music type. Lot of variety is available and our choice differs from time to time. Currently classification is based on tags like Artist, Band (Group), Album, Movie, Year, Bitrate and Genre. Selecting a song or music piece suiting our mood from a large database is difficult and time consuming, since each of the mentioned parameter cannot sufficiently convey the emotional aspect associated with the song. We often select a song or music which best suites our mood at that instant. In spite of this strong correlation, most of the music applications present today are still devoid of providing the facility of mood-aware playlist generation.

What is needed is an additional parameter or rather search filter, in this case "Mood", which signifies the emotion of that particular music piece. Hence a pattern can be obtained for the same. This pattern can help us to determine the given parameter i.e. Mood. Some music patterns represents contentment or relaxing, while some others make an individual feel anxious or frantic. Hence, it is possible to build a mood detection system in a concrete environment, for example, for classical music in western culture.

The relation between mood and music, Music emotion detection and classification has been extensively studied and researched earlier. Mostly pattern recognition approach was preferred. The extensive work done in this field does specify a scope of improvement in the choice of audio features as well as classification for better accuracy. This is where we intend to contribute so that the mood can be automatically and efficiently be detected for a given audio file.

Most of the experimentation done in the field of music mood categorization has been observed with respect to non-Indian music [8]. Music being subjective to cultural backgrounds, it is but natural that Indian Music might need a different treatment as compared to non-Indian music. Our goal is to develop a music emotion categorization system for Indian popular music by analysing the relation of timbre, spectral and temporal features of audio file with the emotion represented by the audio file. To name a few, these features include pitch, tempo beats and rhythm. Grouping of songs might be done by using clustering algorithms like k-means [3]. Mood Based Music Categorization System has some varied applications such as:

Music Therapy: Music therapy is used to cure a patient using music as a medicine. Music therapist uses music and all of its facets like physical, emotional, mental, social, aesthetic and spiritual to help clients. Multimedia Information Retrieval: It consists of extracting semantic information from multimedia data sources. It is used to understand media content by media-specific feature transformations. Music Industry: In music industry, the mood of the song plays a vital role in its perception. Hence, Profits can be maximized by determining the mood of the top selling song.

The paper is organized as follows: In Section 2 we give a brief description of the important papers that we have studied/utilized as a part of our literature survey. In Section 3, we introduce our proposed system model for mood recognition. In Section 4 we present the result and conclusion.

2. Literature review

Various experts in the fields of psychology and musicology have come with models describing human emotions. One of the most ancient of the experiments done by "Hevner'[15] helped in categorizing various adjectives into 8 different groups each representing a class of mood. The model was more of a categorical model wherein a list of adjectives representing the same emotion was grouped together.

"Russell' [14] later came up with the circumplex model representing human emotions on a circle with each mood category plotted within the circle separated from other categories along the polar co- ordinates. "Thayer' [16] too came up with a dimensional model plotted along two axes (Stress versus energy) with mood represented by a two- dimensional co-ordinate system. Vahida Z Attar and his partner [1] inspired from the well-known dimensional model, proposed by Thayer. They used Low-level audio features which can be further divided into two classes of timbral and temporal features. They experimented with the task of mapping audio features of Indian Popular Music with respective moods with the top accuracy ranging in between 75% and 81% with respect to Fmeasure and 70% to 75% precision measure using RandomForest, RandomTree and simpleCART algorithms implemented with bagging approach. The best accuracy w.r.t. area under ROC was observed in the range 0.91 to 0.94.

Yi Liu and YueGao[2]extracted the Audio Features such as Rhythm, timbre, intensity. They used and studied the Gaussian Mixture Model (GMM) and Support Vector Machine (SVM) as classifier. They also implemented the Feature Selection Algorithm named Relief, SFS, fisher and active. Due to the use of feature selection algorithm the obtained Accuracy was 84%.

Hanaa M. Hussain, Khaled Benkrid, Huseyin Seker and Ahmet T. Erdogan [3] have presented FPGA hardware design of the K-means algorithm for Microarray data mining, which proved to be power efficient and useful for server solution. It was observed that K-means algorithm had potential for speeding up data analysis of Microarray datasets.

Shang Lei [5] implemented Feature Selection Algorithm for text categorization named Information Gain. A new method of the text feature selection based on Information Gain and Genetic Algorithm is. This method chooses the feature based on information gain with the frequency of items. Fewzee, Pouria and KarrayFakhri [6] implemented the Pearson's correlation coefficient Classification algorithm to predict the continuous emotional contents of social signals to compare the prediction quality of the different models used in the system, The correlation coefficient of their prediction with the actual values is prevalently used.

Shin-Cheol Lim, Sei-Jin Jang, Soek-Pil Lee, and Moo Young Kim [7] proposed the use of feature- based modulation spectrum instead of a octave-based modulation spectrum to extract FMSFM and FMSCM. Compared with the timbral features such as MFCC, OSC, and DFB, they produced better accuracy in mood classification because they efficiently utilized the time varying characteristics of music in feature domain. Combining FMSFM and FMSCM with the timbral features they obtained better genre and mood classification accuracy by 2.6% and 6.5% than the timbral features, respectively.

Vallabha Hampiholi[8] has presented a method to detect mood in Indian bollywood music based on Thayer's mood model comprising four mood types, Exuberance, Anxiety, Serene, and Depression. Audio features related to Intensity, Timbre and Rhythm (Tempo) are extracted from the musical data. Thus, he concluded that the success rate of detecting the mood accurately for Indian bollywood music is 60%. Also, he observed that the success rate (40%) falls when detecting mood in western music.

Jung Hyun Kim, SungMin Kim, Won Young Yoo [4] proposed mood classification model for music recommendation system. They analyzed the collected music mood tags and AV values from 20 subjects and classify music mood on AV plane into 8 regions by using K-means clustering algorithm. They showed that some regions can be identified by representative mood tags but some mood tags are overlapped in almost regions.

Zhouyu Fu, Guojun Lu, Kai Ming Ting and DengshengZhang. In [10]have proposed the recent development in music classification and annotation. Their survey has provided an up-to-date discussion of audio features and classification techniques used in the literature. In addition, they also have reviewed individual tasks for music classification and annotation and identified both task-specific issues and general open problems. To identify the limit of a music classification system they worked on comparing the performance of automatic Genre classification with human performance. It was found that humans have a strong ability to identify genre classes and correct decisions can be made within a short time span of 10-100ms.

Yongkai Zhao, Deshun Yang, Xiaoou Chen [11] investigated a multi-modal version of Music Mood Classification. Besides features based on the audio signal and features derived from song lyrics, they introduced a third representation, features extracted from a song's MIDI form. They devised and compared three variants of co-training algorithm to combine these three sets of features. The results show that this method can effectively improve the classification accuracy through the use of the complementary nature of lyrics, audio and MIDI representations.

B. Han and group [12] proposed SMERS - A Music Emotion Recognition using Support Vector Regression. In their proposed paper, automatic emotion recognition of music has been evaluated using various machine learning classification algorithms such as SVM, SVR and GMM with remarkable increase in accuracy using SVR as compared to GMM. For further research, the paper suggests more perceptual features should be considered and other classification algorithms such as fuzzy and kNN (k-Nearest Neighbor)

Chia-Chu Liu and team [13] presented an emotion detection and classification system for pop music. The system extracts feature values from the training music files by PsySound2 and generates a music model from the resulting feature dataset by a classification algorithm. The system is designed using a hierarchical framework followed by an accuracy enhancement mechanism. The results show that the system gives satisfactory performance and can be applied to public music database software to provide emotion-based search. The features that affect the perception of emotion are associated with frequency centroid, spectral dissonance and pure tonalness. The paper extols on the deeper relation between these features and music emotion to have a more accurate music mood classification.

Kadek Cahya Dewi, AgusHarjoko[9] showed that the system for music classification based on mood parameters could be developed based on K-Nearest Neighbor and Self Organizing Map. In this research music is classified based on sound similarity obtained from the rhythm patterns based on mood parameters according to Thayer's theory. Clustering results using Self-Organizing Map method is influenced by the number of map units used. Classification of music based on mood parameters by the method of K- Nearest Neighbor and Self Organizing Map with 30 songs reached an accuracy of 66.67%.

3. Proposed system

A. Basic Concept

The prime focus is to categorize the audio into different moods. Following are the moods that we have identified currently for our work so far. Adjacent to the mood category are the adjectives that the mood collectively represents:-

Songs with similar pattern or their similar audio feature range will be grouped together to yield a particular mood. Hence, a mood based playlist will be provided to the user.

B. Preprocessing

To begin with, we selected 50 songs for each mood based on the survey conducted among different people and based on the results of Vallabha Hampiholi[8].We wanted to assert the mood of a particular song based on the generalized perspective of human mind. The survey was conducted among 100 people and the result was tabularized so as to set the range of threshold for each mood.

C. Feature Extraction

JAudio was used for the feature extraction process.Each clip is divided into 0.5 overlapping 32ms-long frames. The extracted features fall into four categories: timbre, intensity, rhythm. The first three sets can express mood information to some degree and are very important for mood detection.

1) Timbre features

Happy songs usually sound bright and vibrant, while grief ones sound pensive and gloomy. Timbre features can be used to judge whether the emotion is negative or positive. The timbre features we used are listed as follows: Centroid, Rolloff Point, Flux, Zero Crossing, Strongest Frequency Via Zero Crossing, Strongest Frequency Via Spectral Centroid, Strongest Frequency Via FFT Maximum, Compactness, MFCC, LPC, Peak based Spectral Smoothness. We, Calculated the mean and standard deviation over all frames.

2) Intensity features

Intensity features can used to judge whether the emotion is very strong or not. For example, if songs express a positive emotion, then using intensity features we can get whether it is enthusiastic or lively. In this paper, the intensity features are RMS and Fraction of Low Energy Windows. By calculating the mean and standard, we got 4 intensity features.

3) Rhythm feature

Through rhythm features, we also can get some information about whether the music emotion is positive or negative. Fast songs tend to be happier than slow ones. We extracted rhythm features including Beat Sum, Strongest Beat and Strength of Strongest Beat. Also by calculating the mean and standard, led to 6 rhythm features. The features are extracted and consolidated for each music piece in a standard file format so as to make it easy for mining the relations between these features w.r.t. the corresponding mood of the audio files.

D. Feature Selection

There are certain features which give similar values for audio of any mood. Hence such features can hinder the accuracy of the system. After conducting survey and feature extraction process, Information Gain algorithm was used to select the defined features and remove the redundant ones. Information gain helps to determine which attribute in a given set of training feature vectors is most useful for

discriminating between the classes to be learned [17]. When a particular classification model has multiple features, there is higher probability that many (if not most) of the features are low information.

These are the features that are common across all classes and therefore contribute meagre information to the classification process. Individually they are innocuous, but in aggregate, low information features can decrease performance.

Eliminating low information features gives your model clarity by removing noisy data. When the higher information features are used, performance is increased and the size of the model is decreased, which results in less memory usage along with faster training and classification.

The total information gain is defined as:

... (1)

Let,

... (1.1)

... (1.2)

... (1.3)

... (2)

Where, Pi is the ratio of conditional attribute P in dataset. When Sj has | Sj | kinds of attribute values and condition attribute Pi partitions set P using attribute Sj, the value of information E(Sj) is defined

Following is a consolidated list of selected features by Information Gain algorithm:

1. Spectral Centroid OSD*-The spectral centroid is a measure used in digital signal processing to characterize a spectrum. It indicates where the "center of mass" of the spectrum is. Perceptually, it has a robust connection with the impression of "brightness" of a sound. It is calculated as the weighted mean of the frequencies present in the signal, determined using a Fourier transform, with their magnitudes as the weights.

2. Spectral Rolloff OSD-Spectral rolloff is defined as the frequency where85% of the energy in the spectrum is below this point. It is often used as an indicator of the skew of the frequencies present in a window.

3. Spectral Flux OSD-Spectral Flux is defined as the spectral correlation between adjacent windows. It is oftenused as an indication of the degree of change ofthe spectrum between windows.

4. Compactness OSD-Compactness is closely related to Spectral Smoothness. The difference is that instead of summing over partials, compactness sums over frequency bins of an FFT. This provides an indication of the noisiness of the signal.

5. Root Mean Square OSD-RMS is calculated on a per window basis. It is defined by the equation:

...

Where N is the total number of samples provided in the time domain. RMS is used to calculate the amplitude of a window.

6. Fraction of Low Energy Windows OSD- This feature is defined as the fraction of previous windows whose RMS is less than the mean RMS. This gives an indication of the variability of the amplitude of windows.

7. Zero Crossings OSD-Zero Crossing is calculated by counting the number of times that the time domain signal crosses zero within a given window. 'Crossing zero' is defined as (xn-1 < 0 and xn> 0) or (xn-1 > 0 and xn< 0) or (xn-1 6= 0 and xn = 0).

8. Strongest Beat OSD- Gives the strongest beat in a signal.

9. Spectral Centroid OA**-The centre of mass of the power spectrum. This is the overall average over all windows.

10. Spectral Rolloff OA-The fraction of bins in the power spectrum at which 85% of the power is at lower frequencies. This is a measure of the right-skewedness of the power spectrum. This is the overall average over all windows.

11. Spectral Flux OA-A measure of the amount of spectral change in a signal. Found by calculating the change in the magnitude spectrum from frame to frame. This is the overall average over all windows.

12. Root Mean Square OA-A measure of the power of a signal. This is the overall average over all windows.

13. Zero Crossings OA-The number of times the waveform changed sign. An indication of frequency as well as noisiness. This is the overall average over all windows.

14. Beat Sum OA-The sum of all entries in the beat histogram. This is a good measure of the importance of regular beats in a signal. This is the overall average over all windows.

*OSD - Overall Standard Deviation

**OA - Overall Average

E. System Architecture

1. Module 1:

The song is given as input which can be an mp3, wav, wmv, au, sndetc file format. It is going to be processed by the system and given as an input to the feature extractor.

2. Module 2:

Using jAudio feature extraction tool the selective audio features are extracted. A feature vector comprising all of these features is extracted for each of the music clip.The feature vectors thus extracted from the attributes of the each music clip - which can be called as a data instance. These feature vectors computed in the memory are stored in a flat file following the standard ARFF file format understood by most of the data mining tools.

3. Module 3:

Then the features are given as input to the Random Forest classification algorithm to carry out the comparison of audio features with the threshold values. The Random Forest Algorithm is explained below:

3.1 Random Forest Algorithm:

In random forest, each node is split using the best among a sub-set of predictors randomly chosen at that node. This somewhat counter intuitive strategy turns out to perform very well compared to many other classifiers, including discriminant analysis, support vector machines and neural networks, and is robust against over fitting.

The Algorithm is implemented as follows:

1. Draw n-tree bootstrap samples from the original data.

2. For each of the bootstrap samples, grow a un- pruned classification or regression tree, with the following modification at each node, rather than choosing the best split among all predictors, randomly sample mtryof the predictors and choose the best split from among those variables. (Bagging can be thought of as the special case of random forests obtained when mtry= p, the number of predictors [18]).

3. Predict new data by aggregating the predictions of the ntreetrees (i.e., majority votes for classification, average for regression).

4. Module 4:

As a result of the output of the fuzzy logic applied to feature values, K-means clustering algorithm is used for clustering songs into four clusters-happy, sad and exciting, silent. The K-means algorithm is elaborated below:

4.1 K-means Algorithm:

The aim of the K-means algorithm is to divide M points in N dimensions into K clusters so that the within-cluster sum of squares is minimized. The algorithm requires as input a matrix of M points in N dimensions and a matrix of K initial cluster centres in N dimensions.

The number of points in cluster L is denoted by NC(L). D(I, L) is the Euclidean distance between point I and cluster L. The general procedure is to search for a K-partition with locally optimal within- cluster sum of squares by moving points from one cluster to another.

The Algorithm is implemented as follows:

1. First, decide the number of clusters k. Here we have k=4 i.e. 4 clusters of 4 different moods. Then: 1. Initialize the center of the clusters µi= some value, i=1...k

2. Attribute the closest cluster to each data point. ...

3. Set the position of each cluster to the mean of all data points belonging to that cluster. ...

4. Repeat steps 2-3 until convergence.

Notation: |c|= number of elements in c

Module 5:

The title of the song and its mood is stored in the database. This database is further utilized in generating a mood aware music playlist.

4. Results and Discussion

Based on the systems that were previously developed, our system is a combination of all of them. Here we have not just used the classifier but also the clustering algorithm. The feature selection algorithm has also given us an upper hand to increase the accuracy and reduce the computation time. Based on the system computed in the above section, we performed a few experiments on the untrained data.

The result summary is presented in Table 2. The untrained musical samples are first classified with listening tests, the results of which are presented in column 4 of Table 2.All the audience members were from Indian cultural background. Then, each clip was classified using the classification system derived in the above section. The result of the classification system is presented in column 3 and the result of the system based classification vs. listening test is presented in column 5 of Table 2.

5. Conclusion and Future Work

From the above discussion on mood based music categorization we conclude that audio features of Bollywood Music were successfully mapped with their respective moods. We observed that the success rate of detecting the mood accurately for Indian Bollywood music is 70%. We also observe that the success rate falls (30%) when detecting mood in western music. The success rate for the western music is expected to be lower as our music classification framework was entirely based on the Indian cultural context and the audiences classifying the western music through listening tests were all from Indian cultural background.

This work will help further analysis on Indian music categorization. In future, this system can be extended to other genres of Indian songs like Hindustani and Carnatic with changes involving audio features and classification techniques. Customization of this system to non-Indian songs cannot be ruled out as well after a thorough experimentation In Bollywood music most of the moods are administered by expressions. Lyrics efficiently liaise along with music to determine these expressions. The amalgamation of lyrics analysis with audio features and facial expression can make the system much stronger with a better accuracy. As part of our future work we will also make use of this system to design a system which can predict the mood of the person by capturing his facial expressions and generate playlist according to the mood.

Acknowledgement

We take this opportunity to thank our project guide and Prof. VaishaliS. Nandedkar, Head, Department of Information Technology for her valuable guidance in the completion of this project. The authors are also thankful to all the staff members of the Department of Padmabhooshan Vasantdada Patil Institute of Technology, Pune for their valuable time, support, comments, suggestions and persuasion. We would also like to thank the institute for providing the required facilities and important books. Last but not the least; we would like to thank our parents for their concrete support and ceaseless faith in us.

References

References

[1] Vahida Z Attar, Aniruddha Ujlambkar "Mood based classification of Indian Popular Music" Modelling Symposium (AMS), 2012 Sixth Asia, ISBN: 978-0-7695-4730-5/12, 29-31 May 2012.

[2] Yi Liu , Yue Gao -"Acquiring mood Information from songs in large music database", Fifth International Joint Conference on INC, IMS, and IDC, ISBM: 978-0-7695-3769-6/09 , 25-27 Aug. 2009.

[3] Hanaa M. Hussain1, Khaled Benkrid1, Huseyin Seker2, Ahmet T. Erdogan-" FPGA Implementation of K-means Algorithm for Bioinformatics Application:An Accelerated Approach to Clustering Microarray Data" NASA/ESA Conference on Adaptive Hardware and Systems., ISBN: 978-1-4577-0599-1/11 , 6-9 June 2011.

[4] JungHyun Kim, SungMin Kim, Won Young Yoo-"Music mood classification model based on arousal-valence values." ICACT, ISBN 978-89- 5519-155-4, 2011.

[5] Shang Lei, "A Feature Selection Method Based on InformationGain and Genetic Algorithm" Computer Science and Electronics Engineering (ICCSEE), ISBN: 978-1-4673-0689-8 , 23-25 March 2012.

[6] Fewzee, Pouria ; Karray, Fakhri, "Continuous Emotion Recognition: Another Look at the Regression Problem", Affective Computing and Intelligent Interaction (ACII), . ISSN : 2156- 8103, 2-5 Sept. 2013.

[7] Shin-Cheol Lim, Sei-Jin Jang, Soek-Pil Lee and Moo Young Kim-"Music Genre/Mood Classification using a Feature-Based Modulation Spectrum", ICMIC , Print ISBN: 978-1-4577- 1128-2, 26-28 Sept. 2011.

[8] Vallabha Hampiholi-"A method for Music Classification based on Perceived Mood Detection for Indian Bollywood Music"- World Academy of Science, Engineering and Technology, vol:72, 2012.

[9] Kadek Cahya Dewi , Agus Harjoko -" Kid's Song Classification Based on Mood Parameters Using K-Nearest Neighbor Classification Method and Self Organizing Map", International Conference on Distributed Frameworks for Multimedia Applications (DFmA), Print ISBN: 978-1-4244-9335-7, 2-3 Aug. 2010.

[10] Zhouyu Fu, Guojun Lu, Kai Ming Ting, and Dengsheng ZhangIn" A Survey of Audio-Based Music Classification and Annotation"- IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 13, NO. 2, APRIL 2011.

[11] Yongkai Zhao , Deshun Yang , Xiaoou Chen- "Multi-Modal Music Mood Classification Using Co-Training"- IEEE, vol: 978-1-4244-5392- 4/10, 2010.

[12] Han, B., Rho, S., Dannenberg, R. B., Hwang, E., "Smers: Music emotion recognition using support vector regression", Proceedings of the 10th Intl. Society for Music Information Conf., Kobe, Japan, pp. 651-656, October 2009.

[13] Chia-Chu Liu, Yi-Hsuan Yang, Ping-Hao Wu, Homer H. Chen, (2006) "Detecting and classifying emotions in popular music", JCIS Proceedings.

[14] Russell J. A., "A circumplex model of aect", Journal of Personality and Social Psychology, vol: 39: 1161-1178, 1980.

[15] Kate Hevner,"Experimental studies of the elements of expression in music", American Journal of Psychology, vol: 48:246268, 1936.

[16] Thayer, R. E., The Bio-psychology of Mood and Arousal", New York: Oxford University Press, 1989.

[17] Liu H., Motoda H. (eds.) "Computational methods of feature selection" , Chapman & Hall/CRC Taylor & Francis Group (ISBN 9781584888789)(411s), e-book, Year: 2008.

[18] Jiawei Han, Micheline Kamber , Jian Pei-"Data Mining Concepts And Techniques 3rd Edition",Morgan Kauffmann Publishers, e-book, Year: June 2011.