Sunday, July 19, 2015

Emotion Recognition and Tone Characterization by DNN + ELM

Let's face it, emotions are tricky!

Method

After struggling with emotion recognition for long time, I decided to implement the method proposed by Microsoft Research last year in the Interspeech conference, an approach using a deep neural network(DNN) and extreme learning machine(ELM). In particular, a DNN was trained to extract segment-level (256 ms) features and ELM was trained to make decisions based on the statistics of these features on a utterance level.

The training was done using the Interactive Emotional Dyadic Motion Capture (IEMOCAP) database from here.

Experimental results in the paper demonstrated that this approach outperforms the state-of-the-art, including OpenEAR.

Here you can also find a video presentation by the author Kun Han himself.

My implementation

all code can be found here

Dependency:

python speech features library: used to extract MFCC features from audio files, which can be replaced by the common feature extraction component once the Red Hen Lab Audio Analysis pipeline is established. But to run the current code, this library must be included in the working folder.

PDNN: A Python Toolkit for Deep Learning, which is used to extract segment-level features by deep neural network. 

Python files:

energy.py
Takes a speech signal and returns the indices of frames with top 10% energy.

Given two audio folders (training and validation, see "folder structure" for the structures of these folders), extracts the segment-level features from audio files in these folders for DNN training.

Given one (testing) audio folder, extracts the segment-level features from audio files in the folder for DNN feature extraction.

Train ELM with the probability features extracted by DNN.

Annotate the recognition results of the test files into Results.txt

Folder Structure:

wav_train (training folder):
has as many subfolders as there are emotions(or tones) to be recognized, each subfolder corresponds to one emotion (or tone) and has one utterance as one separate .wav audio file inside it.

wav_valid (validation folder):
has as many subfolders as there are emotions(or tones) to be recognized, each subfolder corresponds to one emotion (or tone) and has one utterance as one separate .wav audio file inside.

wav_test (testing folder):
has one utterance as one separate .wav audio inside. These files will be annotated and the results are written in Results.txt

Shell Scripts:

Extract features and train DNN and ELM. You can specify the pointer to PDNN, the training folder, and validation folder in this file.

Extract features of test files and recognize the emotions of utterance files in test folder. You can specify the pointer to PDNN and the testing folder here.

Intermediate Files:

dnn.param
contains the parameters of the trained deep neural network

dnn.cfg
contains the configurations of the trained deep neural network

ELMWeights.pickle.gz
contains weight parameters of the extreme learning machine

LabelNumMap.pickle.gz
contains the mapping between the emotion string labels in training/validation folders and integer labels during DNN training.

Output File:

Results.txt
the recognition results

Usage Example:

To train a new emotion recognizer (after specifying the training and validation data):
./train.sh

To predict emotions on new audio files with trained model: 
./test.sh

37 comments:

  1. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post I would like to read this
    Click here:
    python training in marathahalli
    Click here:
    python training in btm

    ReplyDelete
  2. I was looking for this certain information for a long time. Thank you and good luck.
    Click here:
    python online training
    Click here:
    python training in usa

    ReplyDelete
  3. Well done! Pleasant post! This truly helps me to discover the solutions for my inquiry. Trusting, that you will keep posting articles having heaps of valuable data. You're the best! 
    Click here:
    python online training
    Click here:
    python training in usa

    ReplyDelete
  4. Awesome article. It is so detailed and well formatted that i enjoyed reading it as well as get some new information too.
    java training in chennai | java training in USA

    ReplyDelete
  5. Hmm, it seems like your site ate my first comment (it was extremely long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog. I as well as an aspiring blog writer, but I’m still new to the whole thing. Do you have any recommendations for newbie blog writers? I’d appreciate it.

    AWS Interview Questions And Answers


    AWS Training in Bangalore | Amazon Web Services Training in Bangalore

    AWS Training in Pune | Best Amazon Web Services Training in Pune

    Amazon Web Services Training in Pune | Best AWS Training in Pune

    AWS Online Training | Online AWS Certification Course

    ReplyDelete
  6. Whoa! I’m enjoying the template/theme of this website. It’s simple, yet effective. A lot of times it’s very hard to get that “perfect balance” between superb usability and visual appeal. I must say you’ve done a very good job with this.

    AWS Training in Toronto | Amazon Web Services Training in Toronto , Canada

    AWS Training in New york city | Amazon Web Services in New York City


    AWS Training in London | Amazon Web Services Training in London , UK

    ReplyDelete
  7. Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can't wait to read lots of your posts

    angularjs online Training

    angularjs Training in marathahalli

    angularjs interview questions and answers

    angularjs Training in bangalore

    angularjs Training in bangalore

    ReplyDelete
  8. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.
    advanced excel training in bangalore

    ReplyDelete
  9. I am a regular reader of your blog and being students it is great to read that your responsibilities have not prevented you from continuing your study and other activities. Love
    python training in chennai | python training in chennai | python training in bangalore

    ReplyDelete
  10. I was recommended this web site by means of my cousin. I am now not certain whether this post is written through him as nobody else recognise such precise about my difficulty. You're amazing! Thank you!
    angularjs online Training

    angularjs Training in marathahalli

    angularjs interview questions and answers

    angularjs Training in bangalore

    angularjs Training in bangalore

    ReplyDelete
  11. I read this post two times, I like it so much, please try to keep posting & Let me introduce other material that may be good for our community.
    angularjs online training

    apache spark online training

    informatica mdm online training

    devops online training

    aws online training

    ReplyDelete
  12. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.
    Microsoft Azure online training
    Selenium online training
    Java online training
    Python online training
    uipath online training

    ReplyDelete
  13. This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.
    apple service center chennai
    apple service center in chennai
    apple mobile service centre in chennai
    apple service center near me

    ReplyDelete
  14. Really very happy to say, your post is very interesting to read. I never stop myself to say something about it. You’re doing a great job. Keep it up…

    Looking for Best Training Institute in Bangalore , India. Softgen Infotech is the best one to offers 85+ computer training courses including IT Software Course in Bangalore , India. Also it provides placement assistance service in Bangalore for IT.

    ReplyDelete
  15. Whatever we gathered information from the blogs, we should implement that in practically then only we can understand that exact thing clearly, but it’s no need to do it, microsoft azure training because you have explained the concepts very well. It was crystal clear, keep sharing..

    ReplyDelete
  16. Excellent post, it will be definitely helpful for many people. Keep posting more like this.
    DevOps Training in Chennai

    DevOps Course in Chennai

    ReplyDelete
  17. You need to participate in a challenge for probably the best webpage on the web. I will enthusiastically suggest this site!
    best interiors

    ReplyDelete
  18. Happy to visit your blog, I am by all accounts forward to more solid articles and I figure we as a whole wish to thank such countless great articles, blog to impart to us…

    Data Science Training in Hyderabad

    ReplyDelete
  19. Movavi Video Editor 22.3.1 Crack 2022 download activation key offers new and attractive effects, transitions, and other video editing tools. Movavi Video Editor Crack Only

    ReplyDelete
  20. 30 Romantic Wedding Anniversary Wishes for Wife · To the woman I married with love in my heart. To my wife of many years, it never gets old to say I love . 10th Wedding Anniversary Wishes To Wife

    ReplyDelete