Sunday, July 19, 2015

Emotion Recognition and Tone Characterization by DNN + ELM

Let's face it, emotions are tricky!

Method

After struggling with emotion recognition for long time, I decided to implement the method proposed by Microsoft Research last year in the Interspeech conference, an approach using a deep neural network(DNN) and extreme learning machine(ELM). In particular, a DNN was trained to extract segment-level (256 ms) features and ELM was trained to make decisions based on the statistics of these features on a utterance level.

The training was done using the Interactive Emotional Dyadic Motion Capture (IEMOCAP) database from here.

Experimental results in the paper demonstrated that this approach outperforms the state-of-the-art, including OpenEAR.

Here you can also find a video presentation by the author Kun Han himself.

My implementation

all code can be found here

Dependency:

python speech features library: used to extract MFCC features from audio files, which can be replaced by the common feature extraction component once the Red Hen Lab Audio Analysis pipeline is established. But to run the current code, this library must be included in the working folder.

PDNN: A Python Toolkit for Deep Learning, which is used to extract segment-level features by deep neural network. 

Python files:

energy.py
Takes a speech signal and returns the indices of frames with top 10% energy.

Given two audio folders (training and validation, see "folder structure" for the structures of these folders), extracts the segment-level features from audio files in these folders for DNN training.

Given one (testing) audio folder, extracts the segment-level features from audio files in the folder for DNN feature extraction.

Train ELM with the probability features extracted by DNN.

Annotate the recognition results of the test files into Results.txt

Folder Structure:

wav_train (training folder):
has as many subfolders as there are emotions(or tones) to be recognized, each subfolder corresponds to one emotion (or tone) and has one utterance as one separate .wav audio file inside it.

wav_valid (validation folder):
has as many subfolders as there are emotions(or tones) to be recognized, each subfolder corresponds to one emotion (or tone) and has one utterance as one separate .wav audio file inside.

wav_test (testing folder):
has one utterance as one separate .wav audio inside. These files will be annotated and the results are written in Results.txt

Shell Scripts:

Extract features and train DNN and ELM. You can specify the pointer to PDNN, the training folder, and validation folder in this file.

Extract features of test files and recognize the emotions of utterance files in test folder. You can specify the pointer to PDNN and the testing folder here.

Intermediate Files:

dnn.param
contains the parameters of the trained deep neural network

dnn.cfg
contains the configurations of the trained deep neural network

ELMWeights.pickle.gz
contains weight parameters of the extreme learning machine

LabelNumMap.pickle.gz
contains the mapping between the emotion string labels in training/validation folders and integer labels during DNN training.

Output File:

Results.txt
the recognition results

Usage Example:

To train a new emotion recognizer (after specifying the training and validation data):
./train.sh

To predict emotions on new audio files with trained model: 
./test.sh

18 comments:

  1. That was a great message in my carrier, and It's wonderful commands like mind relaxes with  understand words of knowledge by information's.
    Python Training in Chennai

    ReplyDelete
  2. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  3. it is really amazing...thanks for sharing....provide more useful information...
    Mobile app development company

    ReplyDelete
  4. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post I would like to read this
    Click here:
    python training in marathahalli
    Click here:
    python training in btm

    ReplyDelete
  5. I was looking for this certain information for a long time. Thank you and good luck.
    Click here:
    python online training
    Click here:
    python training in usa

    ReplyDelete
  6. Well done! Pleasant post! This truly helps me to discover the solutions for my inquiry. Trusting, that you will keep posting articles having heaps of valuable data. You're the best! 
    Click here:
    python online training
    Click here:
    python training in usa

    ReplyDelete
  7. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.

    DevOps online Training|DevOps Training in USA
    Devops Training in Chennai

    Devops Training in Bangalore

    ReplyDelete
  8. Awesome article. It is so detailed and well formatted that i enjoyed reading it as well as get some new information too.
    java training in chennai | java training in USA

    ReplyDelete
  9. Hmm, it seems like your site ate my first comment (it was extremely long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog. I as well as an aspiring blog writer, but I’m still new to the whole thing. Do you have any recommendations for newbie blog writers? I’d appreciate it.

    AWS Interview Questions And Answers


    AWS Training in Bangalore | Amazon Web Services Training in Bangalore

    AWS Training in Pune | Best Amazon Web Services Training in Pune

    Amazon Web Services Training in Pune | Best AWS Training in Pune

    AWS Online Training | Online AWS Certification Course

    ReplyDelete
  10. Whoa! I’m enjoying the template/theme of this website. It’s simple, yet effective. A lot of times it’s very hard to get that “perfect balance” between superb usability and visual appeal. I must say you’ve done a very good job with this.

    AWS Training in Toronto | Amazon Web Services Training in Toronto , Canada

    AWS Training in New york city | Amazon Web Services in New York City


    AWS Training in London | Amazon Web Services Training in London , UK

    ReplyDelete
  11. Whoa! I’m enjoying the template/theme of this website. It’s simple, yet effective. A lot of times it’s very hard to get that “perfect balance” between superb usability and visual appeal. I must say you’ve done a very good job with this.

    AWS Training in Toronto | Amazon Web Services Training in Toronto , Canada

    AWS Training in New york city | Amazon Web Services in New York City


    AWS Training in London | Amazon Web Services Training in London , UK

    ReplyDelete
  12. Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can't wait to read lots of your posts

    angularjs online Training

    angularjs Training in marathahalli

    angularjs interview questions and answers

    angularjs Training in bangalore

    angularjs Training in bangalore

    ReplyDelete