Skip to content

README.md

Latest commit

 

History

History
104 lines (84 loc) · 5.15 KB

README.md

File metadata and controls

104 lines (84 loc) · 5.15 KB
Raw

RVST598_Speech-Emotion-Recognition

This is my summer (May - Aug) 2019 research project on using machine learning to detect emotions in speech.

Plan

My goal is to detect which emotions are present in a speech sample. Specifically, this problem is called multi-class, multi-label speech emotion recognition. I consider these seven emotions:

  • Neutral
  • Anger
  • Disgust
  • Fear
  • Happy
  • Sad
  • Surprise

A speech sample is a few seconds long and flows through the following data processing pipeline:

  1. Raw waveform (time + amplitude)
  2. Spectrogram (time + frequency + ampltitude)
  3. Log-Mel Spectrogram (time + log-mel + ampltitude)
  4. Convolutional Neural Network (CNN)
  5. Output (one or more of the seven emotions)

Timeline

  • May: Read the Deep Learning with Python textbook by Francois Chollet to pickup deep learning.
  • June: Decide on this project and start collecting and processing the databases.
  • July: Process the databases and start creating the neural network model.
  • August: Complete the neural network training, write the paper, and create the presentation.

Databases Used

  • IEMOCAP: Interactive Emotional Dyadic Motion Capture
  • CREMA-D: Crowd-sourced Emotional Mutimodal Actors Dataset
  • TESS: Toronto Emotional Speech Set
  • RAVDESS: Ryerson Audio-Visual Database of Emotional Speech and Song
  • EmoV-DB: Emotional Voices Database

The databases were not uploaded to Github due to copyright restrictions and size.

Repo Structure and Description

└───repo  # Holds this repository
    ├───data  # Holds the raw and processed databases (not uploaded to Github)
    │   ├───model  # Holds the neural network saved weights and training history
    │   ├───processed  # Holds the log-mel spectrogram audio samples in the form of numpy arrays
    │   └───raw  # Holds the raw audio samples and labels from each database
    │       ├───cremad
    │       ├───iemocap
    │       ├───ravdess
    │       └───tess
    ├───research  # Holds the files resulting from this research
    │   ├───charts  # Holds pie charts created using Chart.js for the paper and presentation
    │   ├───images  # Holds images from training, spectrograms, and testing.
    │   ├───paper  # Holds the paper resulting from this research
    │   └───presentation  # Holds the presentation resulting from this research
    └───src  # Holds the code for preprocessing the databases and for creating/training the neural network model
        ├───audio_processor  # Holds the wav and spectrogram functions
        ├───database_processor  # Holds the database preprocessing functions
        ├───neural_network  # Holds the neural network model
        ├───em_constants.py  # Holds the common emotions used throughout the program
        └───main.py  # Holds the main function to run to train and test the neural network model

How to Run

I developed this program using PyCharm so that's the easiest way to run the code but its possible to run the code using a Terminal or using Anaconda.

Preprocessing the Databases

  1. Copy the raw database samples and labels into their respective folder in /repo/data/raw/[db]. The proper structure is shown below.
  2. Run the corresponding database script in /repo/src/database_processor/[db].py. This processes the raw audio samples into log-mel spectrograms. Note: this takes some time and lots of memory (16GB+).
  3. The processed log-mel spectrograms should appear in the /repo/data/processed folder with filenames like CRE_0-0_1_0_0_0_0_0.npy. The filename format is [DB]_[SAMPLE ID]-[ONE HOT ENCODED EMOTION].npy. The mapping for the one-hot encoding to emotion can be found in the code.
└───data
    └───raw
        ├───cremad
        │   ├───tabulatedVotes.csv
        │   └───AudioWAV
        │       └───1001_DFA_ANG_XX.mp3
        ├───iemocap
        │   ├───data
        │   │   └───S1
        │   └───labels
        │       └───S1
        ├───ravdess
        │   └───Actor_01
        │       └───03-01-01-01-01-01-01.mp3
        └───tess
            └───data
                └───OAF_back_angry.mp3

Training the Neural Network

  1. Check the /repo/data/processed folder for files. If you're running the training on all four database, there should be 21,721 files/samples labeled like CRE_0-0_1_0_0_0_0_0.npy.
  2. Run the script /repo/src/database_processor/processed_db_stats.py to remove the samples with 4 and 5 labels. There should now be 21,675 samples.
  3. Configure the model parameters in /repo/src/neural_network/nn_model.py under the build_model() function.
  4. Configure the training parameters under /repo/src/neural_network/nn_constants.py. This file lets you control the number of epochs, the batch size, the optimizer and loss, and more.
  5. Run the script /repo/src/main.py to train the neural network. This script will automagically create the train, validation, and test sets and display the training history.