Soundwriter
Soundwriter is a software system that allows written/spoken stories to generate accompanying sounds via machine learning and emotion mapping.
Research Paper - Soundwriter: Real-Time Music Generation for Oral Storytelling through Emotion Mapping || Published via International Computer Music Conference 2019
Soundwriter is developed with the goal of enriching the oral storytelling experience. While there have been a few attempts to create an off-line automated scoring system for audio stories, nearly none have investigated a real-time system that is able to classify emotions on both semantic and prosodic features of a narrated story, let alone dynamically generating original music that is catered to the emotions of a given story.
Soundwriter implements a real-time hybrid system where the emotional state of a given story is analyzed by its lexical features and auditory features. For example, emotionally salient words are given ratings based on arousal, valence, and dominance, while emotionally charged prosodic features of a speaker’s voice (pitch contour, speech rate, and intensity) influence the final classification of the story’s emotional state.
For an automated operation, Soundwriter generates sounds via a simple FM synthesis engine designed in Max/MSP. During an oral story reading, the system generates real-time graphic notation for musicians. Various musical notations flow through the screen according to the detected emotional state, allowing musicians to react in real-time and create the music that accompanies the story. The musical figures are pre-defined/ composed by a composer for a given story.
Soundwriter utilizes a Convolutional Neural Network based machine learning algorithm for its speech recognition. This enables the detection of key words as well as informs the system of the reader's pace.
