This book addresses the subject of emotional speech, especially its encoding and decoding process during interactive communication, based on an improved version of Brunswik’s Lens Model. The process is shown to be influenced by the speaker’s and the listener’s linguistic and cultural backgrounds, as well as by the transmission channels used. Through both psycholinguistic and phonetic analysis of emotional multimodality data for two typologically different languages, i.e., Chinese and Japanese, the book demonstrates and elucidates the mutual and differing decoding and encoding schemes of emotional speech in Chinese and Japanese.