Two channels have been distinguished in human interaction: one transmits explicit messages, which may be about anything or nothing; the other transmits implicit messages about the speakers themselves. Both linguistics and technology have invested enormous efforts in understanding the first, explicit channel, but the second is not as well understood. Understanding the other party's emotions is one of the key tasks associated with the second, implicit channel. To tackle that task, signal processing and analysis techniques have to be developed, while, at the same time, consolidating psychological and linguistic analyses of emotion. This article examines basic issues in those areas. It is motivated by the PKYSTA project, in which we aim to develop a hybrid system capable of using information from faces and voices to recognize people's emotions.

Emotion recognition in human-computer interaction

Professional actors' portrayals of 14 emotions varying in intensity and valence were presented to judges. The results on decoding replicate earlier findings on the ability of judges to infer vocally expressed emotions with much-better-than-chance accuracy, including consistently found differences in the recognizability of different emotions. A total of 224 portrayals were subjected to digital acoustic analysis to obtain profiles of vocal parameters for different emotions. The data suggest that vocal parameters not only index the degree of intensity typical for different emotions but also differentiate valence or quality aspects. The data are also used to test theoretical predictions on vocal patterning based on the component process model of emotion (K.R. Scherer, 1986). Although most hypotheses are supported, some need to be revised on the basis of the empirical evidence. Discriminant analysis and jackknifing show remarkably high hit rates and patterns of confusion that closely mirror those found for listener-judges.

/pdf/acoustic-profiles-in-vocal-emotion-expression-4whpq8okig.pdf

Acoustic profiles in vocal emotion expression.

http://www.cs.columbia.edu/~julia/papers/Scherer03.pdf

Vocal communication of emotion: a review of research paradigms

The H&H theory is developed from evidence showing that speaking and listening are shaped by biologically general processes. Speech production is adaptive. Speakers can, and typically do, tune their performance according to communicative and situational demands, controlling the interplay between production-oriented factors on the one hand, and output-oriented constraints on the other. For the ideal speaker, H&H claims that such adaptations reflect his tacit awareness of the listener’s access to sources of information independent of the signal and his judgement of the short-term demands for explicit signal information. Hence speakers are expected to vary their output along a continuum of hyper- and hypospeech. The theory suggests that the lack of invariance that speech signals commonly exhibit (Perkell and Klatt 1986) is a direct consequence of this adaptive organization (cf MacNeilage 1970). Accordingly, in the H&H program the quest for phonetic invariance is replaced by another research task: Explicating the notion of sufficient discriminability and defining the class of speech signals that meet that criterion.

Explaining Phonetic Variation: A Sketch of the H&H Theory

Many authors have speculated about a close relationship between vocal expression of emotions and musical expression of emotions. but evidence bearing on this relationship has unfortunately been lacking. This review of 104 studies of vocal expression and 41 studies of music performance reveals similarities between the 2 channels concerning (a) the accuracy with which discrete emotions were communicated to listeners and (b) the emotion-specific patterns of acoustic cues used to communicate each emotion. The patterns are generally consistent with K. R. Scherer's (1986) theoretical predictions. The results can explain why music is perceived as expressive of emotion, and they are consistent with an evolutionary perspective on vocal expression of emotions. Discussion focuses on theoretical accounts and directions for future research.

/pdf/communication-of-emotions-in-vocal-expression-and-music-2to7fzkqnc.pdf

Communication of emotions in vocal expression and music performance: different channels, same code?

Work on voice sciences over recent decades has led to a proliferation of acoustic parameters that are used quite selectively and are not always extracted in a similar fashion. With many independent teams working in different research areas, shared standards become an essential safeguard to ensure compliance with state-of-the-art methods allowing appropriate comparison of results across studies and potential integration and combination of extraction and recognition systems. In this paper we propose a basic standard acoustic parameter set for various areas of automatic voice analysis, such as paralinguistic or clinical speech analysis. In contrast to a large brute-force parameter set, we present a minimalistic set of voice parameters here. These were selected based on a) their potential to index affective physiological changes in voice production, b) their proven value in former studies as well as their automatic extractability, and c) their theoretical significance. The set is intended to provide a common baseline for evaluation of future research and eliminate differences caused by varying parameter sets or even different implementations of the same parameters. Our implementation is publicly available with the openSMILE toolkit. Comparative evaluations of the proposed feature set and large baseline feature sets of INTERSPEECH challenges show a high performance of the proposed set in relation to its size.

/pdf/the-geneva-minimalistic-acoustic-parameter-set-gemaps-for-36spb4mksx.pdf

The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing

Althought there are numerous books dealing with the science and acoustics of speech, there are relatively few that deal with the singing voice as distinct from the speaking voice. Now, Johan Sundberg's "The Science of the Singing Voice" illustrated with over a hundred instructive and significant diagrams and drawings thoroughly describes the structure and functions of the vocal organs in singing, from the aerodynamics of respiration through the dynamics of articulation."

The Science of the Singing Voice

An articulatory model is presented. It defines a procedure for deriving a set of formant frequencies from information on the state of the lip muscles, the position of the jaw, the shape and position of the tongue body, and larynx height. The acoustic and auditory consequences of varying these parameters individually are reported. The introduction of the jaw as a separate parameter—a feature not used in previous articulatory models—makes it possible to explain why “openness” occurs as a universal phonetic feature of vowel production. According to the explanation proposed, the degree of opening of a vowel corresponds to a position of the jaw that is optimized in the sense that it cooperates with the tongue in producing the desired area function. Such cooperation prevents excessive tongue shape deformation. Our results suggest that, in order to reflect this principle of articulatory synergism, “tongue height,” although primary with respect to its acoustic consequences, should be represented as a derived feature characteristic of the final vocal‐tract configuration.

Acoustical consequences of lip, tongue, jaw, and larynx movement.

The acoustics of the singing voice

The “singing formant” is a high spectrum envelope peak near 2.8 kHz characteristic of vowel sounds produced in male Western opera and concert singing. An acoustical model of the vocal tract is capable of generating such a peak provided that three conditions are met: (1) The cross‐sectional area in the pharynx must be at least six times wider than that of the larynx tube opening. If so, the larynx tube is acoustically mismatched with the rest of the vocal tract, and an extra formant is added to the vocal tract transfer function. (2) The sinus Morgagni must be wide in relation to the rest of the larynx tube. This may tune the frequency of the extra formant to a value between the frequencies of the third and fourth formants in normal speech. (3) The sinus piriformes must be wide. This reduces the frequency of the fifth formant to about 3 kHz. X‐ray studies of a raised and lowered larynx showed that these three conditions may be fulfilled when the larynx is lowered. Thus, the larynx lowering, typical of male ...

https://www.csc.kth.se/utbildning/kth/kurser/DT2212/Singing%20Sundberg.pdf

Johan Sundberg

Papers

The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing

The Science of the Singing Voice

Acoustical consequences of lip, tongue, jaw, and larynx movement.

The acoustics of the singing voice

Articulatory interpretation of the "singing formant".