Harold Hill

Putting the face to the voice: Matching identity across modality

Miyuki Kamachi et al. — Sun, 22 Jan 2012 15:39:18 PST

Speech perception provides compelling examples of a strong link between auditory and visual modalities [1, 2]. This link originates in the mechanics of speech production, which, in shaping the vocal tract, determine the movement of the face as well as the sound of the voice [3, 4]. In this paper, we present evidence that equivalent information about identity is available cross-modally from both the face and voice. Using a delayed matching to sample task, XAB, we show that people can match the video of an unfamiliar face, X, to an unfamiliar voice, A or B, and vice versa, but only when stimuli are moving and are played forward. The critical role of time-varying information is underlined by the ability to match faces to voices containing only the coarse spatial and temporal information provided by sine wave speech [5]. The effect of varying sentence content across modalities was small, showing that identity-specific information is not closely tied to particular utterances. We conclude that the physical constraints linking faces to voices result in bimodally available dynamic information, not only about what is being said, but also about who is saying it.

Range- and domain-specific exaggeration of facial speech

Harold C. Hill et al. — Sun, 22 Jan 2012 15:39:17 PST

Is it possible to exaggerate the different ways in which people talk, just as we can caricature their faces? In this paper, we exaggerate animated facial movement to investigate how the emotional manner of speech is conveyed. Range-specific exaggerations selectively emphasized emotional manner whereas domain-specific exaggerations of differences in duration did not. Range-specific exaggeration relative to a time-locked average was more effective than absolute exaggeration of differences from the static, neutral face, despite smaller absolute differences in movement. Thus, exaggeration is most effective when the average used captures shared properties, allowing task-relevant differences to be selectively amplified. Playing the stimuli backwards showed that the effects of exaggeration were temporally reversible, although emotion-consistent ratings for stimuli played forwards were higher overall. Comparison with silent video showed that these stimuli also conveyed the intended emotional manner, that the relative rating of animations depends on the emotion, and that exaggerated animations were always rated at least as highly as video. Explanations in terms of key frame encoding and muscle-based models of facial movement are considered, as are possible methods for capturing timing-based cues.

Infants' discrimination of faces by using biological motion cues

Janine Spencer et al. — Sun, 22 Jan 2012 15:39:16 PST

We report two experiments in which we used animated averaged faces to examine infants' ability to perceive and discriminate facial motion. The faces were generated by using the motion recorded from the faces of volunteers while they spoke. We tested infants aged 4 ^ 8 months to assess their ability to discriminate facial motion sequences (condition 1) and discrim- inate the faces of individuals (condition 2). Infants were habituated to one sequence with the motion of one actor speaking one phrase. Following habituation, infants were presented with the same sequence together with motion from a different actor (condition 1), or a new sequence from the same actor coupled with a new sequence from a new actor (condition 2). Infants demon- strated a significant preference for the novel actor in both experiments. These findings suggest that infants can not only discriminate complex and subtle biological motion cues but also detect invariants in such displays.

View invariance in facial motion

Tamara Watson et al. — Sun, 22 Jan 2012 15:39:15 PST

Motion may play a large role in the generation of face representations. Facial movement has been shown to facilitate recognition, categorisation of identity, and gender judgment. Dynamic information can be isolated from spatial information by driving a 3-D computer-rendered facial model with the movements of an actor. It has been shown that the perception of static faces is viewpoint-dependent (Hill et al, 1997 Cognition 62 201 - 222). To investigate viewpoint-dependence in dynamic faces an avatar was animated by using actor's movements. Subjects were shown a full-face facial movement. They were then asked to judge which of two rotated moving avatars matched the first face. Test view, orientation, and the type of movement (rigid + nonrigid versus nonrigid) were manipulated. Nonrigid movement alone produced an advantage for upright faces and no effect of view. Rigid and nonrigid movement presented together produced an advantage for upright faces and a decline in performance for larger test rotations. No interaction was found. This suggests that nonrigid facial movement is represented in a viewpoint-invariant manner while the addition of rigid-head movements encourages a more-viewpoint-dependent encoding.

Visual correlates of prosodic contrastive focus in French: description and inter-speaker variability

Marion Dohen et al. — Sun, 22 Jan 2012 15:39:14 PST

This study is a follow-up of previous studies we conducted on the visible articulatory correlates of French prosodic contrastive focus. A two speaker analysis using an automatic lip-tracking device had shown that these correlates existed and were used in visual perception. However the articulatory strategies depended on the speaker. The purpose of this study was thus to extend the analysis to other speakers, examine the similarities and variabilities and try to identify global tendencies. We recorded five speakers of French with a 3D optical tracker using a 13 sentence (subject-verb-object) corpus and four focus conditions (S, V, O or neutral). An articulatory analysis confirmed that visible articulatory correlates exist for all the speakers. The strategies used are mainly of two types: absolute and differential. An analysis of other facial movements showed that an eyebrow raising and/or a head nod can signal focus. This association is however highly inter- and intra-speaker dependent.

Linking the structure and perception of 3-D faces: Gender, ethnicity and expressive posture

Guillaume Vignali et al. — Sun, 22 Jan 2012 15:39:13 PST

A statistical study of human face shape is reported whose overall goal was to identify and characterise salient components of facial structure for human perception and communicative behaviour. A large database of 3-D faces has been constructed and analysed for differences in ethnicity, sex, and posture. For each of more than 300 faces varying in race/ethnicity (Japanese versus Caucasian) and sex, nine postures (smiling, producing vowels, etc) were recorded. Principal components analysis (PCA) and linear discriminant analysis (LDA) were used to reduce the dimensionality of the data and to provide simple, yet reliable reconstruction of any face from components corresponding to the sex, ethnicity, and posture of the face. Thus, it appears that any face can be reconstructed from a small set of linear and intuitively salient components. Psychophysical tests confirmed that the shape is sufficient to estimate sex and ethnicity. Subjects were asked to judge the sex and ethnicity of (a) natural faces and (b) faces synthesised by randomly combining principal component coefficients within the database. Subjects successfully discriminated ethnicity and sex independently of posture, verifying that different combinations of components are required and in differing amounts. Finally, implications of these results for animation and face recognition are discussed, incorporating results of studies currently underway that examine the 'face print' residue of the sex - ethnicity factor analysis.

Effects of haptic feedback on the perception of ambiguous visual stimuli

Harold C. Hill et al. — Sun, 22 Jan 2012 15:39:11 PST

Touch has the potential to affect the interpretation of visual cues to depth. Three experiments are reported investigating the effect of single point haptic feedback (Phantom + Reachin API) on depth reversal for the Mach card and hollow-face illusions. In experiment 1 perspective, stereo and touch cues all affected perception of the Mach card. Touch was not as effective as stereo in resolving ambiguity but did facilitate the perception of concave stimuli. When shown in reverse perspective, both stereo and touch were needed to overrule the percept of convexity. Experiment 2 used reverse perspective stimuli, adding rigid rotation as an addition visual cue. Touch and stereo were again effective in disambiguating the stimulus but motion was not. Experiment 3 used the hollow-face illusion and showed that stereo, touch and orientation (upright/inverted) all interacted in affecting perceived convexity. There was some evidence for learning, with the seen face more like to appear concave after touch than before. Stylus visibility was found not to affect the interpretation of depth. The results are interpreted as evidence that, while touch does not necessarily resolve visual ambiguity, it can affect the interpretation of visual depth cues and may be particularly effective in combination with stereo.

Evaluating the effectiveness of pixelation and blurring in masking the identity of familiar faces

Karen Lander et al. — Sun, 22 Jan 2012 15:39:11 PST

Two experiments are reported that assess how well the identity of highly familiar (famous) faces can be masked from short naturalistic television clips. Recognition of identity was made more difficult by either pixelating (Experiment 1) or blurring (Experiment 2) the viewed face. Participants were asked to identify faces from both moving and static clips. Results indicated that participants were still able to recognize some of the viewed faces, despite these image degradations. In addition, moving images of faces were recognized better than static ones. The practical and theoretical implications of these findings are discussed.

Application of fuzzy NARX to human gait modelling and identification

Seyed Hesami et al. — Sun, 22 Jan 2012 15:39:09 PST

Review of 'Handbook of face recognition'

Harold C. Hill — Sun, 22 Jan 2012 15:39:08 PST

The Handbook of Face Recognition is a collection of chapters designed as an introduction to the state-of-the-art in automatic face recognition. Recognition of identity is the primary focus, but face detection and expression categorisation are also covered in some detail. While this book is largely written by and aimed at engineers, the study of face recognition has always been a multidisciplinary exercise and this volume provides a valuable summary of one discipline's contribution.

Perception of human gestures through observing body movements

Seyed Hesami et al. — Sun, 22 Jan 2012 15:39:06 PST

Adaptation to differences in 3-d face shape across changes in viewpoint and texture

Harold C. Hill et al. — Sun, 22 Jan 2012 15:39:05 PST

Face-adaptation effects can provide clues as to the nature of underlying representations. Two experiments are reported in which an adaptation paradigm was used to test for adaptation to differences in face shape across changes in image properties. All experiments made use of synthesised 3-D face models based on principal components analysis of a database of laser scans of faces drawn from two distinct populations, Japanese and Caucasian. Participants were asked to make an ethnicity judgment. 3-D shape is known to be useful in this task and its variation can be captured by linear discriminant analysis to give a quantifiable variable. For both experiments the point of subjective equality (PSE) was measured before and after adaptation to a set of faces defined in terms of one of the distinct populations. The viewpoint and the texture map used for rendering the adapting faces were varied between subjects. For experiment 1, the pre-adaptation and post-adaptation PSE was measured for faces shown in 15° or 45° views, with adapting stimuli shown in one of these views. Adaptation, that is movement of the PSE in the direction of the adapting population, was found (F1,8 = 20.1, p k 0.05) independent of adapting or test view. In experiment 2, pre-test and post-test PSE was measured for 0°, 30°, and 60° views. Adaptation was always to the 30° view but the texture map used at adaptation varied between subjects. The texture used was an ethnicity specific average, either consistent or inconsistent with the shape information. Again, there was adaptation dependent on the shape defined ethnicity of the adapting stimuli (F1,12 = 36.2, p k 0.05). This was independent of texture map and test view. The results are interpreted as evidence for the high-level representation of 3-D face shape independent of view and surface reflectance.

It's not what you say but the way you say it: Matching faces and voices

Karen Lander et al. — Sun, 22 Jan 2012 15:39:04 PST

Recent studies have shown that the face and voice of an unfamiliar person can be matched for identity. Here the authors compare the relative effects of changing sentence content (what is said) and sentence manner (how it is said) on matching identity between faces and voices. A change between speaking a sentence as a statement and as a question disrupted matching performance, whereas changing the sentence itself did not. This was the case when the faces and voices were from the same race as participants and speaking a familiar language (English; Experiment 1) or from another race and speaking an unfamiliar language (Japanese; Experiment 2). Altering manner between conversational and clear speech (Experiment 3) or between conversational and casual speech (Experiment 4) was also disruptive. However, artificially slowing (Experiment 5) or speeding (Experiment 6) speech did not affect crossmodal matching performance. The results show that bimodal cues to identity are closely linked to manner but that content (what is said) and absolute tempo are not critical. Instead, prosodic variations in rhythmic structure and/or expressiveness may provide a bimodal, dynamic identity signature.

Biological motion and autism spectrum disorder

K Ruparelia et al. — Sun, 22 Jan 2012 15:39:03 PST

Studies have shown that typically developing individuals are very sensitive to visual cues generated by humans and animals. The human visual system enables us to categorise sex and identity through recognition of biological motion of faces, and these abilities take place at less than 12 months of age. In autism, there is evidence of both face- and motion-processing deficits, yet we do not know the extent of any deficits in facial-motion processing. To investigate whether motion provides useful information for categorising faces we conducted a computer-animated study on a group of adults with autism and a matched control group. The categorisation process involved discriminating sequences of animations of an average head with movements captured from real people. These stimuli had identical spatial characteristics and differed only in the way they moved. They were shown upright, inverted, forwards, and backwards. We report that the autistic observers showed consistently higher discrimination thresholds in all conditions than the control group. This corresponds with previous findings of visual recognition of biological motion with point-light displays. This finding is discussed with respect to extracting sophisticated information for social interaction and communication from faces and biological motion in the autistic population.

How different is different? Investigating criteria for different identity judgments

Harold C. Hill et al. — Sun, 22 Jan 2012 15:39:02 PST

Any face seen for the first time will have a closet neighbour in memory. In order to avoid false alarms, we must be able to distinguish similar from identical faces. Work is reported investigating same/different judgments as a function difference in three dimensional shape defined in terms of standard deviation in a principal component based face space. The aim is to determine the criterion difference below which observers respond “same”. A threshold corresponding to a dprime of 1 was also calculated. Both were first measured under three conditions – same view images, different view images and animated images of the faces rotating. Criterion did not differ significantly between presentation conditions and the mean criterion corresponded to 0.45 SD when the average face was used as the reference. Threshold did differ significantly between conditions, and was significantly higher for different view faces (.65SD) than for same view (.39SD) or animated faces (.44SD). The results are interpreted as indicating that the criterion for same/different judgements is relatively stable across presentation conditions, but that sensitivity is lower when making the judgement across a change in view. Further work will investigate whether criterion varies as a function of distinctiveness.

Categorizing sex and identity from the biological motion of faces

Harold C. Hill et al. — Sun, 22 Jan 2012 15:39:01 PST

Head and facial movements can provide valuable cues to identity in addition to their primary roles in communicating speech and expression. Here we report experiments in which we have used recent motion capture and animation techniques to animate an average head. These techniques have allowed the isolation of motion from other cues and have enabled us to separate rigid translations and rotations of the head from nonrigid facial motion. In particular, we tested whether human observers can judge sex and identity on the basis of this information. Results show that people can discriminate both between individuals and between males and females from motion-based information alone. Rigid head movements appear particularly useful for categorization on the basis of identity, while nonrigid motion is more useful for categorization on the basis of sex. Accuracy for both sex and identity judgements is reduced when faces are presented upside down, and this finding shows that performance is not based on low-level motion cues alone and suggests that the information is represented in an object-based motion-encoding system specialized for upright faces. Playing animations backward also reduced performance for sex judgements and emphasized the importance of direction specificity in admitting access to stored representations of characteristic male and female movements.

Judging sex and identity from isolated facial motion

Harold C. Hill et al. — Sun, 22 Jan 2012 15:38:59 PST

Observers made judgements about the sex and identity of an average computer head model animated using 3D motion capture data. The animations differed only in their movement, other cues being constant. Despite the absence of most cues observers were able to judge sex at levels significantly better than chance (Mean 61%, SE 2%. t(15) = 6.5, p<<.05). This finding was replicated using a 2-AFC task (59%, 2%. t(13) = 5.0 p<<.05). Performance with animations played backwards (54%, 3%. t(13) = 1.3, p>.1) or upside-down (57%, 4%. t(13) = 1.9, p=.08) was near chance. Motion provides useful information for sex judgements and this information is at the level of global facial movement rather than individual frames or local image velocities. Observers could also sort animations according to identity--putting different examples of the same person together--significantly better than chance (t(15)= 5.3, p<<.05). They could also identify the odd-one-out of three better than chance (57%, 3%. t(11) = 8.09, p<<.05). This task was disrupted by inversion in the image plane (50%, 3%. t(11) = 4.4, p<<.05) but not by playing backwards (57%, 3%. p>.1). With both manipulations performance remained above chance (t(11) = 5.7, p<<.05 and t(11) = 7.7, p<<.05 respectively). Motion must provide cues to identity that are common between different examples of the same person but different between individuals. This information is disrupted by inversion but not by being played backwards. Thus a simple representation of facial movement provides useful information for face processing tasks. The inversion effects implicate face processing mechanisms. Whereas subjects may be using low-level motion cues to some extent in the identity task, the sex judgement task appears to depend upon sensing dynamic changes in facial configuration.

Recognizing prosody from the lips: Is it possible to extract prosodic focus from lip features?

Marion Dohen et al. — Sun, 22 Jan 2012 15:38:58 PST

The aim of this chapter is to examine the possibility of extracting prosodic information from lip features. The authors used two lip feature measurement techniques in order to evaluate the “lip pattern” of prosodic focus in French. Two corpora with Subject-Verb-Object (SVO) sentences were designed. Four focus conditions (S, V, O or neutral) were elicited in a natural dialogue situation. In the first set of experiments, they recorded two speakers of French with front and profile video cameras. The speakers wore blue lipstick and facial markers. In the second set, the authors recorded five speakers with a 3D optical tracker. An analysis of the lip features showed that visible articulatory lip correlates of focus exist for all speakers. Two types of patterns were observed: absolute and differential. A potential outcome of this study is to provide criteria for automatic visual detection of prosodic focus from lip data.

The perception and production of phones and tones: The role of rigid and non-rigid face and head motion

Denis Burnham et al. — Sun, 22 Jan 2012 15:38:57 PST

There is evidence, mostly with phones (consonants & vowels), that visual concomitants of articulation facilitate speech perception. Here the visual concomitants of lexical tone are considered. In tone languages fundamental frequency variations signal lexical meaning. In a word identification experiment with auditory-visual words differing only in tone, Cantonese perceivers performed above chance in a Visual Only condition. A subsequent study showed augmentation of word pair discrimination in noise in an Auditory-Visual compared to an Auditory Only condition for Cantonese, tonal Thai speakers, and even non-tone Australian speakers). The source of this perceptual information was sought in an OPTOTRAK production study of a Cantonese speaker. Functional Data Analysis (FDA) and Principal Component (PC) extraction suggests that the salient PCs to distinguish tones involve rigid motion of the head rather than non-rigid face motion. Results of a final perception study using OPTOTRAK output in which rigid or non-rigid motion could be presented independently in tone differing or phone differing conditions, suggests that non-rigid motion is most useful for the discrimination of phones, whereas rigid motion is most useful for the discrimination of tones.

Do infants use a generalised motion processing system for discriminating facial motion?

Janine Spencer et al. — Sun, 22 Jan 2012 15:38:56 PST

Previous research has shown that infants aged 4 to 8 months can perceive and discriminate facial motion (Spencer et al, 2004 Perception 33 Supplement, 56). Here we report a study in which we used an animated average face to test infants' ability to discriminate viewpoint invariance of biological facial motion. We familiarised infants with the animated face using one motion sequence of an actor telling a joke. The stimulus was either a full-face view (0°) or a face rotated 60° about a vertical axis (between full-face and profile). Once familiarised, infants were presented with two stimuli. One was the same as the habituation stimulus, shown at a different angle. The other was a novel motion sequence drawn from a different actor telling a different joke. Infants demonstrated a significant preference for the novel motion sequence, suggesting that they were able to encode the face-based motion in a viewpoint-invariant manner. However, in a parallel study of identical design but using inverted stimuli, infants did not demonstrate a face-inversion effect. This indicates that infants were using a general motion-processing system to discriminate between the sequences rather than a face-specific process.