On Learning Associations of Faces and Voices

Abstract

In this paper, we study the associations between human faces and voices. Audiovisual integration, specifically the integration of facial and vocal information is a well-researched area in neuroscience. It is shown that the overlapping information between the two modalities plays a significant role in perceptual tasks such as speaker identification. Through an online study on a new dataset we created, we confirm previous findings that people can associate unseen faces with corresponding voices and vice versa with greater than chance accuracy. We computationally model the overlapping information between faces and voices and show that the learned cross-modal representation contains enough information to identify matching faces and voices with performance similar to that of humans. Our representation exhibits correlations to certain demographic attributes and features obtained from either visual or aural modality alone. We release our dataset of audiovisual recordings and demographic annotations of people reading out short text used in our studies.

Downloads

• Paper (pdf, 0.5 MB)
• Supplementary Material (pdf, 14 MB)
• ArXiv

Code

A reference implementation is available under the permissive MIT License. Please cite the paper if you use the software.

• Github repository

Pre-trained models are avaiable as TensorFlow checkpoints.

• Model trained with voice as the reference modality (zip, 1.3GB)
• Model trained with face as the reference modality (zip, 1.3GB)

Dataset

The following dataset is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0). Please cite the paper if you use any part of this dataset.

• Face/voice dataset (zip, 1.8 GB): audiovisual recordings of speeches
• Face/voice dataset demographic annotations (txt, 4 kB): gender, ethnicity, first language, and age annotations
• VoxCeleb training/test sample split (txt, 20 kB): split, with no shared identities, used in our paper (1: train; 2: test)
• VoxCeleb test sample demographic annotations (txt, 8 kB): gender, ethnicity, first language, and age annotations

The demographic annotations of our dataset and the VoxCeleb test set consist of 5 columns, which denote the following.

• Column 1: anonymized person identifier (in our dataset) or celebrity name (in VoxCeleb)
• Column 2: gender (1: male; 2: female)
• Column 3: first language (1: English as first language; 2: other language)
• Column 4: age
• Column 5: ethnic group (1: American Indian; 2: Asian and Pacific Islander; 3: black or African American; 4: Hispanic or Latino; 5: non-Hispanic white; 6: others)

BibTeX

@inproceedings{Kim2018on,
  author    = {Changil Kim and Hijung Valentina Shin and Tae-Hyun Oh and Alexandre Kaspar and Mohamed Elgharib and Wojciech Matusik},
  title     = {On Learning Associations of Faces and Voices},
  booktitle = {Proceedings of Asian Conference on Computer Vision (ACCV)},
  year      = {2018},
}

Acknowledgments

This work is funded in part by the QCRI–CSAIL computer science research program. Changil Kim is supported by a Swiss National Science Foundation fellowship P2EZP2 168785. We thank Sung-Ho Bae for his help in this work.