Speaker:
Shinji Watanabe, Associate Professor, Carnegie Mellon University

Abstract:
Building speech foundation models is no longer the exclusive domain of large companies, but reproducing one still takes more than a published architecture. The hard part is the groundwork: collecting, cleaning, and automatically annotating data before training begins. Large-scale audio collections are abundant, yet raw audio and transcripts are often misaligned, mislabeled, or otherwise unusable without careful processing. This talk uses our fully transparent open Whisper-style speech models (OWSM) as a case study and focuses on the groundwork behind them. I show how OWSM v4 narrowed a web-scale pool of multilingual speech, around 500K hours from YODAS v2, to a usable training set through alignment, language identification, and CTC-based scoring. I also show how the same kind of pipeline carries over to YODAS v3, a larger, more multilingual release recorded at higher fidelity. I argue that task-specific models are not a stepping stone to be replaced by foundation models; they are the tools that make the data usable. Finally, I present the Phonetic Open Whisper-Style Speech Foundation Model (POWSM), which applies the same recipe to phone recognition, showing that task-specific linguistic and phonetic tools are the key to the groundwork. I then briefly discuss unified speech models spanning generation and understanding as a direction that follows the same open-science principles.

Bio:
Shinji Watanabe is an Associate Professor at Carnegie Mellon University, Pittsburgh, PA. He received his B.S., M.S., and Ph.D. (Dr. Eng.) degrees from Waseda University, Japan. He was a research scientist at NTT Communication Science Laboratories, Kyoto, Japan, from 2001 to 2011, a visiting scholar at Georgia Institute of Technology, Atlanta, GA, in 2009, and a senior principal research scientist at Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA, from 2012 to 2017. Before Carnegie Mellon University, he was an associate research professor at Johns Hopkins University, Baltimore, MD, USA, from 2017 to 2020. His research interests include automatic speech recognition, speech enhancement, spoken language understanding, and machine learning for speech and language processing. He has published over 600 papers in peer-reviewed journals and conferences and received several awards, including the best paper award from Interspeech 2024 and EMNLP 2024. He has been a member of several technical committees, including the IEEE Signal Processing Society Speech and Language Technical Committee (SLTC, Chair). He is an IEEE and ISCA Fellow.