Dr. Muchao Ye, Assistant Professor in the Department of Computer Science (CS), College of Liberal Arts and Sciences (CLAS), at the University of Iowa, recently received a National Science Foundation Grant (NSF) for his research: “Toward Human-Centered, Generalizable, and Verifiable Automated Video Surveillance: A Verbalized Vision-Language Model Paradigm.”
Ye, the project’s principal investigator and lead, is joined by collaborator and project co-lead Dr. Pan He, Assistant Professor in the Department of Computer Science and Software Engineering (CSSE) at Auburn University’s Samuel Ginn College of Engineering.
Making Video Surveillance Systems Smarter and More Trustworthy
“Our research is about making automated video surveillance systems smarter, more understandable, and more trustworthy. Today, surveillance cameras can generate enormous amounts of video, but people still have to spend a great deal of time watching or reviewing that footage to understand what happened. Artificial intelligence can help automate this process, but current systems may struggle with complex events and often provide an answer without clearly explaining how they reached it,” says Dr. Ye.
“In this project, we want to develop AI systems that can not only recognize and describe important events in videos but also explain their reasoning in a way that people can understand and verify,” explains Ye. “For example, instead of simply flagging something as an unusual event, the system could describe what happened, why it considers the event unusual, and provide information that helps a human operator verify its report.”
The desired outcome is an automated video surveillance framework that reduces the amount of video people need to review while still keeping humans informed and involved in important decisions.
“Ultimately, we hope this research will make AI-assisted video analysis more useful and trustworthy for applications such as traffic monitoring and workplace safety,” offers Dr. Ye.
More About Dr. Muchao Ye
“My previous research has focused on making AI systems for video understanding not only accurate, but also more understandable to people. In my CVPR (computer vision and pattern recognition) work, ‘VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models,’ I developed a verbalized learning approach that enables vision-language models to identify unusual events in videos while also generating human-understandable explanations of why those events are considered anomalous.”
That work made Ye particularly interested in a broader question: “If we want AI to assist people in real-world video surveillance, how can we make these systems capable of explaining their reasoning, understanding more complex events over time, and producing outputs that people can actually verify and trust?”
“This NSF project is a natural extension of that research. We are taking the ideas behind VERA beyond individual anomaly detection tasks and developing a more comprehensive framework for human-centered automated video surveillance.”
Collaboration with Dr. Pan He of Auburn University
“I first met Dr. Pan He at a conference when I was still a PhD student,” explains Muchao Ye. “We connected through our shared research interests in computer vision and artificial intelligence and have stayed in touch. Dr. He is an expert in computer vision and has worked on real-world video understanding problems, including near-accident detection in traffic videos. His experience in analyzing complex events across space and time makes his expertise particularly well suited to this project.”
Their respective roles in the project reflect their complementary expertise. Dr. Ye will serve as the overall project lead and will primarily lead the work on developing methods that allow vision-language models to learn and explain human-understandable concepts, as well as methods for making AI-generated surveillance reports more verifiable and trustworthy. Dr. He will primarily lead the research on spatiotemporal video understanding—developing methods that allow AI systems to synthesize information across long video sequences and multiple camera views to better understand and report complex events. The two will also work closely across all parts of the project to ensure that these components come together as a unified system.
Improving the Real-World Impact of AI
The Department of Computer Science at the University of Iowa is excited to have several faculty, including Dr. Muchao Ye, who work at the leading edge of AI and machine learning.
“I am very grateful to the National Science Foundation for supporting this research. This award gives us an exciting opportunity to explore fundamental research questions in AI while working toward technologies that can have meaningful real-world impact,” says Dr. Muchao Ye. “I am also especially excited that the project will provide research and educational opportunities for our students and allow us to collaborate across institutions.”
Learn with Dr. Muchao Ye
Muchao Ye is currently teaching a course in Advanced Artificial Intelligence for undergraduate and graduate Computer Science majors, undergraduate Computer Science & Engineering majors, and undergraduate Data Science majors.