Title: Multimodal assessment of neuropsychiatric disorders using audiovisual recordings

Date: Oct 30th (Monday)

Time: 1pm – 3pm

Location: Woodruff Memorial Research Building, 101 Woodruff Circle , 4th Floor, BMI classroom, Emory University

Virtual: https://zoom.us/j/93683399472

Zifan Jiang

Machine Learning PhD Student

Biomedical Engineering
Georgia Institute of Technology

Committee

1 Dr. Gari D. Clifford (Advisor)

2 Dr. Eva L. Dyer

3 Dr. Cassie S. Mitchell

4 Dr. Bilal Haider

5 Dr. Ali Bahrami Rad

Abstract

Over one billion people worldwide live with a neuropsychiatric disorder,  yet most do not have access to adequate diagnosis and care. Accurate, fast, and accessible detection of those disorders is critical to early and effective interventions. Over the last decade, digitally administered assessments have emerged as one of the most promising approaches. Moreover, the increasing use of telemedicine in psychiatry and neurology in recent years presented an unprecedented opportunity to use audiovisual data for accessible neuropsychiatric assessments without the limitation of geographical location and specialized hardware.

This dissertation describes the use of low-cost audiovisual data collected from in-lab and remote mobile devices to assess neuropsychiatric conditions by extracting and combining various behavioral and physiological indicators. First, we showed that facial and speech emotions can be effectively estimated from audiovisual data collected in interviews and used for major depressive disorder evaluation. Then, we presented the automated assessment of cognitive impairment using facial emotions and viewing behaviors recognized from videos passively collected from a mobile device. Lastly, we further improved the scalability and accessibility by extracting facial, vocal, linguistic, and cardiovascular features from audiovisual data collected remotely from heterogeneous mobile devices and validating them in both clinician-rated and self-rated mental health condition evaluation.