Abstract
There is increasing interest in automated measurement approaches, utilizing computer vision and machine learning, in behavioral research. These approaches have the potential to produce objective measurement of early facial expressions, yet there has been limited implementation in infants and young children. This three-study project examined how objective measurement approaches using computer vision and machine learning can be applied to address developmental questions involving early facial expressions. We assessed the reliability between automated measurement of mouth opening in newborn infants and expert observer coding (Study 1), the convergent validity of automated measurement of infant smiling and cry-faces in the face-to-face/still-face procedure (FFSF; Study 2), and the construct validity of objective measurement of social gaze and social gaze involving smiling during a clinical assessment in young children with suspected autism spectrum disorder (ASD; Study 3). Specifically, Study 1 evaluated the performance of a machine learning algorithm that recognized expert coding of the presence and absence of mouth opening at a frame-level from facial features extracted with computer vision, yielding 96% overall accuracy with an expert coder. Study 2 indicated that expert coding and automated measurement produced similar patterns of change in 4-month-old infants’ smiling and cry-face expressions and mother smiling expressions in a widely-used assessment protocol. In Study 3, we employed objective facial expression analysis from first-person video and found associations between objectively measured social gaze behaviors and clinician-rated autism severity scores in children with suspected ASD during an ASD assessment. Results from this project highlight the potential of automated measurement to enhance objective understanding of early facial expressions at different ages and shed light on the reliability and validity of these methods.