14:20 - 14:40
Zhiyao Duan
University of Rochester
Audio-visual Analysis of Musical Performances
Music is not just an art of sound. The visual aspects of musical performances play an important role in expressing performers’ ideas and emotions and in engaging the audience in live concerts and music videos. In this talk, I will present our work on audio-visual analysis of musical performances. Specifically, we propose a framework to solve the association between music sound tracks and players shown in the video for string ensembles. The basic idea is to correlate acoustic events and their corresponding motions (e.g., note onsets and bow strokes, vibrato pitch fluctuations and fingering hand vibration). With the same techniques, we further propose an audio-visual approach to vibrato detection and analysis for string ensembles. We show that it significantly outperforms audio-only approaches. This opens the door to many musical performance analysis tasks such as the analysis of expressiveness and playing techniques in polyphonic music.
Design downloaded from free website templates.