A Smart Background Music Mixing Algorithm for Portable Digital Imaging Devices

Abstract

In this paper, we propose a smart background music (BGM) mixing algorithm for portable digital imaging devices to enable users to enjoy video content with BGM. The proposed algorithm automatically adjusts the BGM output energy based on the activity and energy of foreground audio (FGA) contained in a video file. To this end, the proposed algorithm classifies each segment of FGA as speech, non-speech, or a mixed signal. After that, it estimates a scale factor for mixing FGA and BGM according to the signal classification result and the energy of FGA. In addition, a fade-in and fade-out process is incorporated in the proposed algorithm in order to improve the perceptual quality of output audio at the boundaries where signal classification is changed. In order to demonstrate the effectiveness of the proposed algorithm, we implement it on a portable digital imaging device in real time and compare the user's preference of the proposed algorithm with those of conventional algorithms that mixes FGA with BGM based on voice activity detection or a predefined fixed scale factor. It is shown from the experiments that the proposed algorithm is pretty much preferred by around 79%, compared to the conventional algorithms.