Traditional synchronization schemes of multimedia applications are based on temporal relationships between inter- and intrastreams. These schemes do not provide good synchronization in the presence of random delay. As a solution, this paper proposes an adaptive content-based synchronization scheme that synchronizes multimedia streams by accounting for content in addition to time. This approach to synchronization is based on the fact that having two streams sampled close in time does not always imply that these streams are close in content. The proposed scheme primary contribution is the synchronization of audio and video streams based on content. The secondary contribution is adapting the frame rate based on content decisions. Testing adaptive content-based and adaptive time-based synchronization algorithms remotely between the American University of Beirut and Michigan State University showed that the proposed method outperforms the traditional synchronization method. Objective and subjective assessment of the received video and audio quality demonstrated that the content-based scheme provides better synchronization and overall quality of multimedia streams. Although demonstrated using a video conference application, the method can be applied to any multimedia streams including nontraditional ones referred to as supermedia like control signals, haptic, and other sensory measurements. In addition, the method can be applied to synchronize more than two streams simultaneously. 1. Introduction Three types of applications are recognized over the Internet: asynchronous, synchronous, and interactive synchronous. Asynchronous applications do not involve simultaneous transfer of media streams, like file transfer and web browsing. Such applications do not require strict timing from the network. In synchronous applications, such as video clip viewing, multiple media streams (audio and video) are to be transferred simultaneously. In this type of applications, synchronous display of media streams is required. However, since no interactivity is involved, these applications do not enforce strict delay requirements on the network. In this case, applications can buffer data before starting to render audio and video to mask the network delays. As for the interactive synchronous applications, they include real-time applications with an interactive nature such as video conferencing and networked gaming. They also include nontraditional applications using “supermedia” such as remote surgery and teleoperation [1, 2]. Due to their interactive nature these applications can
References
[1]
I. Elhajj, N. Xi, W. K. Fung, Y. H. Liu, Y. Hasegawa, and T. Fukuda, “Supermedia enhanced internet based telerobotics,” Proceedings of the IEEE, vol. 91, no. 3, pp. 396–421, 2003.
[2]
I. Elhajj, H. Hummert, N. Xi, and Y. H. Liu, “Synchronization and control of supermedia transmission via the internet,” in Proceedings of the International Symposium on Intelligent Multimedia, Video and Speech Processing, Hong Kong, 2001.
[3]
C. M. Huang and C. Wang, “Synchronization for interactive multimedia presentations,” IEEE Multimedia, vol. 5, no. 4, pp. 44–61, 1998.
[4]
S. Khanvilkar, F. Bashir, D. Schonfeld, and A. Khokhar, “Multimedia networks and communication,” in The Electrical Engineering Handbook, W. Chen, Ed., Academic Press, 2004.
[5]
H. McGurk and J. MacDonald, “Hearing lips and seeing voices,” Nature, vol. 264, no. 5588, pp. 746–748, 1976.
G. M. Garner, F. F. Feng, E. H. S. Ryu, and K. Den Hollander, “Timing and synchronization for audio/video applications in a converged residential ethernet network,” in Proceedings of the 3rd IEEE Consumer Communications and Networking Conference, vol. 2, pp. 883–887, January 2006.
[8]
Y. Xie, C. Liu, M. J. Lee, and T. N. Saadawi, “Adaptive multimedia synchronization in a teleconference system,” Multimedia Systems, vol. 7, no. 4, pp. 326–337, 1999.
[9]
M. Yang, N. Bourbakis, Z. Chen, and M. Trifas, “An efficient audio-video synchronization methodology,” in Proceedings of th IEEE International Conference on Multimedia and Expo, pp. 767–770, Beijing, China, July 2007.
[10]
D. Young, S. SampathKumar, and P. Rangan, “Content-based inter-media synchronization,” in Proceedings of the Multimedia Computing and Networking, vol. 2417, pp. 202–214, San Jose, Calif, USA, February 1995.
[11]
C. H. Wei and C.T. Li, “Design of content-based multimedia retrieval,” in Proceedings of the 5th WSEAS International Conference on Circuits, Systems, Electronics, Control & Signal Processing, Dallas, Tex, USA, 2006.
[12]
B. Furht and P. Saksobhavivat, “A fast content-based multimedia retrieval technique using compressed data,” in Proceedings of the Conference on Multimedia Storage and Archiving Systems III, pp. 561–571, Boston, Mass, USA, 1998.
[13]
J. Calic, S. Sav, E. Izquierdo, S. Marlow, N. Murphy, and N. E. O'Connor, “Temporal video segmentation for real-time key frame extraction,” in Proceedings of the IEEE International Conference on Acoustic, Speech, and Signal Processing, pp. IV/3632–IV/3635, , Orlando, Fla, USA, May 2002.
[14]
S. C. Chen, M. L. Shyu, N. Zhao, and C. Zhang, “An affinity-based image retrieval system for multimedia authoring and presentation,” in Proceedings of the ACM International Workshop On Multimedia Databases, New Orleans, La, USA, 2003.
[15]
V. S. W. Eide, F. Eliassen, O. C. Granmo, and O. Lysne, “Supporting timeliness and accuracy in distributed real-time content-based video analysis,” in Proceedings of the 11th ACM International Conference on Multimedia, pp. 21–32, Berkeley, Calif, USA, November 2003.
[16]
A. J. Perrott, A. T. Lindsay, and A. P. Parkes, “Real-time multimedia tagging and content-based retrieval for CCTV surveillance systems,” in Proceedings of the Internet Multimedia Management Systems III, Boston, Mass, USA, 2002.
[17]
H. Blanken, T. Grabs, H.-J. Schek, R. Schenkel, and G. Weikum, “Intelligent search on XML data,” in Lecture Notes in Computer Science (LNCS), vol. 2818, pp. 217–230, Springer, Berlin, Germany, 2003.
[18]
Y. Li, A. Markopoulou, J. Apostolopoulos, and N. Bambos, “Content-aware playout and packet scheduling for video streaming over wireless links,” IEEE Transactions on Multimedia, vol. 10, no. 5, Article ID 4543842, pp. 885–895, 2008.
[19]
Y. Li, Z. Li, M. Chiang, and A. Robert Calderbank, “Content-aware distortion-fair video streaming in congested networks,” IEEE Transactions on Multimedia, vol. 11, no. 6, Article ID 5235177, pp. 1182–1193, 2009.
[20]
R. Steinmetz, “Human perception of jitter and media Synchronization,” IEEE Journal on Selected Areas in Communications, vol. 14, no. 1, pp. 61–72, 1996.
[21]
F. Halsall, Multimedia Communications: Applications, Networks, Protocols and Standards, Addison-Wesley, Swansea, UK, 2001.
[22]
K. Peker and A. Divakaran, Framework for Measurement of the Intensity of Motion Activity of Video Segments, Mitsubishi Electric Research Laboratories, 2003.
[23]
D. Wu, Y. T. Hou, and Y. Q. Zhang, “Transporting real-time video the challenges and approaches,” Proceedings of the IEEE, vol. 88, no. 12, pp. 1855–1874, 2000.
[24]
I. Richardson, Video Codec Design: Developing Image and Video Compression Systems, John Wiley & Sons, 3rd edition, 2002.
[25]
A. B. Watson, “Toward a perceptual video quality metric,” in Proceedings of the Conference on Human Vision and Electronic Imaging III, Proceedings of SPIE, San Jose, Calif, USA, 1998.
[26]
M. Narbutt and L. Murphy, “Adaptive payout buffering for audio/video transmission over the internet,” in Proceedings of the IEE UK Teletraffic Symposium, Dublin, Ireland, May 2001.
