Multiplexing the Elementary Streams of H.264 Video and MPEG4 HE AAC v2 Audio, De-multiplexing and Achieving Lip Synchronization

Television broadcasting applications such as ATSC-M/H, DVB[16] require that the encoded audio and video streams to be transmitted across a network in a single transport stream containing fixed sized data packets that can be easily recognized and decoded at the receiver. MPEG2 part1 specifies two layers of packetization to achieve a transport stream suitable for digital transmission. In a broadcasting system, multiplexing is a process in which two or more elementary streams are converted into a single transport stream ensuring synchronous playback of the elementary streams and proper buffer behavior at the decoder. This paper presents a scheme to multiplex the elementary streams of H.264 video and HE AAC v2 audio using the MPEG2 systems specifications[4], then de-multiplex the transport stream and playback the decoded elementary streams with lip synchronization or audio-video synchronization. This paper briefly introduces the MPEG2 systems, two layers of packetization namely program elementary stream (PES) and transport stream (TS). It also introduces the concept of timestamps. The proposed multiplexing and de-multiplexing algorithms followed to achieve synchronization are explained. It is shown that during decoding the audio-video synchronization is achieved with a maximum skew of 13ms.