Skip to content
Documentation

Multi audio

This document details paella-core support for video playback with multiple audio tracks.

Compatibility

Section titled “Compatibility”

paella-core supports multiple audio tracks through the StreamProvider APIs. However, the underlying video technology being used to play the stream must also support multiple audio tracks. For example, the audio track APIs for .mp4 files are supported by some browsers, but not all, and for that reason the mp4 video plugin does not support multiple audio tracks: the player behaviour would not be the same in all browsers. However, the hls video format does allow the use of video streams with multiple audio tracks, so the hls_video_plugin.md does implement this feature.

Integration with multi track audio

Section titled “Integration with multi track audio”

The integration between the underlying video technology and the paella-core APIs for selecting the active audio tracks is done in the implementation of the video format plugins. Specifically, within the API of a video format plugin, the async supportsMultiaudio() function must be implemented, which returns a bool value indicating whether the video has multiple audio tracks. In this case, if the format supports multiple audio tracks, this function will return true or false depending on whether the particular video stream we are playing has multiple audio tracks. If the underlying technology does not support multiple audio tracks, this function will always return false, and the rest of the multiple audio API functions will remain unimplemented.

See the documentation on video plugins to learn more about integrating multiple audio tracks into a video plugin.

Video manifest

Section titled “Video manifest”

As you can see in more detail in the document video_maniofest.md, if the video has several streams, only one of them can contain an audio track. The rest of the videos will have the audio muted. This is due to limitations on certain platforms. For example, on most mobile platforms, it is not possible to play two videos simultaneously if both have audio.

The stream that is responsible for providing the audio will be the main audio stream, and we use the "role": "mainAudio" attribute to indicate this. If we have several videos with audio tracks, we will obviously have to mark as mainAudio the stream that supports multi-track audio technology, in case they are streams of different types.

Handle multiple audio tracks

Section titled “Handle multiple audio tracks”

The easiest way to handle multiple audio tracks is via the es.upv.paella.audioSelector plugin, within the @asicupv/paella-basic-plugins plugin library. To access these functions programmatically, you can use the APIs of the StreamProvider object.

Additional information

Section titled “Additional information”