Media Processing

Phonexia Media Processing is a utility service designed to manipulate audio streams. It provides essential audio processing capabilities such as channel mixing, splitting, and merging, which are often required as preprocessing steps for other Phonexia technologies or for post-processing results.

Key features

• Audio Conversion: Converts audio formats and manipulates audio channels (mix, split, or extract specific channels).
• Audio Merging: Combines multiple audio streams into a single multichannel stream.
• Audio Splitting: Splits a single audio stream into multiple channels or streams based on a time segmentation (e.g., using output from Speaker Diarization or Voice Activity Detection).

Operations

The service exposes its functionality through three main gRPC methods: Convert, Merge, and Split.

Convert

The Convert operation performs synchronous media conversion. Its primary function is to handle channel manipulation based on the provided configuration.

Supported channel modes:

• MIX: Mixes all input channels into a single mono audio stream.
• SPLIT: Separates input channels into multiple distinct mono audio streams.
• KEEP: Retains only the specific channels defined in the request, discarding the others.

This operation is useful when you need to prepare audio for technologies that expect specific channel formats (e.g., mono input) or when you need to extract a specific channel from a multichannel recording.

Merge

The Merge operation accepts multiple input audio streams and combines them into a single multichannel audio stream.

• Input: Multiple audio streams, each identified by a unique tag.
• Output: A single audio stream where each input stream becomes a distinct channel.
• Order: The order of channels in the output is preserved based on the input sequence.

This is effectively the inverse of splitting channels. It allows constructing a multichannel recording from separate sources (e.g., customer and agent recordings).

Split

The Split operation takes a single audio stream and a segmentation (a list of time intervals assigned to tags) and produces separated audio outputs. This is particularly useful for separating speakers based on Diarization results or isolating speech segments based on Voice Activity Detection.

info

The input audio for the Split operation must be mono (1 channel). If you have multichannel audio, use the Convert operation with ChannelMode.MIX first.

• Input: A single mono audio stream and a list of Segments.
• Segmentation: Each segment maps a time interval (start, end) to a tag (e.g., Speaker ID).
  • Segments for the same tag are automatically normalized (sorted by time and overlapping segments are merged).
• Silence Filling: Time ranges not covered by any segment for a given tag are filled with low-amplitude dithered noise (amplitude 0.00001) instead of absolute silence. This prevents issues with artifacts in some processing pipelines that disfavor absolute digital silence.

Output modes:

• MULTICHANNEL: Returns a single multichannel audio stream. Each unique tag from the segmentation becomes a separate channel.
• SEPARATE: Returns multiple independent mono audio streams. Each unique tag produces a separate stream, sent one after another.

Use case: Diarization output processing

If you have a mono recording and a Diarization result (identifying "Speaker A" and "Speaker B"), you can use Split to create a stereo file where "Speaker A" is on the left channel and "Speaker B" is on the right channel (using MULTICHANNEL mode), or two separate files (using SEPARATE mode).

Technical details

Audio format support

The service uses FFmpeg internally for audio processing.

info

There is also a version of the docker image available with free FFmpeg libraries. For specific versions, use the -free tag suffix (e.g., 1.0.0-free). For the latest version, use the free tag. This version uses the ffmpeg-free package from Fedora.

• Input: Supports standard audio formats (WAV, FLAC, MP3, OGG, etc.) and Raw Audio (PCM16, PCM32, A-law, mu-law).
  • For Raw Audio, you must provide the RawAudioConfig in the request (sample rate, encoding, number of channels).
• Output: The service outputs audio in WAV format.

Constraints

• Max Audio Length: The service can be configured with a maximum audio length limit (via --max_audio_length). Requests exceeding this limit will fail.
• Memory: Processing is done using temporary files on disk, but sufficient RAM is required for FFmpeg operations.

API reference

The functionality is defined in the phonexia.grpc.technologies.media_processing.v1 package.

• Service: MediaProcessing
• Methods:
  • Convert(stream ConvertRequest) returns (stream ConvertResponse)
  • Merge(stream MergeRequest) returns (stream MergeResponse)
  • Split(stream SplitRequest) returns (stream SplitResponse)

For detailed message definitions, refer to the Protocol Buffer documentation.