AI-Powered Active Speaker Detection for Multi-Camera Video Production

MicrocosmWorks developed a multimodal fusion model that correlates lip movement visual features extracted from each camera feed with the audio signal using cross-attention layers. The model outputs per-frame speaker probability scores for each visible face, achieving 94% accuracy even when multiple participants speak simultaneously.

MicrocosmWorks optimized the inference pipeline to run on NVIDIA T4 GPUs with TensorRT acceleration, achieving under 150ms end-to-end latency from frame capture to speaker identification. This latency is well within the acceptable range for live production switching, where typical cut delays are 300-500ms.

MicrocosmWorks trained the model on diverse occlusion scenarios and implemented a temporal smoothing algorithm that maintains speaker tracking through brief occlusions using audio-only confidence scores. When visual confidence drops below a threshold, the system falls back to audio source localization using beamforming data from multi-microphone arrays.

MicrocosmWorks built a companion control module that translates speaker detection outputs into standard tally/control signals compatible with Blackmagic ATEM via the ATEM SDK and NewTek NDI for TriCaster systems. Production directors can set the system to auto-switch or advisory mode where it suggests cuts without executing them.

MicrocosmWorks builds custom AI video analysis systems at rates of $30-$50/hr, with a multi-camera active speaker detection system including model training, TensorRT optimization, and switcher integration typically requiring 500-750 development hours. The model training phase requires GPU compute resources that usually add $2,000-$5,000 to the project cost.