oaDub — Real-Time Multimodal AI Simultaneous Tab Dubbing Engine
An advanced Chrome/Edge Manifest V3 extension streaming 16kHz raw PCM tab audio over bidirectional WebSockets to Google Gemini Live API, synthesizing instant 24kHz voice translation with zero server intermediaries.
Project Overview
Language barriers on global video platforms such as YouTube, Coursera, Twitch, and web conferencing platforms hinder continuous learning and technical mastery. Traditional subtitle extensions impose continuous cognitive load, whereas legacy AI dubbing platforms require minutes of batch post-processing before rendering an alternative video track.
oaDub introduces an instant, real-time auditory translation paradigm. Built as a high-performance browser extension complying with the strict Chrome Manifest V3 lifecycle, oaDub captures live media tab audio, streams raw 16kHz Little-Endian PCM audio chunks across a low-latency bidirectional WebSocket directly to Google Gemini 3.5 Live Multimodal API, and receives streaming 24kHz PCM audio buffers. Through a client-side Web Audio API dual-channel mixer with background original volume ducking, oaDub delivers real-time simultaneous voice translation in under 400 milliseconds—all with zero intermediate servers and complete API key privacy.
flowchart LR
subgraph Browser ["Web Browser Environment (Manifest V3)"]
subgraph Tab ["Active Video Tab (YouTube / Coursera)"]
Video["Media Stream / Tab Audio"]
end
subgraph Offscreen ["Manifest V3 Offscreen Document"]
Capture["chrome.tabCapture API"]
AudioCtx["Web Audio API Context\n16kHz Downsampler"]
PCM_Enc["Little-Endian PCM Encoder"]
Capture --> AudioCtx --> PCM_Enc
end
subgraph ExtensionEngine ["Extension Background & Audio Mixer"]
Mixer["Dual-Channel Audio Mixer\nOriginal Audio Ducking (-18dB)"]
Speaker["🔊 Output Device (Speaker / Headset)"]
Mixer --> Speaker
end
end
subgraph CloudAI ["Google AI Multimodal Infrastructure"]
GeminiLive["⚡ Gemini 3.5 Live API\nBi-directional WebSocket Endpoint"]
SpeechSynthesis["Streaming Speech-to-Speech\n24kHz Audio Buffer Engine"]
GeminiLive <--> SpeechSynthesis
end
Video --> Capture
PCM_Enc -- "16kHz Mono PCM Frames (WebSocket)" --> GeminiLive
SpeechSynthesis -- "24kHz Synthesized PCM Stream" --> Mixer
style Browser fill:#08090A,stroke:#38BDF8,stroke-width:1px,color:#EDEDED
style Offscreen fill:#0E131F,stroke:#0284C7,stroke-width:1px,color:#EDEDED
style ExtensionEngine fill:#14171D,stroke:#38BDF8,stroke-width:1px,color:#EDEDED
style CloudAI fill:#0B1120,stroke:#60A5FA,stroke-width:2px,color:#EDEDEDArchitectural Benchmark: oaDub Live Extension vs. Traditional Dubbing Tools
| Operational & Technical Metric | Post-Processed Cloud Dubbing | Standard Subtitle Extensions | oaDub Live Gemini Engine |
|---|---|---|---|
| Translation Latency | 5 – 30 Minutes (Batch Render) | 1.5s – 3.0s (Text STT & Translate) | < 400ms (Direct Live Audio Stream) |
| Audio Output | Pre-rendered Replacement Video | None (Text Only / Visual Distraction) | Simultaneous 24kHz Natural Voice |
| Server Infrastructure | Heavy GPU Cloud Render Farms | Translation API Proxy Servers | Zero-Server / Direct Client WebSocket |
| Privacy & Security | Video Audio Uploaded to 3rd Party | Token Logged on Remote Servers | 100% Client Offline Key (Zero Data Retention) |
| Browser Integration | External Web Platform / Manual Upload | Standard Content Script | Manifest V3 Offscreen Document & Tab Capture |
Technical Deep-Dive
1. Manifest V3 Offscreen Document & Web Audio Tab Capture Pipeline
Modern Chrome extensions operate under Manifest V3 restrictions where background service workers lack DOM access and HTMLMediaElement lifecycle support. oaDub resolves this through an isolated Offscreen Document architecture. When enabled on any active video tab, chrome.tabCapture routes the raw media stream into a high-precision AudioContext, downsampling the multi-channel browser stream to a standardized 16,000 Hz 16-bit mono Little-Endian linear PCM stream ready for neural ingestion.
2. Low-Latency Bi-directional WebSocket & Gemini 3.5 Live Protocol
Rather than relying on disjointed Speech-to-Text (STT) followed by Text Translation (MT) and Text-to-Speech (TTS), oaDub connects directly to the Google Gemini Live Multimodal WebSocket interface. Chunked binary PCM audio frames are streamed continuously, preserving prosody and intent. Gemini Live responds with a streaming sequence of 24,000 Hz raw PCM audio chunks, achieving end-to-end voice-to-voice simultaneous translation latency below 400 milliseconds.
3. Dual-Channel Dynamic Audio Mixing & Intelligent Volume Ducking
To deliver an authentic simultaneous interpretation experience, oaDub implements an intelligent audio ducking algorithm via Web Audio API GainNodes. The original tab audio is maintained in the background at an attenuated level (-18dB) so the listener perceives natural ambient audio, while the synthesized translated voice is overlaid with pristine clarity. Dynamic circular jitter buffers smooth out variable network arrival times without introducing cumulative delay.
4. Zero-Server Privacy Architecture & Microsoft Edge Store Distribution
User privacy is guaranteed by design: oaDub features zero intermediary backend servers, zero telemetry proxies, and zero third-party databases. The user's Google AI Studio API key is encrypted and stored exclusively inside chrome.storage.local. All WebSocket handshakes occur strictly between the user's browser client and Google's official Gemini endpoint. Officially certified and published on Microsoft Edge Add-ons with automated CI/CD release pipelines.
Technical Stack
- Runtime & Platform
- Chrome / Microsoft Edge Extension (Manifest V3)
- AI & Protocol
- Google Gemini 3.5 Live API / Multimodal Bi-directional WebSocket
- Audio Ingestion
- Web Audio API, AudioContext, Offscreen Document, 16kHz PCM Downsampling
- Output Synthesis
- 24kHz Low-Latency PCM Audio Stream & Dynamic Buffer Queues
- Security & Privacy
- chrome.storage.local Client-Only API Key Encryption
- Distribution
- Official Microsoft Edge Add-ons Store Certified
Business Impact & Outcomes
By eliminating intermediate proxy servers and providing direct tab audio streaming, oaDub enables barrier-free real-time global media consumption for technical professionals, students, and researchers with zero operational infrastructure cost and complete data sovereignty.
Interested in a similar system architecture?
Reach out directly to discuss technical consulting, bespoke implementation, or system audits.