Independent product project · Android debug build validated 24 July 2026
An Android overlay that captures permitted playback audio, streams ordered PCM frames to an authenticated local worker, and turns finalized German speech into translated captions and reusable vocabulary cards.
Language learners watching German media need captions and vocabulary without routing microphone audio through a cloud service. Playback capture, speech segmentation, inference backpressure, ordered delivery, overlay permissions, and local-network trust all have to work together.
The Android 10+ client captures 16 kHz mono playback audio, sends framed PCM over an authenticated WebSocket, and renders finalized caption and vocabulary events in an overlay that also supports Samsung DeX. A Python worker uses WebRTC voice activity detection, an ordered utterance queue, Faster Whisper, translation, and deterministic vocabulary selection.
A debug APK was assembled, five Android tests passed, and all six isolated Python worker tests passed. The implementation preserves utterance order under inference pressure, rejects invalid frames and non-speech, deduplicates vocabulary, exposes explicit stop controls, and never requests microphone permission.
The project connects Android media APIs, Kotlin service lifecycle, a binary streaming protocol, Python asynchronous processing, speech inference, backpressure, local-network security, systemd operation, and user-facing recovery controls in one end-to-end product.
Permitted Android playback capture produces fixed 20 ms PCM frames. An authenticated WebSocket carries a versioned hello message and binary audio to a local worker. Voice activity detection finalizes utterances, the worker transcribes and translates them in order, and the client binds captions and vocabulary to the same utterance identifier.
The implemented local-first audio and caption path, including the consent, backpressure, and operator-controlled worker boundaries.
Read the diagram notesThe Android 10+ client captures 16 kHz mono playback audio, sends framed PCM over an authenticated WebSocket, and renders finalized caption and vocabulary events in an overlay that also supports Samsung DeX. A Python worker uses WebRTC voice activity detection, an ordered utterance queue, Faster Whisper, translation, and deterministic vocabulary selection.
Capturing application playback keeps room noise and microphone permission out of the path. The trade-off is that Android and the source application must explicitly permit capture.
Voice activity detection and ordered processing preserve sentence boundaries and prevent busy inference from silently dropping audio. Backpressure can increase latency when the worker is undersized.
A paired worker keeps audio on an operator-controlled network and makes model choice explicit. The operator must secure the listener, token, firewall, and TLS boundary.
The worker requires a generated pairing token and compares it using a constant-time check before accepting streaming audio.
The Android app retains up to 500 deduplicated vocabulary items and exposes stop controls from both the application and foreground-service notification.
Installation creates a dedicated environment and systemd user service; WSS certificate paths and network binding remain explicit operator configuration.
Android unit tests: 5 passed
Debug APK: assembled
Worker tests: 6 passed
Microphone permission: not requested
Streaming protocol: authenticated, ordered utterancesChecks shown here are project-specific; no portfolio-wide claim is applied automatically.
Links are limited to public-safe source material and sanitized evidence. Credentials, state, plans, identifiers, private topology, and backup contents are intentionally excluded.
Playback capture, streaming, transcription, overlay, and trust-boundary notes.
Publish a sanitized device-to-caption latency benchmark and validate the WSS path on a second Android device.