Skip to main content

Voice Translation

Voice Translation lets you understand in real time what others say in games or voice calls in VR, just like reading subtitles, and it can also translate your own voice in real time for others, reading the translation aloud with the built-in Speech Synthesis (TTS). Speech recognition and local translation both run on your own machine—no audio needs to be uploaded, keeping your privacy secure.

Feature Overview​

Voice Translation captures and recognizes two audio streams simultaneously, translating each independently:

  • Speaker Channel (loopback capture): Captures the voices of others in games, voice calls, or meetings, then recognizes and translates them into your native language, displayed as an overlay in front of you, achieving a "real-time subtitle" effect.
  • Microphone Channel: Captures your own speech, then recognizes it and translates it into the target language (such as English), making it easy to communicate with foreign players in multilingual social settings (such as VRChat).

The recognition language, translation language, and display style of each channel can be configured independently, without interfering with each other. The microphone channel also supports speech synthesis playback, playing the translation directly for others to hear.

How It Works​

Voice Translation is accomplished cooperatively by the main program and an independent voice worker process:

  1. The main program communicates with the subprocess through a named pipe, capturing microphone and speaker (system loopback) audio.
  2. The subprocess uses Silero VAD for real-time voice activity detection, automatically segmenting the start and end of each utterance.
  3. The detected speech segments are handed to the Qwen3-ASR model (inference based on llama.cpp) to be transcribed into text; you can also switch to an online recognition service.
  4. The transcribed results are then handed to the translation engine for translation (supporting local translation models, or a custom translation API you configure), and are displayed in real time in the main program's overlay.
  5. If Speech Synthesis (TTS) is enabled, the translation of the microphone channel is synthesized into speech by the Qwen3-TTS model and played back.

Local recognition and local translation are performed entirely on your own machine; audio never leaves your computer.

Audio Sources​

  • Microphone: Captures your own voice. The device list shows the available input devices.
  • Speaker (loopback capture): Captures the sound currently being played by the system, i.e., what others say in the game or voice call.
    • The capture source can be set to "System" (full-system loopback) or a specific application. If some headset / sound card devices cannot loop back system audio at the full-system level, you can specify a particular program under "Select Application" to capture only that program's audio. Saved programs are retained in the options even when they are not currently running, and take effect automatically once they run again.

In the "Audio Device Settings" card on the Voice Translation Settings page, you can select a specific device for the microphone and speaker separately. The device list is enumerated automatically by the subprocess and refreshed periodically, so there is no need to refresh it manually.

Automatic Language Recognition​

The speech recognition language is set to "Auto" by default, in which case the ASR determines the language being spoken on its own, with no manual switching needed. You can also specify a fixed recognition language for the speaker and microphone separately in settings, to improve recognition accuracy for specific languages.

The translation source language for Voice Translation is fixed to "Auto", meaning the translation engine determines the language automatically from the recognized text. Settings no longer provide a source language selection entry; you only need to specify the translation target language for each channel.

Real-Time Volume Monitoring and Thresholds​

In the "Audio Device Settings" card, there is a real-time volume bar below each of the microphone and speaker device entries:

  • The volume bar shows the currently captured sound level in real time; when it crosses the threshold it fills green, indicating that it is judged to be valid speech.
  • The red threshold line on the volume bar can be dragged directly (or clicked to position it), and the "↺" button on the right restores the threshold to its default value.
  • If the volume bar does not react, please confirm that voice translation is enabled—real-time volume can only be monitored after voice translation is turned on.

Setting a reasonable threshold effectively prevents ambient noise from being misjudged as speech, reducing meaningless recognition and translation.

Streaming Output​

After enabling "Simulated Streaming Output", recognition results are displayed in increments of roughly once per second (intermediate results appear first and are replaced once the final result stabilizes), making the display feel more like real-time subtitles; when disabled, the full result is displayed at once after an utterance has been completely recognized. You can switch this at any time in the "ASR Settings" card on the Translation Engine page (a confirmation prompt appears first when enabling it).

Pause and Resume​

In voice translation mode, a single short press of the trigger will pause / resume speech recognition. When paused, the model remains in memory, so resuming requires no reload and is very fast. Flicking the left or right joystick clears the currently displayed voice translation result.

In addition, in voice translation mode, pressing the controller's A / X button while near your ear quickly toggles the microphone channel on or off.

For detailed operational pacing in the mode, please get familiar with it through the software's built-in in-VR step-by-step tutorial.

Translation Engine Selection​

The translation engine for voice translation has two modes to choose from:

  • Local: Uses the bundled Hy-MT2 local translation model for offline translation, requiring no internet connection and no API key, running fully locally. The local model automatically unloads after 30 seconds of idleness to free memory, and reloads on demand when needed next.
  • Custom: Uses your own API key to access third-party translation services (general machine translation or LLM large models), offering a richer choice of languages and quality.

Note: Voice Translation currently does not support the "Built-in Translation Service" mode; only local or custom modes are supported. Please configure it in the "Voice Translation Engine Settings" card on the Translation Engine page.

Engine Modes for Speech Recognition and Speech Synthesis​

In addition to the translation engine, speech recognition (ASR) and speech synthesis (TTS) each also support two engine modes, both configured on the Translation Engine page:

  • Local (default): Uses the bundled local models for offline inference, generating no network traffic.
  • Custom: Connects to an online service (currently Alibaba Bailian DashScope is supported). You can choose a "real-time" or "non-real-time" model preset, or manually enter the model name and API URL; if no API key is entered, the interface will show a prompt.

After choosing online recognition / online synthesis, there is no need to download local models, which significantly reduces disk and memory usage, but an internet connection is required and the corresponding service's usage quota is consumed.

Speech Synthesis (TTS)​

After enabling "Enable Speech Synthesis (TTS)", the translation of the microphone channel is synthesized into speech and read aloud, so the other party can directly hear the translation result. The related settings are located in the "Speech Synthesis (TTS)" card on the Voice Translation Settings page and the "TTS Settings" card on the Translation Engine page:

  • Voice: Default (random) plus built-in voices such as Female A / Female B / Male A / Male B.
  • Voice Cloning: Click the "Microphone Clone" or "Speaker Clone" button to capture a human voice; after accumulating about 10 seconds it is automatically saved as a "Custom (Cloned)" voice. Please use clean, continuous single-speaker voice.
  • TTS Real-Time Streaming Playback: When enabled, playback starts while inference is still running, for lower latency; when disabled, playback happens all at once after synthesis completes.
  • Playback Output Device / Playback Volume: Specifies the output device and volume for the spoken audio.
  • Synchronized Speaker Playback: When enabled, TTS audio is also sent to the speaker device in the audio settings (during synchronized playback, recognition on the speaker channel is automatically suppressed, preventing your own translation from being recognized back).
  • Microphone Pass-Through: When TTS is not enabled (or voice translation is off / paused), forwards the microphone capture directly to the TTS playback device, so the in-game microphone does not need to switch devices frequently.

Virtual Audio Cable: If you want to send the spoken audio into the in-game microphone, you need to install the third-party virtual audio driver VB-Audio Virtual Cable. Key points for configuration: select CABLE Input as the playback output device, and select CABLE Output as the in-game microphone device.

Tip: TTS currently has a noticeable increase in VRAM usage when using GPU inference (it may use about 6.5 GB of VRAM); a confirmation prompt appears before switching to GPU, so please choose carefully.

VRChat Chatbox Sending​

When enabled, voice translation results are automatically sent to VRChat's Chatbox via the OSC protocol, so your translations appear in the VRChat interface in real time. You can control separately:

  • Whether to send the translation of the speaker (what others say)
  • Whether to send the translation of the microphone (what you say)
  • The OSC port (VRChat's default receiving port is 9000)
  • Message mode: Comparison mode (translation on top, original below) or translation-only mode
  • Message flow direction: new messages appear upward or downward
  • Number of retained messages (1 to 10)

Subject to VRChat Chatbox limitations, a single message supports at most about 135 characters and up to 9 lines; overly long content wraps automatically. There is also a minimum interval between two sends to ensure the chat box displays stably.

Inference Device and Performance​

For speech recognition, local translation, and speech synthesis, you can select the inference device on the Translation Engine page:

  • CPU (default): Best compatibility; in this case you can adjust the number of inference cores (range 1 to the device core count, default 4). A higher core count is faster, but uses more computing resources.
  • GPU: Choose a graphics card enumerated by the subprocess via Vulkan (integrated or discrete, showing the device's actual name). After selecting GPU, the inference core count no longer participates in computation, and the corresponding slider is hidden automatically.

Memory Requirements​

  • Speech recognition model: Selecting "Fast (0.6B)" requires about 2.5 GB of available memory; selecting "High-Precision (1.7B)" requires about 3 GB of available memory. If the system's available memory is insufficient, a prompt will be shown and loading will be skipped.
  • Local translation model (Hy-MT2) and local speech synthesis model (Qwen3-TTS): When enabled, the models are additionally loaded as needed, so please reserve enough available memory.

Usage Steps​

  1. Make sure the software is installed and launched, and the VR device is running.
  2. Enable "Automatically enable voice translation at startup" in Preferences (or manually enable it after launching while in voice translation mode).
  3. On the Translation Engine page, in the "Voice Translation Engine Settings", "ASR Settings", and "TTS Settings", select a local or custom engine and complete the configuration.
  4. On the Voice Translation settings page, confirm that the microphone and speaker devices, capture source, volume threshold, recognition language, and translation target language are all set correctly.
  5. In VR (in voice translation mode), long press the trigger for about 1 second to start voice translation, and you can see the translation of others or yourself in real time; short press the trigger to pause / resume recognition.

For more detailed setting descriptions, please refer to Voice Translation Settings.