The world of voice AI is experiencing a fundamental shift. While traditional voice agents rely on a pipeline of separate Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS) components, a new generation of voice-to-voice models promises to revolutionize how AI systems understand and respond to human speech.
Historical context: This article was published in May 2025. Model launches, forecasts, latency figures, and cost ranges below describe that period unless a later check is stated. As of 22 July 2026, GPT-4o mini TTS is deprecated and current realtime model IDs/prices should be verified in provider documentation.
These unified models can process audio input directly and generate audio output without the intermediate text conversion steps, potentially offering lower latency, more natural conversations, and better preservation of vocal nuances like emotion and tone.
What Are Voice-to-Voice Models?
Voice-to-voice models, also known as speech-to-speech models, are AI systems that can:
- Process audio input directly rather than first converting speech to text
- Generate audio output natively without requiring separate text-to-speech synthesis
- Preserve paralinguistic information such as speaker identity, emotion, and intonation
- Handle non-verbal sounds like laughter, sighs, or background noise contextually
Unlike traditional pipeline approaches that follow the STT → LLM → TTS sequence, these models work end-to-end in the audio domain.
Implementations Discussed in May 2025
OpenAI's Realtime API
OpenAI has been at the forefront with their Realtime API, powered by GPT-4o voice models. In March 2025, they announced three new proprietary voice models: gpt-4o-transcribe, gpt-4o-mini-transcribe and gpt-4o-mini-tts, along with significant improvements to their speech-to-speech capabilities.
Key Features:
- Ultra-low latency: The Realtime API enables natural conversation flow with minimal delay
- Steerable voices: OpenAI's new text-to-speech model delivers more nuanced and realistic-sounding speech and is more "steerable" than previous-generation models, allowing developers to control not just what is spoken, but how things are spoken
- Interruption handling: Natural turn-taking where users can interrupt the AI mid-speech
- Multimodal capabilities: Can handle both text and audio input seamlessly
Google's Gemini Live and Multimodal Live API
Google has introduced significant voice capabilities with Gemini 2.5, including native audio output for more natural conversational experiences and the Multimodal Live API that enables real-time, low-latency interactions with sub-second response times.
Key Features:
- Bidirectional streaming: Allows for concurrent sending and receiving of text, audio and video data with sub-second latency, outputting the first token in 600 milliseconds
- Advanced voice features: Affective Dialogue where the model detects emotion in the user's voice and responds appropriately, and Proactive Audio where the model will ignore background conversations and know when to respond
- Multiple speaker support: First-of-its-kind support for multiple speakers, enabling text-to-speech with two voices via native audio out
- Multilingual capabilities: Works in over 24 languages with seamless switching
Related Google research: AudioPaLM combined text- and speech-language-model capabilities for speech recognition and translation research. It was a Google project, not a Meta model.
Meta's Research Efforts
The Meta items discussed in the May 2025 article were research and forward-looking announcements, not a current production catalog:
- Voicebox: Meta's Voicebox can perform text-to-speech synthesis in six languages, as well as edit and remove noise from speech recordings, trained on over 50k hours of audio data
- Future Llama Models: In 2025, Meta described plans for improved native voice and interruption capabilities. Treat that as a historical roadmap statement and verify today's released APIs before implementation.
Technical Architecture
Voice-to-voice models typically employ one of two architectural approaches:
1. Unified Architecture
These models process audio tokens directly alongside text tokens in a single neural network:
- Audio is converted to discrete acoustic tokens
- The model learns to map between audio and text representations
- Output can be either text tokens or audio tokens
- Audio tokens are converted back to waveforms using specialized decoders
2. Hybrid Architecture
Some models combine the benefits of both approaches:
- Use native audio understanding for the current user utterance
- Maintain conversation history as text to reduce token count
- Replacing older audio turns with an accurate text representation can reduce context usage, but the ratio and quality trade-off are model-, language-, and workload-specific
Benefits of Voice-to-Voice Models
1. Lower Latency
They can potentially reduce orchestration latency by replacing separate STT and TTS calls. The May 2025 article cited early sub-600 ms examples, but that was not a general or current guarantee; compare end-of-speech to first audible output on the active endpoint.
2. Preserved Paralinguistic Information
Improved ability to understand the nuances of human conversation and more natural voice output. These models can:
- Maintain speaker identity across translations
- Preserve emotional tone and intonation
- Handle non-verbal audio cues contextually
3. Better Conversational Flow
Voice-to-voice models enable more natural interruptions and turn-taking, making conversations feel more human-like.
4. Reduced System Complexity
Eliminating the need for separate STT and TTS components simplifies the overall architecture and reduces potential points of failure.
Limitations and Challenges in the May 2025 Snapshot
1. Higher Token Consumption
The 2025 article observed that audio histories could consume more tokens than text histories and become expensive or slow over long conversations. Current tokenization and context handling are model-specific, so measure the active endpoint.
2. Production Readiness Was Workload-Dependent
The original article characterized speech-to-speech APIs as immature for many production workloads. That is a historical assessment, not a current market-wide verdict. Validate the exact endpoint's reliability, observability, controls, regional support, lifecycle, and task quality against your requirements.
3. Reduced Control and Interpretability
Lack of interpretability: you can't easily inspect what the model "heard" or plans to say, since it's not outputting text. That makes debugging and fine-tuning behavior more difficult.
4. Training Data Challenges
Training such models requires paired voice conversation data – which is rarer than text chat data. There are not many large-scale datasets of human-to-human spoken conversations with rich emotional content.
5. Cost Considerations
Voice-to-voice models can be expensive to run due to their complexity and higher computational requirements.
Performance Comparison: Voice-to-Voice vs. Pipeline
Based on implementations available in May 2025, the approaches were commonly compared as follows:
| Metric | Traditional Pipeline | Voice-to-Voice |
|---|---|---|
| Latency | 800-1000ms typical | 300-600ms possible |
| Audio Quality | Very good with modern TTS | Excellent, more natural |
| Interruption Handling | Good with proper engineering | Excellent, more natural |
| Cost per Minute | $0.02-0.10 historical estimate | $0.20+ historical estimate |
| Debugging | Easy to inspect each step | Limited visibility |
| Customization | High flexibility | Limited currently |
| Production Readiness | Mature and stable | Emerging, some limitations |
Cost Analysis
In the May 2025 snapshot, speech-to-speech APIs were relatively expensive. Current services use token- and duration-sensitive billing, so the old per-minute ranges below are historical estimates rather than current quotes.
The article's historical comparison used:
- Traditional pipeline: $0.02-0.10 per minute
- Voice-to-voice models: $0.20+ per minute
Do not use those ranges as current quotes. Recalculate an active pipeline from its STT, LLM, and TTS billing units, and calculate a realtime model from its current text/audio token or session pricing.
The higher cost is primarily due to:
- Larger context sizes for audio processing
- More complex model architectures
- Higher computational requirements
- Less optimization compared to mature text-based systems
The Road Ahead
What to Re-evaluate After the 2025 Snapshot
The 2025 forecast is now historical. Before choosing an architecture, verify the active model IDs, lifecycle dates, language coverage, interruption behavior, observability, moderation controls, regional availability, and text/audio billing terms. Measure those properties on the exact current endpoint instead of assuming the forecast occurred.
Long-term Vision
Voice-to-voice models represent a step toward more natural human-AI interaction. Future developments may include:
- Universal speech understanding: Models that can handle any language or dialect
- Emotional intelligence: Advanced recognition and generation of emotional states
- Real-time translation: Seamless cross-language communication with voice preservation
- Multimodal integration: Combining voice with visual and other sensory inputs
Choosing the Right Approach
Factors That Can Favor Voice-to-Voice Models
- Ultra-low latency requirements: When every millisecond matters
- Natural conversation experiences: For applications prioritizing human-like interaction
- Emotional AI applications: When preserving tone and emotion is crucial
- Creative and experimental projects: Where cutting-edge capabilities are valued
Factors That Can Favor Traditional Pipelines
- Production-critical applications: Where reliability and debugging are paramount
- Cost-sensitive deployments: When budget constraints are tight
- Custom requirements: When specific STT, LLM, or TTS customizations are needed
- Enterprise integrations: Where mature tooling and support are required
Conclusion
Voice-to-voice models can offer natural, low-latency interactions, while pipelines can offer clearer component boundaries and transcript-level control. Neither architecture is universally more production-ready; suitability depends on the exact current endpoint and workload.
In the article's 2025 context, the traditional pipeline approach remained the pragmatic choice for many voice AI applications, offering modularity and mature tooling. Current suitability depends on reliability, observability, latency, control, and cost requirements.
The future likely holds a coexistence of both approaches, with voice-to-voice models gradually capturing more use cases as they mature and costs decrease. For now, the choice depends on your specific requirements, budget, and tolerance for cutting-edge technology.
Use the Voice Agent Pricing Calculator to model a source-linked pipeline cost scenario. It does not measure model quality, reliability, or suitability; evaluate those separately on your workload.