Back to Blog
AIPublished on October 23, 2026

Engineering Full-Duplex Speech AI: Inside Continuous Acoustic Tokenization, Cross-Talk Resolution, and Sub-80ms Latency

Cascaded voice pipelines are fundamentally incapable of natural conversational flow. We deconstruct the architecture of true full-duplex speech foundation models, examining dual-stream causal attention, acoustic RVQ codecs, and barge-in arbitration.

The Latency Mirage: Why Cascaded Voice Pipelines Fail at Human Realism

For over a decade, conversational artificial intelligence has relied on a fragmented, three-stage pipeline: Automatic Speech Recognition (ASR), followed by a Large Language Model (LLM) reasoning core, followed by Text-to-Speech (TTS) synthesis. While this modular abstraction allows independent optimization of each subsystem, it creates an insurmountable physical barrier to realistic human interaction: compounding serialization latency.

In a cascaded pipeline, the system cannot reliably formulate a response until the ASR model has emitted finalized text tokens. Even with aggressive endpointing algorithms and partial-transcript streaming, the sequence of operations imposes a structural floor on glass-to-glass latency:

  1. Acoustic Endpointing & VAD (Voice Activity Detection): 200ms–400ms to verify user utterance completion without accidental truncation.
  2. Streaming ASR Transcription: 100ms–250ms to reach stable beam-search convergence.
  3. LLM Time-to-First-Token (TTFT): 150ms–350ms, constrained by KV-cache lookup and prefill phase compute.
  4. TTS Synthesis Chunking & Buffering: 150ms–300ms required for the vocoder to accumulate sufficient phonetic context for natural prosody.

Total end-to-end latency in production systems routinely exceeds 800ms to 1200ms. In human sociolinguistics, typical conversational turn-taking gaps average 200ms to 250ms. Latencies above 300ms break conversational synchrony; latencies above 600ms induce cognitive friction, prompting speakers to interrupt, repeat themselves, or abandon flow.

More critically, the cascaded architecture enforces a half-duplex operational model. The system either listens or speaks; it cannot simultaneously perceive, plan, and vocalize while remaining receptive to overlapping acoustic events, such as backchanneling ('mm-hmm', 'yeah') or sudden semantic interruptions.

True full-duplex speech models abandon textual serialization entirely. By operating directly on continuous acoustic latent spaces, these architectures achieve concurrent ingress and egress processing, collapsing latency down to sub-80ms thresholds.


Continuous Acoustic Tokenization: Neural Audio Codecs

To eliminate intermediate textual representations, an end-to-end full-duplex model must ingest raw PCM audio and emit raw PCM audio in near real-time. This requires a neural audio codec that compresses high-bandwidth waveforms (e.g., 24kHz 16-bit PCM = 384 kbps) into discrete, highly compact structural tokens without discarding pitch, timbre, emotion, and ambient context.

State-of-the-art implementations utilize Residual Vector Quantization (RVQ) operating over deep convolutional or transformer-based encoder-decoder frameworks (such as SoundStream, EnCodec, and Mimi).

Raw Waveform (24kHz) 
     │
     ▼
[ Strided 1D Convolutions ]  --> Temporal Downsampling (e.g., 320x)
     │
     ▼ Latent Vector Z ∈ ℝ^{D × T} (75Hz frame rate)
[ RVQ Codebook Layer 1 ]    --> Q_1(Z)  ──────────┐
     │ Residual: R_1 = Z - Q_1(Z)               │
     ▼                                         │
[ RVQ Codebook Layer 2 ]    --> Q_2(R_1) ─────────┐│ (Aggregated Tokens)
     │ Residual: R_2 = R_1 - Q_2(R_1)          ││
     ▼                                         ││
    ...                                        ▼▼
[ RVQ Codebook Layer N ]    --> Q_N(R_{N-1}) ─────► Hierarchical Quantized State

In an RVQ pipeline:

  1. The raw audio waveform $x$ is mapped to a continuous latent representation $z = f_E(x)$ at an aggressive temporal downsampling factor (e.g., 320x, yielding a 75 Hz token rate from a 24 kHz signal).
  2. The vector $z$ is quantized through a cascade of $N$ discrete codebooks. The first codebook quantizes $z$ into $q_1$. The residual $e_1 = z - q_1$ is then passed to the second codebook, yielding $q_2$, and so forth:

$$z \approx \hat{z} = \sum_{i=1}^{N} q_i(e_{i-1})$$

In early layers (Codebooks 1–2), the tokens capture high-level semantic data, phonetic alignment, and lexical content. In deeper layers (Codebooks 3–8+), the tokens capture acoustic residuals, including vocal tract geometry, reverberation, prosodic dynamics, and background acoustic environments.

For full-duplex modeling, this hierarchy is exploited: the language backbone can predict only the primary semantic codebook autoregressively, while a parallel non-autoregressive or shallow transformer head predicts the remaining residual layers conditioning the vocoder synthesis.


Dual-Stream Causal Transformer Architecture

To listen and speak at the exact same physical instant, the foundational language transformer cannot rely on standard unidirectional causal autoregression over a single sequence. Instead, it must maintain two concurrent causal streams within a shared temporal manifold.

At each discrete temporal step $t$ (e.g., every 13.3ms, corresponding to a 75Hz frame):

  • Input Stream ($I_t$): Encompasses the user's incoming audio tokens produced by the client encoder.
  • Output Stream ($O_t$): Encompasses the assistant's generated audio tokens fed to the client decoder.

The input stream and output stream are concatenated or projected into interleaved token tuples at frame $t$, allowing the attention mechanism to observe cross-stream dependencies within a strict causal horizon.

Time Step:       t-2         t-1          t          t+1
User Audio:   [ U_{t-2} ] [ U_{t-1} ] [ U_t     ] [ U_{t+1} ] ... (Ingress Stream)
                  │           │           │           │
               ┌──┴───────────┴───────────┴───────────┐
               │   Shared Transformer Backbone with  │
               │       Dual Causal Masking Logic     │
               ┌──┬───────────┬───────────┬───────────┘
                  │           │           │           │
Agent Audio:  [ A_{t-2} ] [ A_{t-1} ] [ A_t     ] [ A_{t+1} ] ... (Egress Stream)

The Dual Causal Mask

Standard causal masking ensures that token $k$ can only attend to tokens $j \le k$. In a dual-stream architecture, the masking matrix must enforce two distinct invariant rules:

  1. Temporal Causality: The agent’s output token at frame $t$ ($A_t$) can attend to all prior user tokens ${U_0, \dots, U_t}$ and all prior agent tokens ${A_0, \dots, A_{t-1}}$.
  2. Non-Leaking Ingress: The user input representation $U_t$ does not depend on future agent outputs, preventing hallucinations from contaminating incoming audio feature tracking.

When the user is speaking and the agent should remain silent, the model outputs a dedicated acoustic token sequence corresponding to zero-energy comfort noise or an explicit <SILENCE> token in its RVQ output codebook.


Interleaving and Inference Architecture in PyTorch

The following PyTorch module demonstrates the core structural mechanism of dual-stream token embedding and custom attention mask generation for a continuous full-duplex inference loop.

import torch
import torch.nn as nn
import torch.nn.functional as F

class FullDuplexAttentionMask(nn.Module):
    def __init__(self, max_seq_len: int):
        super().__init__()
        self.max_seq_len = max_seq_len

    def generate_mask(self, total_frames: int, device: torch.device) -> torch.Tensor:
        """
        Generates a 2D causal attention mask for interleaved dual streams.
        Stream layout per frame t: [User_Token_t, Agent_Token_t]
        Total sequence length = 2 * total_frames
        """
        seq_len = 2 * total_frames
        # Initialize with negative infinity for causal prevention
        mask = torch.full((seq_len, seq_len), float('-inf'), device=device)
        
        for i in range(seq_len):
            frame_i = i // 2
            type_i = i % 2  # 0 for User, 1 for Agent
            
            for j in range(seq_len):
                frame_j = j // 2
                type_j = j % 2
                
                # Standard historical causality: can always attend to past frames
                if frame_j < frame_i:
                    mask[i, j] = 0.0
                elif frame_j == frame_i:
                    # Within the same time-frame:
                    # User token can only attend to itself (not simultaneous agent token)
                    if type_i == 0 and type_j == 0:
                        mask[i, j] = 0.0
                    # Agent token can attend to itself AND simultaneous user token
                    elif type_i == 1:
                        mask[i, j] = 0.0
                        
        return mask

class DualStreamAudioBackbone(nn.Module):
    def __init__(self, vocab_size: int, embed_dim: int, num_heads: int, num_layers: int):
        super().__init__()
        self.embed_dim = embed_dim
        self.user_embedding = nn.Embedding(vocab_size, embed_dim)
        self.agent_embedding = nn.Embedding(vocab_size, embed_dim)
        self.type_embedding = nn.Embedding(2, embed_dim) # 0: User, 1: Agent
        
        decoder_layer = nn.TransformerDecoderLayer(
            d_model=embed_dim,
            nhead=num_heads,
            dim_feedforward=embed_dim * 4,
            batch_first=True
        )
        self.transformer = nn.TransformerDecoder(decoder_layer, num_layers=num_layers)
        self.mask_generator = FullDuplexAttentionMask(max_seq_len=2048)
        self.head = nn.Linear(embed_dim, vocab_size, bias=False)

    def forward(self, user_tokens: torch.Tensor, agent_tokens: torch.Tensor) -> torch.Tensor:
        """
        user_tokens:  (batch, frames)
        agent_tokens: (batch, frames)
        """
        b, t = user_tokens.shape
        
        u_emb = self.user_embedding(user_tokens) + self.type_embedding(torch.zeros_like(user_tokens))
        a_emb = self.agent_embedding(agent_tokens) + self.type_embedding(torch.ones_like(agent_tokens))
        
        # Interleave along sequence dimension: [U_0, A_0, U_1, A_1, ...]
        interleaved = torch.empty((b, 2 * t, self.embed_dim), device=user_tokens.device, dtype=u_emb.dtype)
        interleaved[:, 0::2, :] = u_emb
        interleaved[:, 1::2, :] = a_emb
        
        mask = self.mask_generator.generate_mask(t, device=user_tokens.device)
        
        # Forward pass through decoder backbone
        latent = self.transformer(tgt=interleaved, memory=interleaved, tgt_mask=mask, memory_mask=mask)
        logits = self.head(latent)
        
        # Return logits specifically for Agent token positions (odd indices)
        agent_logits = logits[:, 1::2, :]
        return agent_logits

Resolving the Overlap Dilemma: Barge-in vs. Backchanneling

The fundamental technical challenge of full-duplex speech is not concurrent audio decoding—it is behavioral arbitration. In natural human communication, conversational overlap manifests in two opposing behaviors:

  1. Competitive Overlaps (Barge-ins): The listener desires to seize the floor. The current speaker must yield, stop vocalization immediately, and process the new incoming semantic instruction.
  2. Non-Competitive Overlaps (Backchannels): The listener vocalizes brief phatic expressions ('yeah', 'right', 'uh-huh') or emotional cues (laughter) that signal active listening without requesting the floor. The current speaker must continue speaking without hesitation or disruption.

Traditional VAD-based systems execute a blunt rule: if user audio energy crosses an amplitude threshold while the agent is generating audio, playback is immediately killed (an aggressive hardware barge-in). This results in erratic, unnatural stops whenever the user simply clears their throat or murmurs agreement.

Semantic Interruption Arbitration

In an end-to-end full-duplex neural architecture, barge-in logic is governed by continuous semantic-acoustic scoring within the transformer's latent representations:

$$\mathcal{P}(\text{Yield} \mid U_{\le t}, A_{< t}) = \sigma\left(\mathbf{W}_y \cdot \mathbf{h}_t^{\text{agent}} + \mathbf{b}_y\right)$$

Where $\mathbf{h}_t^{\text{agent}}$ is the hidden state vector of the model at time $t$.

  • If the incoming user tokens $U_{t-k \dots t}$ correspond to non-competitive acoustic codes (such as short duration vocables, low pitch inflection, or transient energy), $\mathcal{P}(\text{Yield})$ stays near zero. The model continues driving its autoregressive output stream uninterrupted.
  • If the user tokens establish lexical density and prosodic patterns indicative of an assertive utterance (e.g., 'Hold on, wait a second...'), the output prediction layer transitions its egress distribution to emit silence tokens within 40ms to 80ms, gracefully executing an interruption shutdown.

Because the unified transformer backbone processes the incoming user tokens directly through its KV-cache throughout the entire interaction, the context of the user's interruption is already prefilled in memory. There is no cache invalidation, no context rebuild, and zero initialization delay required to formulate the appropriate recovery response.


Sub-80ms Latency Budget and Engineering Optimizations

To make full-duplex interaction viable over actual WAN network links, systems must maintain end-to-end processing well within an 80ms latency envelope. This requires ruthless optimization across every stage of the inference loop:

+-------------------------------------------------------------------------+
|                         Total Budget: 80ms                              |
+-------------------+--------------------+-------------------+------------+
| Client Mic Packet | Transport & Ingress| Neural Compute    | Jitter/DAC |
| 15ms - 20ms       | 20ms - 25ms        | 25ms - 30ms       | 10ms       |
+-------------------+--------------------+-------------------+------------+

1. Frame Chunking and Sliding Convolutional Windows

The audio encoder cannot wait for long audio sequences. Encoders must operate with tiny strided receptive fields: a typical configuration processes 20ms audio chunks (480 samples at 24kHz) with a lookahead buffer capped at strictly $\le 10\text{ms}$.

2. Rolling KV-Cache Slicing

Retaining every attention token over a 30-minute conversation is computationally non-viable at 75 tokens per second (135,000 tokens per stream). A dual-stream system must utilize hybrid architectures:

  • Sliding-Window Attention: Local attention spans (e.g., last 256 frames / ~3.4 seconds) handle high-fidelity acoustic matching, phonetic coherence, and prosodic continuity.
  • Compressed Global Context: Sparse memory banks, Cross-Attention Latent Recurrences, or linear state-space models (SSMs) maintain long-horizon conversational state without linearly expanding quadratic attention costs.

3. Asymmetric Codebook Generation (Speculative Decoding)

Predicting all 8 to 32 RVQ codebooks autoregressively per frame would saturate GPU tensor cores. State-of-the-art designs predict only Codebook 0 (the semantic backbone) inside the large causal language transformer.

The remaining residual layers ($Q_1$ to $Q_N$) are predicted using a tiny, non-autoregressive feed-forward or depthwise-separable convolutional network running in parallel over the synthesized latent frame. This eliminates 90% of the forward-pass overhead without degrading voice naturalness.


The Architecture of the Next Decade

Full-duplex speech models represent an architectural phase shift. By abandoning the modular paradigm of discrete ASR, text LLMs, and TTS vocoders in favor of unified, continuous-time acoustic transformers, real-time voice AI crosses the chasm from robotic command-and-response interfaces to natural conversational dynamics.

When models can perceive subtle pitch inflections, emit empathetic backchannels in real-time, and gracefully yield the floor the moment an interruption begins, interaction shifts from transactional prompt engineering to genuine socio-linguistic communion.

#AI#Speech Synthesis#Deep Learning#Audio Transformers#Natural Language Processing