BreezeRT: Serving Ultra-Low-Latency TTS

All tests in this article were run on H100. We compare PyTorch, vLLM-Omni, SGLang-Omni, and BreezeRT with the same Breeze TTS 2 model across 1–16 concurrent requests, focusing on first-packet latency.

We adapted and tuned PyTorch, vLLM-Omni, and SGLang-Omni for Breeze TTS 2. The chart shows the best results we have measured with each framework. Our adaptations have not yet been merged into their open-source repositories.

Breeze TTS 2 first encodes text and runs Backbone Prefill. For each speech frame, the Backbone predicts the first audio code, Depth Decoder generates the remaining 15 codes, and the Codec produces waveform samples. Each stage has different compute requirements, execution frequency, and suitable batch sizes.

Concurrent requests are often at different stages: A has audio codes ready to decode, B is generating its next frame, and C has just arrived and is waiting for first audio. Continuing with A would leave B and C waiting. The scheduler needs to switch between requests before a full sentence is finished.

BreezeRT schedules text processing, Backbone Prefill, Backbone Decode, Depth, and audio decoding separately. One runtime manages request progress, caches, and pending audio. After each stage, it chooses which stage and requests to run next.

The scheduler picks requests by priority, then batches work within the chosen stage. Each stage has its own batch limit, so the Codec and Backbone can use different batch sizes. New requests can enter between stages, and active streams get time to decode. The same runtime handles chunking, graph replay, and deadline scheduling.

A larger first chunk makes the user wait for more frames before hearing audio. But sending one frame at a time means running the Codec and scheduler more often. That overhead adds up as requests increase.

BreezeRT gradually increases chunk sizes, using 1, 2, 4, 8, and 16 frames, for example. Small chunks start playback quickly. Once some audio is buffered, larger chunks reduce the number of decoding calls.

The Codec also needs to remember earlier work so it does not decode the same audio again. BreezeRT stores sliding-window Transformer KV, causal-convolution history, and the transposed-convolution overlap tail for each request, then processes only new frames. These caches preserve context and smooth chunk boundaries as chunk sizes change.

In the supporting comparison, the first request received audio after about 2500 ms with a fixed 16-frame first chunk, and about 150 ms with progressive chunks.

With a smaller first chunk, CPU launch overhead becomes easier to see. Generating one speech frame takes many model operations. Even a fast GPU can sit idle between submissions. BreezeRT reduces compute overhead through compilation and uses CUDA Graphs to submit captured GPU operations in one replay.

CUDA Graph replay requires fixed tensor shapes, but text lengths, batch sizes, and audio chunk sizes vary. Capturing every combination would use too much memory. BreezeRT keeps a set of common shapes and splits or pads inputs to fit them.

The Text Encoder packs tokens and splits work by request count and token capacity to reduce padding. Attention remains separate for each request. Backbone Prefill works in token chunks, with paged KV storing each request’s generation history so it can reuse existing graphs and buffers.

Depth Decoder generates 15 more audio codes per frame. BreezeRT compiles the decoder stack and output heads as whole units, fusing compatible elementwise operations into fewer kernels. This cuts intermediate tensor reads and writes and kernel launches—repeated work that adds up across concurrent requests. KV, position, and input/output buffers are allocated in advance. The full autoregressive loop is captured in one graph, so one replay runs all 15 steps in dependency order.

Text encoding, Backbone, Depth, and audio decoding all use graphs, with common shapes warmed up at startup. Each request keeps its own sampling settings and random state. With chunking unchanged, the supporting comparison measured first-request latency of about 150 ms, falling to 100 ms with Depth compilation and 35 ms with graphs across all stages.

New requests are waiting to start speaking; active streams are waiting for their next chunk. Always serving new requests first can interrupt playback. Generating too far ahead for active streams makes new requests wait. Predictor uses playback deadlines to decide which work goes first.

New requests get startup priority. For active streams, the first-output time and total audio duration give an estimated playback deadline. Predictor records recent stage timings by batch size and allows a margin when estimating whether the remaining Backbone, Depth, and Codec work can finish in time.

For example, A is ready to decode, while B needs one more generation step to join its batch. The scheduler waits for B if batching saves time and still meets A’s playback deadline. Otherwise, A runs first. If the target chunk cannot be completed in time, the runtime can send the frames already generated rather than leave the player waiting.

Predictor uses a lightweight timing estimate. In a 300-request comparison, requests with potential playback gaps fell from 14 to 0. The maximum extra buffer needed to avoid a gap fell from about 330 ms to 0 ms.