Measuring the real-time speech stack, layer by layer.

Real-time speech systems are often described with one number: latency.

That number hides most of the system.

Between a person finishing a thought and hearing a response, audio must be captured, represented, transported, interpreted, reasoned over, converted back into speech and scheduled for playback. Some of those stages are computational. Some are network-bound. Some exist only because of the architecture we chose.

We built the stack layer by layer so that we could measure those boundaries before trying to optimize them.

The result was not a single record latency number. The more important finding was that several of our largest improvements came from removing computation and removing waits, rather than making existing stages incrementally faster.

That eventually changed the question we were asking.

We started with:

How do we make the speech pipeline faster?

We ended this phase asking:

What computation actually needs to run continuously for realtime intelligence?

Before speech recognition, language models or synthesis, we needed to know what the surrounding system itself was doing.

The first apparatus captured browser audio, preserved per-session evidence and measured events independently on the client and backend.

This immediately exposed a basic but important measurement problem:

The browser and backend operate in independent monotonic clock domains. Therefore a timestamp produced by one system cannot simply be subtracted from a timestamp produced by another and called one-way latency.

Instead, we measured intervals locally:

For one representative transport observation:

The important result was not that 98.0 ms was unusually fast. It was that the measurement told us where not to spend engineering time.

Only 18.7 ms of that observed application round trip was inside the instrumented server section. Eliminating the server work entirely would still leave most of the observed path untouched.

What this changed

The first lesson was methodological:

Before optimizing a stage, establish that the number attributed to that stage is actually measuring it.

This principle remained important throughout the rest of the experiments.

The next experiments introduced speech recognition.

Whole-utterance recognition gave us a useful semantic control because the recognizer had access to the complete spoken sequence before finalizing its interpretation.

Then we compared that behavior against streaming recognition using the same frozen audio.

A simple correction exposed the problem:

USER SPEECH

meeting at 4 PM
        ↓
       no
        ↓
actually 5 PM

At one point in the stream, this state is perfectly reasonable:

A few hundred milliseconds later, it is wrong:

Streaming recognition could appear faster by finalizing before the later correction arrived. That produced one of the most important findings of the first phase:

The paired experiment demonstrated that ordinary latency measurements can reward a system for committing too early.

This suggested that a realtime system should not represent intermediate interpretation using only:

FINAL / NOT FINAL

It needs a richer lifecycle:

PROVISIONAL → UPDATE → REVISE → SUPERSEDE → COMMIT
                                      │
                                      └────→ CANCEL

That is not merely a speech-recognition problem. Any continuous system acting before all future information is available has to reason about revision and commitment.

We deliberately built a conventional end-to-end control before proposing a new architecture.

The implementation was intentionally ordinary. Its purpose was measurement.

Without a measurable control system, an architectural improvement is difficult to distinguish from a different workload, provider state or measurement boundary.

Repeated frozen-input runs exposed significant end-to-end variance. Successful executions sometimes completed in roughly 4–7 s, while slow successful runs could extend toward 18–22 s; failures were also observed.

The strongest observed concentration of variance was inside the measured language-model stage, but that observation alone did not establish the root cause. So rather than redesigning the entire pipeline, we isolated that stage.

The isolated model experiment produced an unexpected trace. The model connection had started successfully, but visible response content did not immediately appear. Instrumentation then exposed a large amount of hidden reasoning activity preceding the simple conversational answer.

For the simple conversational test used in this experiment:

The integrated comparison later showed the measured language-model total falling from approximately:

while campaign median end-to-end time moved from approximately:

Only the model-stage change should be causally attributed to the reasoning intervention because other stage distributions also changed between the integrated campaigns.

The important conclusion was not:

reasoning is bad.

The supported conclusion was narrower:

Deep reasoning was unnecessary for this simple reflex conversational path, and performing it anyway imposed a large latency cost.

We can express the problem as:

Traditional optimization tends to focus on:

But in this experiment, the larger gain came from:

Engineering principle

The cheapest computation is computation that never needs to run.

Once the conversational model path was reduced, speech generation became a more visible barrier. Originally we were measuring the time required to generate an entire waveform.

But a realtime listener does not care when the final byte of a response exists. They care when they can begin hearing it.

So the metric changed from:

to:

or TTFA.

Five paired comparisons produced:

with median paired improvement of approximately:

All five streaming runs in that paired experiment succeeded. This did not prove continuous browser playback by itself. It proved that waiting for complete speech generation was an avoidable latency barrier worth removing.

The first integrated progressive-audio design contained another potential serial step:

GENERATED AUDIO
      ↓
COMPRESSED AUDIO
      ↓
GENERAL DECODER
      ↓
PCM
      ↓
BROWSER

Before optimizing that decoder path, we checked whether the intermediate representation was necessary. It was not. The resulting control instead used a direct uncompressed audio path conceptually equivalent to:

The significant architectural change was not a faster decoder. It was:

Again the same pattern appeared:

Before making a stage faster, ask whether the stage needs to exist.

After integrating progressive speech delivery and explicit browser playback scheduling, three frozen smoke sessions produced the following observations.

The browser playback layer contributed very little observed delay compared with the upstream stages during these smoke runs.

What the experiment supports:

Progressive audio delivery and explicit browser scheduling worked in the measured B3 control.

What it does not support:

full duplex proven                 NO
human naturalness proven          NO
production latency = 1.64 s       NO
gapless playback universally      NO
native realtime architecture      NO

B3 was intentionally frozen rather than endlessly optimized.

The progression looks simple when reduced to its important decisions.

Or as an optimization table:

The recurring result was:

when the removed work was unnecessary. This was the main engineering lesson of the cascade phase.

Because B3 was becoming a better version of the wrong abstraction to optimize indefinitely. The control still looks approximately like:

Speech output is progressive. The full system is not. Major semantic stages still wait for explicit boundaries.

Further optimization could reduce another few hundred milliseconds while leaving the deeper architecture intact. So B3 became the control.

The research question moved from:

How do we accelerate this cascade?

to:

Why should realtime intelligence be organized as a cascade at all?

Conversational systems process information with radically different temporal characteristics.

• A waveform may change tens of thousands of times per second.
• Interaction state changes at a much slower rate.
• Meaning changes around words or semantic events.
• Speaker identity may remain stable for an entire session.
• Deep reasoning may only be needed occasionally.

Conceptually:

There is no obvious requirement that the same expensive global computation execute at every one of these clocks. That produced the current multi-clock research hypothesis.

A monolithic continuously expensive system can be represented conceptually as:

Our current research hypothesis separates continuous state maintenance from occasional expensive cognition:

with the desired regime:

where:

• $N$ represents ongoing conversational updates
• $C_{\text{state}}$ represents relatively cheap state maintenance
• $K$ represents global-context invocations
• $J$ represents deeper reasoning invocations

The point is not the equation itself. The research question is whether $K/N$ can become small without destroying semantic correctness.

The conceptual direction is therefore different from simply running every stage faster.

This figure should intentionally remain conceptual. The public report does not need to reveal:

• exact internal state representation;
• routing thresholds;
• model architecture choices;
• confidence functions;
• training strategy;
• commit-policy implementation;
• benchmark generation internals.

The useful idea is enough:

Cheap state should be able to evolve continuously while expensive global computation is invoked according to informational need rather than acoustic clock rate.

The self-correction experiment also implies another architectural separation. A realtime system may begin reasoning before it has enough evidence to safely expose an irreversible result. Therefore:

A future realtime system should be allowed to prepare possibilities without treating every intermediate interpretation as permission to act. Conceptually:

OBSERVE
   │
   ▼
PROVISIONAL STATE
   │
   ├────────▶ THINK
   │             │
   │             ▼
   │         possible plan
   │
   ├────────▶ REVISE
   │
   ├────────▶ CANCEL
   │
   └────────▶ COMMIT ─────▶ SPEECH / ACTION

This is especially important once realtime interaction is connected to tools rather than only conversation.

The current research branch compares different policies for maintaining state across an incremental conversational stream. The public version does not need to reveal the complete benchmark corpus. A representative conceptual sequence is enough:

Rahul
   ↓
meeting
   ↓
tomorrow
   ↓
4 PM
   ↓
no
   ↓
actually 5 PM

The system must support more than token accumulation. It must preserve operations such as:

WAIT
UPDATE
REVISE
RECALL
COMMIT
CANCEL

The current experimental framing compares three general approaches:

The active R0 plan evaluates accumulation, correction, late constraints, cancellation, exact historical recall and cheap conversational events, while recording latency, correctness, memory behavior and expensive-compute frequency.

One particularly useful metric is:

where $GIR$ is the Global Invocation Ratio.

Lower is not automatically better. A system that never invokes expensive cognition but continually loses meaning is useless. So compute has to be measured jointly with quality:

The exact objective function and thresholds remain part of the experimental program rather than a public claim.

B3 is the frozen engineering control.

It gives us a measurable reference system with progressive speech delivery.

Native realtime is the active research direction.

The next question is not whether another API can save another hundred milliseconds. It is:

What computation needs to exist continuously?

If rapidly changing interaction state can be maintained cheaply while global cognition runs only when the information requires it, realtime intelligence may be able to improve latency, compute efficiency and state continuity together.

If that hypothesis fails, the experiments should show us that too. That is the purpose of the next phase.