Voice Memory for Agentic
Speech Recognition

Chao-Han Huck Yang* · Zih-Ching Chen · Piotr Żelasko · Zhehuai Chen · Jagadeesh Balam · Boris Ginsburg

NVIDIA · *Work done at NVIDIA

Voice Memory pipeline: a speaker's audio reaches a frozen voice agent that reads an external text memory and decides per utterance to act or abstain, while an on-device optimizer improves the memory offline. — At inference, a frozen corrector reads a learnable, human-readable text memory and decides per utterance whether to *act* on the ASR hypothesis or *abstain* and keep the 1-best. Offline, an optimizer distills that memory from scored rollouts using forward passes only (i.e., no weight updates) so the learned skill generalizes within a domain and transfers across domains and corrector families.

🎬 The listener–thinker walkthrough

Noisy speech comes in. A frozen decoder emits its 5-best guesses, a frozen corrector reads a per-domain memory.md, and it makes one decision per utterance: act on the hypothesis, or abstain and keep the 1-best. Every transcript, hypothesis list, and memory rule below is real. That was taken verbatim from the HyPoradise test sets and from memories the optimization loop actually learned.

3 real cases · noisy far-field, voice command, financial news

Step 1: Speech arrives

Step 2: The listener decodes frozen Whisper emits its 5-best hypotheses

Step 3: The corrector reads its memory

memory.md

Step 4: One decision per utterance

✓ SELECTED

act

Edit the hypothesis using the memory's rules.

✓ SELECTED

abstain

Hold still — keep the 1-best untouched.

Step 5: Final transcript

named entity, kept verbatim fixed from another hypothesis generative — in no hypothesis

What this buys, at benchmark scale

0→0%

weighted WER, ten HyPoradise domains,
zero weight updates

0→0%

WER on noisy far-field speech
(CHiME-4)

0→0%

harmful-edit rate on financial news
(over-correction, lower is better)

ρ = 0

a self-formed memory recovers more
than the n-best oracle on ATIS

0 bytes

the deployed WSJ memory —
0 parameters added at inference

Voice Memory · restraint is the operative skill · the artifact is a readable file

Method — the listener–thinker architecture

An agentic speech system in the listener–thinker format is a tuple (B_φ, M_θ, π, s, O). Both models stay frozen; the only thing that ever changes is the text state s — a per-domain memory.md.

Listener · synchronous · stream time

👂 Decode, read, decide

A frozen decoder B_φ maps audio to 5-best evidence H. A frozen corrector M_θ reads H and the memory s, then samples one action per utterance.

audio → B_φ → H = (h₁ … h₅) → M_θ(H, s) → π(act | abstain) → transcript

Thinker · asynchronous · between runs

🧠 Rollout, edit, gate

An optimizer O turns scored rollouts, which failures F and successes G into bounded edits of the memory. An edit is kept only if it strictly improves a held-out score v.

rollouts → (F, G) → s′ = O(s, F, G) → accept iff v↑ → memory.md

The two roles are coupled only through the memory — no gradients, no weight updates, and the learned skill stays auditable, portable, and reversible.

The per-utterance decision

act

Edit

Recoverable headroom exists: reconcile the hypotheses, apply the memory's substitutions and formatting rules — sometimes recovering tokens present in no hypothesis (ρ > 1).

abstain

Hold still

The 1-best is already right, or the edit would only trade one surface form for another. Restraint is the skill the loop mostly learns that unconstrained GER breaks correct tokens on up to 64% of its edits.

A learned memory is small and readable

The ATIS memory below was formed by the validation-gated loop with Claude as corrector and optimizer — it reaches ρ = 1.28, above the n-best oracle. Excerpt, verbatim:

memories/memory_atis_claude.md

# ATIS ASR correction memory (restraint-first)

- RESTRAINT first: keep majority verbatim ASR tokens; edit only a clearly-wrong token.
  Over-correction raises WER.
- Reconcile the 5 hyps by per-position majority vote; the truth is usually the most
  common token.
- Carrier/airport initialisms are spaced letters: "twa" -> "t w a",
  "usair"/"us air" -> "u s air".
- Split run-on tokens: "showflights" -> "show flights", "sevenpm" -> "seven pm".
- Do not invent function words ("in"/"and"/"the"/"a") or place spellings not seen
  in the hyps.

Results

8.36 → 7.52%

weighted WER · 10 domains · 16,108 test utterances

0 / 10

datasets regress below their 1-best baseline

parameters added to the inference path

↓ WER — lower is better · ↑ ρ — higher is better (fraction of the 1-best-to-oracle gap recovered)

Breadth — full HyPoradise test, open corrector (Whisper-v2-Large + Qwen3-30B-A3B)

Gains concentrate where recoverable headroom is largest; nothing regresses.

Dataset	Raw ASR WER ↓	Voice Memory ↓	+ Few-shot ↓
ATIS	8.40	3.40	3.00
CHiME-4 (noisy)	12.69	10.46	10.28
CORAAL	25.83	25.65	25.44
CommonVoice	15.44	13.16	13.16
LRS2	11.85	10.69	10.78
LS-Clean	1.79	1.63	1.64
LS-Other	3.67	3.39	3.41
SWBD	15.86	15.47	15.33
TED-3	4.44	4.20	4.05
WSJ	6.03	5.38	5.36
WEIGHTED	8.36	7.52	7.47

Noise robustness — Robust HyPoradise (no fine-tuning, no latent noise vector)

Where static GER over-corrects on genuinely degraded speech, the memory recovers up to 47% of the 1-best-to-oracle gap — and correctly abstains where nothing is recoverable.

Condition	1-best ↓	Oracle ↓	Static GER ↓	Voice Memory ↓	ρ (recovered) ↑
CHiME-4 · real recorded / simulated far-field noise
test-real	9.8	7.8	9.9	9.5	0.17
test-simu	12.7	10.3	12.5	11.7	0.39
dev-real	7.8	6.3	8.1	7.3	0.31
dev-simu	9.7	7.9	9.8	8.8	0.47
NOIZEUS · pooled by SNR
0 dB	42.9	34.9	41.5	41.4	0.19
5 dB	14.5	9.2	14.8	12.6	0.36
10 dB	5.0	2.1	5.2	4.9	0.05
15 dB	2.7	0.9	2.7	2.8	−0.06

Portability — the memory is the asset, not the model

A second frozen corrector (Claude) reading memories at inference, 100-utterance tests. A MiniMax-written memory transfers; a self-formed one beats the n-best oracle on ATIS.

Domain	1-best ↓	Oracle ↓	No memory ↓	+ MiniMax memory ↓	+ Self-formed memory ↓
ATIS	8.2	4.9	6.7	6.1	3.9 (ρ = 1.28)
WSJ	5.3	4.1	6.1	4.6	6.2
LS-Other	4.8	3.3	4.3	4.5	4.8
LS-Clean	1.9	0.7	1.7	1.8	1.8

Case gallery

Eight representative cases spanning the taxonomy: Voice Memory fixes genuine errors, suppresses the harmful edits static GER introduces, and in the generative cases recovers a transcript present in no hypothesis. Tokens that disagree with the reference are marked.

BibTeX

@article{yang2026voicememory,
  title   = {Voice Memory for Agentic Speech Recognition},
  author  = {Yang, Chao-Han Huck and Chen, Zih-Ching and {\.Z}elasko, Piotr and
             Chen, Zhehuai and Balam, Jagadeesh and Ginsburg, Boris},
  journal = {arXiv preprint arXiv:2607.26410},
  year    = {2026},
  note    = {Model and learned memories: \url{https://huggingface.co/huckiyang/voice-memory}}
}