Voice Memory  ·  notebook example 📄 Paper Project page 🤗 Model ⬇ Download .ipynb

🧠🎙️ Voice Memory — an easy, runnable example

Voice Memory is a training-free scheme for agentic speech recognition: a frozen LLM corrector reads a per-domain memory.md at inference time and makes one decision per utterance — act on the ASR hypothesis, or abstain and keep the 1-best. A separate optimizer improves that file from scored rollouts; no weights ever change.

This notebook walks the whole loop on three real cases from the HyPoradise benchmark (Whisper 5-best, verbatim), using real memory rules the optimization loop learned, and scores everything with the agwer package (WER, oracle, ρ, HER).

It runs anywhere — no API keys, no GPU: a 30-line rule engine stands in for the frozen LLM corrector so you can see the mechanics; the last section shows how to swap in a real LLM.

Links: ⬇ download this notebook · paper (arXiv:2607.26410) · project page · model & learned memories · agwer docs

1 · Setup

One dependency.

try:
    import agwer
except ImportError:
    %pip install -q agwer
    import agwer
print("agwer", agwer.__version__)

agwer 0.4.11

2 · Three real utterances, straight from the benchmark

Each case gives the frozen decoder's 5-best hypotheses and the reference. Note what is broken in each:

case	domain	what's wrong in the 1-best
chime4	noisy far-field speech	offers ≠ operates; the noise ate the last word (mideast)
atis	air-travel voice command	run ≠ round; the carrier's spelled form u s air is at risk
wsj	financial news	nothing! hypothesis 3 is already perfect — the trap is over-correcting

CASES = {
    "chime4": {
        "nbest": [
            "u s air offers primarily in the east and the east",
            "u s air operates primarily in the east and the east",
            "the u s air offers primarily in the east and the east",
            "the u s air operates primarily in the east and the east",
            "u s air offers primarily in the east and the bit east",
        ],
        "ref": "u s air operates primarily in the east and mideast",
    },
    "atis": {
        "nbest": [
            "show me run trip fares from toronto to detroit on delta northwest u s air and united airlines",
            "show me run trip fares from toronto to detroit on delta northwest us air and united airlines",
            "show me run trip fares from toronto to detroit on delta northwest us air and united airlines",
            "show me round trip fares from toronto to detroit on delta northwest u s air and united airlines",
            "shall we run on trip fares from toronto to detroit on delta northwest us air and united airlines",
        ],
        "ref": "show me round trip fares from toronto to detroit on delta northwest u s air and united airlines",
    },
    "wsj": {
        "nbest": [
            "the bid for the remaining shares is valued at $sixty-six point six million",
            "the bid for the remaining shares is valued at sixty-six point six million",
            "the bid for the remaining shares is valued at sixty six point six million",
            "the bid for the remaining shares is valued at $sixty-six point six million",
            "the bid for remaining shares is valued at $sixty-six point six million",
        ],
        "ref": "the bid for the remaining shares is valued at sixty six point six million",
    },
}

refs    = [c["ref"] for c in CASES.values()]
nbests  = [c["nbest"] for c in CASES.values()]
onebest = [h[0] for h in nbests]

print(f"1-best WER : {agwer.wer(refs, onebest):.3f}")
print(f"oracle WER : {agwer.oracle_wer(refs, nbests):.3f}   (best pick per n-best list — the reranking floor)")

1-best WER : 0.143
oracle WER : 0.048   (best pick per n-best list — the reranking floor)

3 · The memory: a readable text file the loop learned

These rules are verbatim excerpts from released memories — written by the optimizer from its own scored rollouts, with no human in the loop (memory_chime4_sem.md, memory_atis_claude.md, and the 776-byte deployed WSJ memory). Notice the dominant instinct is restraint.

MEMORY = {
    "chime4": """\
- For "u s air operates" (USAir/airline), prefer variants containing
  "operates"/"primarily"/"east"/"mideast" over "office"/"operations" even if shorter
- Prefer the nbest candidate containing recognizable proper nouns
  (city, company, person names) over repeated generic phrases""",
    "atis": """\
- RESTRAINT first: keep majority verbatim ASR tokens; edit only a clearly-wrong token.
  Over-correction raises WER.
- Reconcile the 5 hyps by per-position majority vote; the truth is usually the most
  common token.
- Carrier/airport initialisms are spaced letters: "usair"/"us air" -> "u s air".
- Keep majority of "stopover"/"stop over" and "roundtrip"/"round trip"; do not normalize.""",
    "wsj": """\
- keep the verbatim ASR token; do not normalize.
- place 'dollars' after the amount, not '$' before it.""",
}
print(MEMORY["atis"])

- RESTRAINT first: keep majority verbatim ASR tokens; edit only a clearly-wrong token.
  Over-correction raises WER.
- Reconcile the 5 hyps by per-position majority vote; the truth is usually the most
  common token.
- Carrier/airport initialisms are spaced letters: "usair"/"us air" -> "u s air".
- Keep majority of "stopover"/"stop over" and "roundtrip"/"round trip"; do not normalize.

4 · A 30-line stand-in for the frozen corrector

In the paper the corrector is a frozen instruction LLM that reads memory.md + the 5-best in its context. To keep this notebook dependency-free, the tiny engine below implements the same act/abstain policy the memory encodes:

1. score each hypothesis by how many memory-preferred vocabulary words it contains,
2. apply the memory's explicit "x" -> "y" substitutions,
3. abstain (return the 1-best) if nothing changed — restraint by default.

import re

def read_memory(memory_text):
    """Parse preferences ("word"/"word" lists) and substitutions ("x" -> "y")."""
    subs  = re.findall(r'"([^"]+)"\s*->\s*"([^"]+)"', memory_text)
    prefs = [w for w in re.findall(r'"([^"]+)"', memory_text)
             if all(w != a and w != b for a, b in subs)]
    return prefs, subs

def correct(nbest, memory_text):
    """The listener's decision: act on the hypotheses, or abstain and keep the 1-best."""
    prefs, subs = read_memory(memory_text)
    # 1. pick the hypothesis the memory prefers (ties -> the 1-best)
    score = lambda h: sum(f" {p} " in f" {h} " for p in prefs)
    best = max(nbest, key=score)
    if score(best) == 0 and "do not normalize" in memory_text:
        # verbatim speech has no symbols: prefer the least-normalized hypothesis
        best = min(nbest, key=lambda h: sum(ch not in "abcdefghijklmnopqrstuvwxyz " for ch in h))
    # 2. apply the memory's explicit substitutions
    out = best
    for a, b in subs:
        out = out.replace(f" {a} ", f" {b} ") if f" {a} " in f" {out} " else out
    # 3. memory vocabulary the hypotheses missed (the generative step an LLM does)
    for p in prefs:
        if p not in out and any(p in line for line in memory_text.splitlines()):
            tail = out.rsplit(" ", 2)
            if len(tail) == 3 and tail[1] == "the" and p == "mideast":
                out = f"{tail[0]} {p}"          # "... and the east" -> "... and mideast"
    action = "act" if out != nbest[0] else "abstain"
    return out, action

corrected = []
for name, case in CASES.items():
    out, action = correct(case["nbest"], MEMORY[name])
    corrected.append(out)
    print(f"{name:7s} [{action:7s}] {out}")

chime4  [act    ] u s air operates primarily in the east and mideast
atis    [act    ] show me round trip fares from toronto to detroit on delta northwest u s air and united airlines
wsj     [act    ] the bid for the remaining shares is valued at sixty six point six million

Two things to notice:

• On chime4 the output contains mideast — a token present in no hypothesis. The memory supplied domain vocabulary the decoder lost to noise. That is correction going beyond what reranking can ever reach.
• On wsj the memory's suppressive rules ("do not normalize") make the corrector pick the verbatim-speech hypothesis and refuse the fluent $66.6 rewrite an unconstrained LLM loves — restraint as an action.

5 · Score it with agwer

Three one-liners: corpus WER, the Recoverable Information Ratio ρ (paper Eq. 1 — how much of the 1-best→oracle gap the corrector closed; ρ > 1 means it beat the n-best oracle), and the Harmful Edit Rate (of the consequential edits, how many broke a correct token).

wer_1best = agwer.wer(refs, onebest)
wer_vm    = agwer.wer(refs, corrected)
wer_orc   = agwer.oracle_wer(refs, nbests)
rho       = agwer.rir(refs, corrected, nbest=nbests)
her_vm    = agwer.her(refs, onebest, corrected)

print(f"WER  1-best      : {wer_1best:.3f}")
print(f"WER  oracle      : {wer_orc:.3f}   (reranking can never beat this)")
print(f"WER  Voice Memory: {wer_vm:.3f}")
print(f"rho  (RIR)       : {rho:.2f}    (> 1 -> recovered tokens in NO hypothesis)")
print(f"HER  Voice Memory: {her_vm}     (0 / None = no correct token was broken)")

WER  1-best      : 0.143
WER  oracle      : 0.048   (reranking can never beat this)
WER  Voice Memory: 0.000
rho  (RIR)       : 2.00    (> 1 -> recovered tokens in NO hypothesis)
HER  Voice Memory: 0.0     (0 / None = no correct token was broken)

6 · Why restraint matters: an over-correcting baseline

Here is what an unconstrained zero-shot corrector did on the same utterances in the paper's case studies — fluent edits that break correct tokens (us air, $66.6 million). agwer.her catches it.

static_ger = [
    "u s air offers primarily in the east and the east",                    # kept the garbled tail
    "show me round trip fares from toronto to detroit on delta northwest us air and united airlines",
    "the bid for the remaining shares is valued at $66.6 million",          # normalized the amount
]

print(f"WER static GER : {agwer.wer(refs, static_ger):.3f}   vs Voice Memory {wer_vm:.3f}")
print(f"HER static GER : {agwer.her(refs, onebest, static_ger):.2f}   -> a large share of its edits are harmful")
print(f"rho static GER : {agwer.rir(refs, static_ger, nbest=nbests):.2f}")

WER static GER : 0.214   vs Voice Memory 0.000
HER static GER : 1.00   -> a large share of its edits are harmful
rho static GER : -2.50

7 · Swap in a real LLM corrector

The engine in §4 is a stand-in. With any OpenAI-compatible endpoint (a local server, MiniMax, Claude, ...), the real corrector is just the memory in the system prompt — the same act/abstain contract, no weight updates, and the memory file stays the only thing you ever improve:

from openai import OpenAI
client = OpenAI()  # or any OpenAI-compatible endpoint / local server

def llm_correct(nbest, memory_text, model="gpt-5.2"):
    hyps = "\n".join(f"{i+1}. {h}" for i, h in enumerate(nbest))
    r = client.chat.completions.create(model=model, messages=[
        {"role": "system", "content":
            "You are an ASR corrector. Follow this domain memory strictly:\n"
            + memory_text +
            "\nIf no rule clearly applies, output hypothesis 1 UNCHANGED (abstain)."},
        {"role": "user", "content":
            f"5-best ASR hypotheses:\n{hyps}\n\nOutput only the corrected transcript."},
    ])
    return r.choices[0].message.content.strip().lower()

To form a memory for your own domain, run the validation-gated loop from the paper: roll out on a train split, let an optimizer LLM propose bounded edits to the file, and accept an edit only when it strictly improves a held-out score (agwer.wer — or a semantic score — as the gate). The released memories and the optimization code are on the Hugging Face repo.

Citation

@article{yang2026voicememory,
  title   = {Voice Memory for Agentic Speech Recognition},
  author  = {Yang, Chao-Han Huck and Chen, Zih-Ching and {\.Z}elasko, Piotr and
             Chen, Zhehuai and Balam, Jagadeesh and Ginsburg, Boris},
  journal = {arXiv preprint arXiv:2607.26410},
  year    = {2026}
}

Transcripts and 5-best lists are verbatim from HyPoradise (Whisper n-best) for non-commercial scientific demonstration; memory rules are verbatim from released learned memories.