Inside Inkling: Audio Design

Thinking Machines Lab just released Inkling, their first open-weights model: a 975B-parameter Mixture-of-Experts Transformer with 41B active parameters, a context window of up to 1M tokens, and Apache-2.0 weights, pretrained from scratch on 45 trillion tokens of text, images, audio, and video. A smaller sibling, Inkling-Small (12B active), is previewed alongside it. The authors say plainly that it is not the strongest model available. It is built as a broad, customizable base, and as the background reasoning model for their real-time interaction models, where a user talks to the system with voice and vision. Read that way, the interesting question is not "is it SOTA," but "where did they spend capacity, and where did they refuse to?"

Audio is the sharpest answer, and it is the part I care most about as a speech researcher. Inkling has no audio encoder. Sound becomes roughly twenty tokens per second through a discretize-and-embed front end, then the same decoder that reads text does all the listening. This post walks through that decision in three parts: how audio is modeled, how well that actually works, and how the resulting interleaved inference is served. If you build voice agents or omni models, each part has one idea worth taking home.

Everything here traces to a primary source: the Thinking Machines model post and model card for architecture and benchmarks, the SGLang and Thinking Machines write-up on LMSYS Org for serving, SGLang PR #31358 for the audio rate, and the Tinker cookbook audio recipes for reproducible fine-tuning numbers. Figures from those sources are attributed inline.

Part 1 · ModelingAudio as tokens, not features

The usual way to give a language model ears is to bolt on an audio encoder: run the waveform through a stack of convolutions and transformer blocks, often tens to hundreds of millions of parameters, then project the features into the LM's embedding space. Inkling skips all of it. The model post is explicit that the multimodal inputs are encoder-free. Audio comes in as dMel spectrograms (Bai et al., 2024), "transformed via a light-weight embedding layer and processed jointly with text tokens." In plain terms: audio is tokenized, the same way text is.

Here is the whole pipeline, one step at a time. The point of the design is that nothing heavier happens.

Concretely: 16 kHz mono audio becomes an 80-channel log-mel spectrogram with a hop of 800 samples (50 ms) and a 1600-sample (100 ms) window. Each of the 80 mel energies in a frame is quantized to one of 16 levels; that is the "d" in dMel. The entire audio "encoder" is then a single embedding layer:

nn.Embedding(n_mel_bins × mel_vocab_size = 80 × 16 = 1280 → 6144)

Each (bin, level) pair owns one row of the table. A frame's 80 rows are summed into one 6144-dimensional vector, with an optional RMSNorm, and that vector is the audio token. The parameter cost is 1,280 × 6,144 ≈ 7.9M weights. For scale, that is two orders of magnitude smaller than a typical speech encoder checkpoint.

Why this is a reasonable bet

Why should summing 80 lookups work at all? Three observations:

• The bin index is carried in the table, not the position. Each mel bin owns its own block of 16 rows, so which bin a level came from is baked into the embedding itself. Summing is then a learned linear combination of per-(bin, level) vectors: a bag-of-bins with a fixed vocabulary of \(1280\) atoms. No positional bookkeeping inside the frame is needed.
• Discretization makes audio look like text. A continuous feature vector would need a projector and careful normalization to sit alongside text embeddings. Sixteen levels per bin turns each bin into a discrete symbol the model embeds exactly like a subword. The tokenizer does no "understanding," just as a text tokenizer doesn't.
• Perception is pushed into the shared decoder. The bet is that a 975B MoE is more than capable of turning a spectrogram into meaning, so a dedicated perceptual tower is redundant capacity. Vision makes the same bet with a 4-layer hMLP over 40×40 patches. Audio takes it further, to essentially zero layers.

The rate itself falls out of one constant. In the serving code, InklingAudioEncoderParams sets audio_token_duration_s = 0.05 and window_size_multiplier = 2.0. From these, hop = 0.05 × 16000 = 800, window = 1600, and, with no frame stacking or pooling, 16000 / 800 = 20 frames (tokens) per second. The released config exposes audio_mode: "dmel", n_mel_bins: 80, mel_vocab_size: 16, and the quantizer's clamp range (dmel_min_value: −7.0, dmel_max_value: 2.0). The rate lives in the code.

Because the rate is fixed and low, the context budget is trivial to reason about. The model card asks for WAV at 16 kHz and recommends keeping clips within 20 minutes. That guidance is about the training distribution, not a context ceiling: 24k tokens is a rounding error against 1M.

Part 2 · PerformanceDoes a lookup table cost you audio quality?

The obvious worry about throwing away the encoder, and quantizing the mel bins to 16 levels, is that you lose fidelity. The evidence says you largely don't. Here is the audio section of the model post's own evaluation table, reported at thinking effort 0.99, against the omni models they compare to:

Numbers from the model post, effort 0.99. Bold marks the best open-weights score per row. Two footnotes from the source worth keeping: AudioMC scores for the other models were evaluated internally by the Inkling team since they are not on the official leaderboard, and VoiceBench grades with hard-coded string matching, so a system message was added instructing models to follow the expected answer format. Kimi K2.5 and K2.6 report no audio numbers.

The honest read: Inkling leads every open-weights model on AudioMC by a wide margin, sits within a point of Qwen3.5 Omni-Plus on VoiceBench, and trails it on MMAU. Gemini 3.1 Pro, closed, leads all three. For a first release that hears through 20 embedding lookups per second, "among the strongest open-weights audio models" is not marketing. It is what the table says.

Leaderboards aside, what I find more useful are reproducible downstream results. The Tinker cookbook ships three audio fine-tuning recipes, and their reported numbers are a clean read on what the encoder-free representation can carry:

A few things worth drawing out:

• Clean speech is already solved at zero-shot. LibriSpeech WER ~0.06 out of the box means the dMel front end preserves enough signal for accurate transcription with no audio-specific tuning. The recipe authors note that most GRPO groups end up constant-reward on LibriSpeech because the base model is already too good at it. A fun problem to have.
• It adapts fast with LoRA. Medical dictation, Indian-accented English dense with drug names, roughly halves WER (0.157 to 0.072), lifts medical-entity recall from missing one term in five to fewer than one in eleven (0.806 to 0.915), and drops the character error rate on those entities from 0.102 to 0.037, all from a single SFT stage. It even transcribes drugs absent from the train split, like oxcarbazepine and somatropin. That is generalized transcription, not memorized vocabulary.
• Paralinguistics survive discretization. This is the one I would have bet against: 16 levels per mel bin sounds lossy for prosody. Yet the Expresso recipe classifies speaking style at 0.852 accuracy (macro-F1 0.861), up from 0.361. SFT alone reaches 0.757, and a GRPO stage with reward \( 0.5\cdot\mathbb{1}[\text{style}] + 0.5\cdot\max(0, 1-\text{WER}) \) adds the rest, while WER improves to 0.044. The representation carries both what was said and how. Sixteen levels is apparently enough.

Caveat, stated plainly. These recipes are small (hundreds to a few thousand clips) and explicitly meant to demonstrate the pipeline, not to chase SOTA. The Expresso authors note they did not tune hyperparameters. Read them as existence proofs: the encoder-free dMel representation is accurate zero-shot on clean speech, cheaply adaptable, and keeps paralinguistic detail.

Part 3 · InferenceServing interleaved audio, and the kernels it took

Inkling's decoder makes three departures from the common recipe. Each one breaks an assumption a standard decoder-only server is tuned for, which is why day-0 support (built with RadixArk on SGLang and Miles) needed dedicated kernels.

• Short convolutions. Per-channel causal convs of width \(W=4\), applied at two points: after the key and value projections, and on the attention and MLP residual-branch outputs: \( y_{t,c} = x_{t,c} + \sum_{d=0}^{W-1} w_{c,d}\,x_{t-d,c} \). Cheap short-range mixing, but four extra ops per layer that a normal attention prologue doesn't expect.
• Relative position, not RoPE. A learned relative bias added to the logits (Shaw et al., 2018; Huang et al., 2018). The authors report it extrapolates better to long sequences than RoPE. Sliding-window and global layers interleave 5:1 with 8 KV heads.
• Shared-expert sink. The MoE largely follows DeepSeek-V3 (256 routed plus 2 shared experts, top-6, sigmoid router with aux-loss-free load balancing), with one twist: the selected routed and shared scores are normalized jointly, \( (w_{\text{routed}}, w_{\text{shared}}) = \operatorname{normalize}(\log \sigma(s_{\text{routed}} \Vert s_{\text{shared}})) \), so the shared experts compete for probability mass instead of being an always-on add.

Where audio touches the serving path

At serve time, one audio item is a single sentinel token that expands into \(f\) d-mel-frame positions. The dMel featurization is CPU work done at render time, and the server reports the realized count per response as meta_info["audio_tokens"]. Two consequences follow. First, MoE routing is recorded after this expansion, so Miles' rollout routing replay (R3) indexes expert IDs over the media-expanded sequence, where the 20-per-second audio frames appear as concrete token positions. Second, the ShortConv branch keeps a small per-request convolution state, so a served request now has three heterogeneous state types to manage: full-attention KV, sliding-window KV, and ShortConv conv state.

That three-state shape is the interesting serving problem. SGLang handles all three through a single UnifiedRadixCache over typed components (FULL, SWA, MAMBA; the conv state reuses the pool built for Mamba-style layers even though Inkling has no SSM), with HiCache tiering and prefill-decode disaggregation transferring all three. MXFP8 KV roughly doubles cache capacity on Blackwell (≈4.7 µs quantization, ~5% decode penalty). Speculative decoding runs an 8-layer MTP chain captured in a single CUDA graph with a fully sync-free decode round, plus an optional DFlash draft model trained with the Modal team.

The architecture-specific kernels are where the measured wins are:

End to end on a B200 node at 8192-in / 1024-out:

The same architecture is trained with RL through Miles, which preserves train-inference consistency via precision-aligned ShortConv, relative-attention, and FP32-MoE kernels plus routing replay. Adapter-only LoRA sync ships just the low-rank delta between runtimes over CUDA IPC, cutting weight-update latency from 49.4 s to 2.5 s. That is the mechanism behind the fast audio fine-tuning in Part 2.

CodaWhat I don't know yet

A close read is also an inventory of gaps. Filing them here so they can be checked rather than implied away:

• The "flow" audio mode. The serving config types audio_mode as Literal["dmel", "flow"], and "flow" appears in no document I have found. The model card is clear that this release outputs text only, so my guess is an audio output path (a flow-matching decoder or vocoder) that exists in the codebase but is not exposed. If true, Inkling could eventually speak, not just listen. That is a guess, and it is the highest-value open question here.
• The exact quantizer. How log-mel values map onto the 16 levels (uniform over the clamp range? µ-law? learned?) is undocumented, and the released config's dmel_min_value: −7.0 disagrees with the serving code's default of −1.5. The checkpoint value should win, but it is worth confirming. Likewise the exact row layout of the embedding table is a convention I have assumed, not read from a spec.
• Scope. Behavior beyond the 20-minute guidance, multilingual and noisy speech, overlapping speakers, and streaming are all uncharacterized in public sources, as is audio's share of the 45T pretraining tokens. Note also that the production RL recipes keep the vision and audio front ends frozen during post-training; training them is experimental.

The design in one line

Inkling treats audio the way it treats text: tokenize, embed, and let the shared decoder do the perception. A dMel spectrogram, a ~7.9M-parameter lookup table, twenty tokens a second, no encoder. Judging by the benchmark table, the zero-shot WER, and how quickly ASR, medical dictation, and emotion tasks adapt, very little is lost for it. It is a modest amount of machinery for a capability that usually costs much more.

Sources & references

• Thinking Machines Lab, Inkling model post and model card (weights on Hugging Face): architecture, modalities, and the audio benchmark table (AudioMC, MMAU, VoiceBench).
• SGLang and Thinking Machines Lab, "SGLang and Miles Add Day-0 Support for Inkling," LMSYS Org (2026): serving optimizations and performance figures.
• SGLang PR #31358: the audio d-mel rate, feature extractor, and sentinel expansion.
• Tinker cookbook audio recipes (asr, medical_asr, emotion): the fine-tuning numbers in Part 2.
• R. He Bai et al., "dMel: Speech Tokenization Made Simple," 2024: the discretized-mel scheme.
• P. Shaw et al., "Self-Attention with Relative Position Representations," 2018; C.-Z. A. Huang et al., "Music Transformer," 2018.

The audio-modeling numbers (20 tokens/s, 50 ms hop, ~7.9M-parameter embedding) were derived from the linked SGLang source. Benchmark numbers are quoted from the model post's evaluation table, and downstream numbers from the Tinker cookbook recipe READMEs. Embedded performance figures are © the SGLang / Thinking Machines authors, reproduced with attribution. The interactive pipeline, interleaved-inference, and shared-sink diagrams are original. This post was drafted with help from Claude Code, and every claim was checked against the primary sources before posting.