Audio Language Models

What is an audio language model?

An audio language model is a regular causal LM (think ChatGPT or Claude) that’s been taught to read sound the way it reads text. You give it a waveform and a prompt (“transcribe this”, “what genre is this?”, “write a caption for this clip”), and it generates a text response. Internally, the audio is converted into a sequence of vectors that get fed into the LLM right alongside the text embeddings. The LLM never sees the waveform itself, only vectors that summarize it.

The five papers in this session all follow this same skeleton, but each picks a different audio encoder, a different projection module, and a different training schedule. The architectural distance between them is small. The conceptual distance is even smaller. But reading them back-to-back can feel overwhelming, because the surface area adds up. We lay out the shared recipe first, then look at what each paper changes about that recipe and why, ending with AudioChat, which extends the recipe in the other direction by letting the LM write audio back.

The five papers, in the order they appear here:

• LTU (Listen, Think, Understand) (Gong et al., 2024). AST encoder, linear projection, LLaMA-7B. The first paper to formalize the perception-to-understanding curriculum.
• LLark (Gardner et al., 2024). Jukebox encoder, linear projection, LLaMA-2-7B-Chat. A music specialist that uses a generative encoder.
• Music Flamingo (NVIDIA, Nov 2025). AF-Whisper, two-layer MLP, Llama 2, Rotary Time Embeddings, GRPO reasoning. Built on top of Audio Flamingo 3 and post-trained for music.
• Audio Flamingo Next (NVIDIA, Apr 2026). AF-Whisper, two-layer MLP, Qwen-2.5-7B with 128k context, plus a streaming TTS module for voice-to-voice.
• AudioChat (Chen et al., Feb 2026). Adobe + CMU. The bidirectional one: read and write audio from a single LM via Audio Transfusion Forcing, with chain-of-thought reasoning interleaved into the audio token stream.

If audio LMs are new, this is the place to start. The interactive below walks through the pipeline a 1-second clip travels: raw waveform, mel spectrogram, frozen encoder, projector, and finally a short bundle of audio tokens sitting at the front of the LLM’s input sequence. Every paper in this session is a variant of this one pipeline.

The five panels recur across the rest of the post under different names. The encoder differs by paper (AST in LTU, Jukebox in LLark, AF-Whisper in the Flamingo line, codec tokens in AudioChat). The projector differs by paper (linear, MLP, Q-Former). What stays frozen and what gets thawed differs by paper. The pipeline shape, though, is shared, and once it’s clear end-to-end, the rest of the post reads as variations on this one theme.

What trips people up

Before getting into the papers, here are four things that consistently trip up reading groups, especially folks coming from an adjacent ML field. Each card has a tiny figure to anchor the idea.

The vision-LM cousins: LLaVA and BLIP-2

Quick stop, because we keep mentioning these. LLaVA and BLIP-2 are image-and-text models from 2023 that do exactly what audio LMs do, but with pictures. You hand them an image and a prompt (“what’s in this photo?”, “write a caption”), and they generate text. The recipe is the same one we just walked through, with the audio encoder swapped for a vision encoder.

LLaVA uses a frozen CLIP Vision Transformer to encode the image, a small projector (originally linear, later a 2-layer MLP) to remap features into LLM space, and a LoRA-trained LLM on top. Two-stage curriculum: first align the projector, then instruction-tune the LLM. BLIP-2 is the same idea but the projector is a Q-Former (a small transformer with learned query vectors that cross-attend to the vision features and emit a fixed-size token bundle).

If you’ve never used either, that’s fine. The point is just that the audio papers in this post are doing the same thing, with a sound encoder instead of a vision encoder. ViT becomes AST, BEATs, AF-Whisper, or Jukebox. Image patches become audio frames. “Describe this image” becomes “describe this clip.” The fusion math is identical. Anyone who already understands why LLaVA works already understands why LTU works; they just haven’t looked at it yet.

The projector ladder

The projector is the only module with any real architectural variety across these four papers. The interactive below has all four variants side by side; click through to see what each one looks like.

The four variants form a complexity ladder. The linear projector is one matrix W ∈ ℝ^(d_audio × d_llm), on the order of 10M parameters. It works because the upstream encoder is already strong, so the projection just needs to remap dimensions. The MLP variant adds one nonlinearity (Linear → GeLU → Linear) for slightly more expressive remapping. Both variants are one-to-one in sequence length: every encoder frame produces one LLM token. They’re a fine choice when the encoder already produces a manageable number of frames.

The Q-Former is a small transformer with a fixed bank of K learned query vectors (typically 32 or 64) that cross-attend to the encoder output. It produces exactly K tokens regardless of input length, which is how you turn 1024 audio frames into 32 LLM-ready tokens. The Perceiver Resampler is basically a multi-layer Q-Former with the same compression property, and it’s what original DeepMind Flamingo used.

The shorthand: linear and MLP say “audio frames already are the tokens, just remap them.” Q-Former and Perceiver say “I have many audio frames; please summarize them into a fixed-size set.” Same conceptual job, different tradeoffs in sequence-length cost and compute.

Soft prompt vs. gated cross-attention

This is where the Flamingo naming trap really lives. There are two ways to get audio tokens into a causal LM, and the four papers in this post all use the simpler one.

The left panel is what every paper in this post does. Audio tokens get prepended to the text token sequence, and the model runs a vanilla causal LM forward pass over the whole concatenated sequence. The LLM is unchanged structurally: same blocks, same attention, same training loop. You can drop this into any HuggingFace causal LM and it works.

The right panel is the original DeepMind Flamingo design. Audio tokens don’t go into the main sequence. Instead, every transformer block (or every k-th block) gets an extra gated cross-attention branch with the update rule y ← y + tanh(α) · xattn(y, audio_tokens).

α is a learnable scalar initialized to zero. Since tanh(0) = 0, the augmented LM is bit-identical to the pretrained LM at step 0, and the gate opens slowly during training. That was the slick 2022 trick: keep the LM frozen, never risk forgetting, just bolt on new attention layers and let them learn from zero influence upward.

So why has almost everyone moved away from it? Three reasons. First, it requires architectural surgery; you have to insert custom layers into the LLM’s forward pass, and it doesn’t compose cleanly with HuggingFace transformers or LoRA. Second, long audio gets awkward, because the Resampler caps N at 64 and an hour-long clip would need a much bigger Resampler. Third, the soft-prompt approach is just easier to implement and seems to work comparably in practice. The LLaVA-style soft prompt won on engineering grounds, and Audio Flamingo, despite the name, picked it up.

The freeze schedule

Almost every audio-LM paper follows the same training-stage skeleton. The projector gets warmed up first, the LLM joins via LoRA or full fine-tuning, and open-ended tasks come in at later stages. The papers differ in how many stages they break this into and what data each stage uses. The interactive below has each paper’s recipe.

The pattern that recurs in all four: the encoder is frozen everywhere, always. The encoder was trained on far more audio than your instruction-tuning corpus has, so freezing it acts as a regularizer that also saves compute. Unfreezing it would risk overfitting to the much smaller caption distribution. LTU and LLark both use four stages with LoRA-only LLM training. Music Flamingo and Audio Flamingo Next train the LLM at full rank rather than LoRA, but still freeze AF-Whisper across every stage.

LTU has the cleanest framing of this: perception, then understanding. Stage 1 trains the projector on closed-ended classification tasks (“Is this a saxophone? Yes/No”) so the model first learns to perceive the audio. Only after the connector has found audio-shaped representations the LLM can read does the schedule introduce free-form QA, where the LLM has to reason about what it’s hearing. Starting with open-ended QA causes the LLM to fall back on its language prior and produce plausible-sounding answers without conditioning on the audio at all, a subtle and pervasive failure mode the paper explicitly identifies.

The end-to-end pipeline

The animation below renders the full pipeline as a single sequence. Each stage adds one transformation, with running shape annotations showing what’s getting passed forward at each step.

The shape contract is the load-bearing idea. After projection, audio embeddings are vectors of dimension d_llm, and text embeddings are vectors of dimension d_llm. The LLM operates on a sequence of (N + M, d_llm) vectors and runs a standard forward pass. Same shape language at every stage.

LTU: perception, then understanding

LTU is the cleanest expression of the paradigm. The audio encoder is the Audio Spectrogram Transformer (AST), pretrained on AudioSet-2M. A 10-second waveform gets converted to a 128-bin log-Mel spectrogram (25 ms window, 10 ms hop), split into 16×16 patches, processed by AST, and the resulting 512×768 embeddings are mean-pooled and projected to 32 audio tokens of dimension 4096 by a single learnable projection layer. Those 32 tokens are concatenated with text embeddings to form the input to LLaMA-7B (instruction-tuned via Vicuna).

Training has four stages with a strict perception-to-understanding curriculum. Stage 1 freezes the encoder and the LLM and trains only the projector on closed-ended classification (sound event tagging, acoustic feature description). Stage 2 introduces LoRA on the LLM but stays on closed-ended tasks. Stages 3 and 4 progressively fold in open-ended free-form QA. The authors put the motivation directly: “We find open-ended tasks to be too difficult for the model at the beginning of training. The model tends to use its language capability to answer the free-form open-ended question instead of conditioning on the audio input.” LTU therefore gates the harder task type until the connector has done its job.

The instruction-tuning data is OpenAQA-5M, a corpus of 1.9M closed-ended and 3.7M open-ended QA pairs the authors generated using GPT-3.5. About 5% of the questions are intentionally unanswerable, with answers like “This cannot be determined from the audio.” That tiny fraction does an outsized amount of work: it teaches the model to refuse hallucinations rather than confabulate plausible content.

LLark: a generative encoder for music

LLark is the music-specialist counterpart to LTU and shares the same skeleton. The biggest design choice is the encoder. Instead of AST (a discriminative encoder trained for classification), LLark uses Jukebox-5B, a generative music model. The 36th layer of Jukebox is extracted, mean-pooled to 10 Hz, and projected by a single linear layer to 32 audio tokens of dimension 4096 that get prepended to the LLaMA-2-7B-Chat input. The authors note that the projection layer adds “fewer than 0.1% additional parameters relative to the base models.” A single matrix is enough.

Why a generative encoder? The argument is that models trained to predict audio learn richer representations than models trained to classify it, especially for music, where labels are coarse relative to the signal content. This is the only architectural delta vs. LTU. Everything else (linear projector, 32 tokens, freeze-then-LoRA training) is the same recipe.

The training data is roughly 1.2M instruction triplets of the form (audio, query, response), generated by feeding Jukebox-derived metadata (tempo, key, mode, chord progressions) to GPT-3.5 with task-specific prompts spanning music understanding, captioning, and reasoning. The audio comes from MusicCaps, YouTubeM, MusicNet, FMA, MTG-Jamendo, and MagnaTagATune.

Music Flamingo: AF3 with RoTE and GRPO

Music Flamingo is the first of the two NVIDIA papers and the point where the recipe starts to evolve. It’s built on top of Audio Flamingo 3, inheriting the AF3 architecture (AF-Whisper encoder, two-layer MLP adapter, Llama 2 backbone) and post-training that architecture for music understanding. The audio encoder, AF-Whisper, is a Whisper-based custom encoder that takes 16 kHz log-Mel input and produces 1280-dimensional embeddings. Audio is processed in 90-second chunks, projected to the LLM input space by a two-layer MLP, and prepended as a soft prompt. Despite the name, no gated cross-attention.

The architectural delta is Rotary Time Embeddings (RoTE), a replacement for Llama 2’s standard RoPE positional encoding where the rotation angle θ comes from absolute timestamps τᵢ rather than token indices. RoTE grounds positional reasoning in elapsed time, which matters for long-form music, where temporally-anchored queries (“what happens around minute 4:30”) are way more useful than token-anchored ones.

Training has four stages. AF3-SFT fine-tunes the AF3 base on music data, multi-speaker ASR, and singing data. MF-SFT adds the MF-Skills dataset (about 4.3M captions across 3M hours of music, curated through an MLLM filtering pipeline) and MF-Think (300K chain-of-thought examples generated by gpt-oss-120b). MF-WarmUp is a cold-start SFT on the CoT pairs, which prepares the model for the RL phase. MF-GRPO applies Group-Relative Policy Optimization with three reward functions: a format reward for output-structure conformance, an accuracy reward for answer correctness, and a structured-thinking reward for valid reasoning-chain emission.

The encoder stays frozen across all four stages. The projector and the Llama 2 LLM are trained at full rank (no LoRA), which is a notable departure from the LTU/LLark recipe.

Audio Flamingo Next: long audio and voice

Audio Flamingo Next is the most recent of the four. The encoder is still AF-Whisper (frozen at every stage), the projector is still a two-layer MLP audio adapter, but the LLM swaps to Qwen-2.5-7B with native 128k context support. Three architectural deltas matter.

First, AF-Next supports long-audio inputs of up to 30 minutes in a single forward pass, made affordable by Sequence Packing and Hybrid Sequence Parallelism, two efficiency tricks for keeping a 128k-token sequence within a practical training budget. That’s roughly an order of magnitude longer than AF3’s ten-minute window. Second, AF-Next introduces time-grounded chain-of-thought via the AF-Think-Time dataset, a new corpus of about 43K samples where each reasoning chain is anchored to specific timestamps in the audio. Each chain averages 446 words and cites time ranges explicitly (“At 0:23–0:31 the snare enters; at 1:14 the texture thins out”). RoTE is what makes this possible, since the LLM has to reason about when events occur in addition to what occurs. Third, AF-Next adds a streaming TTS module, a separate decoder-only transformer that generates audio tokens conditioned on the LLM’s subword outputs. The streaming TTS realizes voice-to-voice operation: the model hears, reasons, and speaks back without any external synthesizer.

Training has six stages. The two pre-training stages align the audio adapter on short audio (30 seconds to 1 minute) with 8K context tokens. The two mid-training stages progressively expand the supported audio length to 10 minutes and then 30 minutes, with the context window growing from 24K to 128K tokens. The post-training stage runs GRPO with rule-based rewards plus 386K safety samples. The CoT-training stage performs SFT on AF-Think-Time and then a second GRPO pass to refine the timestamp-grounded reasoning. As in Music Flamingo, the encoder is frozen at every stage, and the projector and LLM train at full rank.

AudioChat: when the LM also writes audio

The four papers above read audio and write text. AudioChat (Chen, Seetharaman, Kumar, Nieto, Watanabe, Salamon, Jin, Adobe and CMU, Feb 2026) flips that direction. It reads audio and writes audio, alongside text reasoning, all from one LM. The motivation is what the authors call audio stories: clips with multiple speakers, foreground sound effects, and environmental backgrounds layered together. Reading audio stories is hard because captioning has to disentangle several streams at once. Writing them is hard because the model has to plan the layering before synthesizing it.

The architectural delta is the Audio Transfusion Forcing training objective. A vanilla causal LM predicts only text tokens given a context. Transfusion forcing predicts both text tokens and audio tokens against an interleaved target sequence: a structured chain-of-thought that decomposes each user instruction into reasoning steps, then emits the audio tokens those steps describe. The loss is a single next-token loss running over the merged stream, so the model learns to flip back and forth between modalities mid-utterance. Compared to the soft-prompt fusion of the other four papers (audio prefix into text output, with the modality boundary fixed at the start of the sequence), transfusion forcing has no modality boundary at all. Audio and text tokens occupy the same sequence at every position.

The training data is bootstrapped by LLM tool-calling agents simulating user-system dialogues. One agent role-plays the user (“make the door slam louder, then add background rain”), another the system. Their exchanges, along with the resulting audio tokens, become the supervision. AudioChat sidesteps the lack of natural multi-turn audio data by simulating it with already-fluent LLMs rather than collecting it. The same tool-calling pattern shows up again in the LLM-for-Audio-Effects papers from Week 7, where the tools are EQ and reverb modules instead of editing primitives.

What the model can do, all from one checkpoint:

• Storytelling: generate a multi-element soundscape from a text prompt (“an audio scene for approaching an old castle door”, or “a bustling medieval marketplace”).
• Editing: follow open-ended instructions to modify an existing clip (“make the door opening sound more dramatic, and faster”; sound addition or removal; volume; spatial positioning; multi-operation chains).
• Understanding: produce fine-grained captions of complex scenes with multiple speakers and sound effects. This is the same task LTU and AF-Next target, evaluated on the harder multi-source distribution.

The most interesting consequence, for our purposes, is the planning. Transfusion forcing places chain-of-thought inside the audio sequence rather than ahead of it as a text preamble. The model writes “first the door creaks, then a footstep, then the wind picks up”, and the actual audio tokens for those events follow inline. The same machinery works for editing: the model reasons about what to change before changing it. AudioChat is the closest these five papers get to a unified read-and-write audio LM.

Side by side

The five papers picked different points on a small set of axes. Here’s the one-page summary.

How they connect

Read in chronological order, the five papers tell a coherent story. LTU and LLark, both 2024, set the recipe: a frozen encoder, a small linear projector, prepended audio tokens, projector-first training, and LoRA on the LLM as the second stage. Music Flamingo and Audio Flamingo Next, both from NVIDIA in late 2025 and early 2026, scale that recipe (same soft-prompt fusion, same frozen encoder) while pushing on data scale, context length, and post-training. AudioChat, from Adobe and CMU in early 2026, breaks with the soft-prompt pattern entirely. Audio tokens stop being a prefix and start participating in the generated sequence, which opens the door to bidirectional read-and-write. The architectural variation across the first four papers is small. AudioChat is where the recipe actually changes shape.

A few open questions came up while we were prepping this session, in case you want to use them as discussion prompts.

• The shift from LoRA (LTU and LLark) to full fine-tuning (Music Flamingo and AF-Next) tracks the increase in data scale, but neither NVIDIA paper ablates LoRA against full fine-tuning at the larger scale. We don’t currently know whether full-rank training is preferred because it actually outperforms LoRA in this regime, or just because the compute is now affordable.
• Q-Former and Perceiver Resampler architectures are largely absent from current audio LMs, even though BLIP-2 still uses Q-Former on the vision side and Audio Flamingo 1 used a Perceiver. Modern audio LMs prepend hundreds of MLP-projected tokens instead. We’re unsure whether the shift is driven by larger context windows that make compression unnecessary, or by an unmeasured quality difference.
• The audio encoder is frozen at every stage in every paper, including Music Flamingo’s three-million-hour MF-Skills training. The regularization argument for freezing rests on the encoder having seen more data than the SFT set, but at this scale of SFT data, that argument might be worth revisiting.
• Music Flamingo builds on AF3 rather than on LLark, suggesting that music-specific work continues to feed back into general audio-LM research rather than diverging into a separate sub-community. We’re unsure how this affects the way music-specific results get evaluated and consumed.
• AudioChat moves chain-of-thought inside the audio token stream rather than running it as a text preamble. The other four papers either don’t reason explicitly or run CoT as text-only (Music Flamingo’s MF-Think, AF-Next’s AF-Think-Time). We don’t have an apples-to-apples ablation that would tell us when interleaved CoT actually helps over text-prefix CoT, and when it’s just architectural overhead.
• The bidirectional read-and-write setup also reopens the LoRA-vs-full-rank question from a different angle. AudioChat trains the LM jointly with the codec interleaving, which may not be tractable in a LoRA budget. We’re unsure what the minimum surgery would be to retrofit transfusion forcing onto an existing read-only audio-LM.

Papers & slides

Models, demos & tools