Audio Representations

Why Representations?

Almost every system in the prior reading group sessions assumed it could already see audio in a useful form. DDSP took pitch and loudness as input. The transcription models took mel-spectrograms. Music generation models like VampNet took discrete audio tokens. None of them learned, from scratch, what audio means; they were each handed a representation and asked to do work on top of it. This session is about the layer underneath: how a model learns a representation in the first place, and what choices it makes about what to preserve.

We’re working through three papers that span two paradigms. CLAP — Contrastive Language-Audio Pretraining — is the audio-side port of the CLIP recipe: pair an audio encoder with a text encoder and train them to agree on which (audio, caption) pairs go together. We cover two CLAP papers, the original Microsoft CLAP (Elizalde et al., 2022) and LAION-CLAP (Wu et al., 2023) which scaled it up by an order of magnitude, because their differences are instructive about what scale and engineering buy you on top of a fixed objective. MERT — Music undERstanding model with large-scale self-supervised Training — Li et al., 2024 takes the opposite approach: learn from audio alone, hide chunks of the signal, predict what was hidden, no language anywhere in the loop. CLAP yields representations that are describable (queryable with a natural-language sentence); MERT yields representations that are characterizable (they encode pitch, harmony, and timbre as features for downstream tasks, but cannot be queried in words). Both are useful; the design choices are revealing.

Core Concepts

CLAP: A Technique With Several Implementations

“CLAP” names a technique — Contrastive Language-Audio Pretraining — before it names any one paper. The technique is the CLIP recipe ported to audio: pair an audio encoder with a text encoder, project both into a shared vector space, train with the symmetric InfoNCE loss so that matched (audio, caption) pairs land near each other and mismatched pairs land apart. At least two notable implementations exist, and we’ll walk through both before getting into the math, because they make the same big idea concrete in revealingly different ways.

Two notable implementations

The walkthrough below uses LAION-CLAP’s specifics where the engineering diverges (because Feature Fusion and K2C are LAION-only), but the loss-function math and the geometric intuition apply equally to both. After training, the encoders from either model are useful in isolation: the audio encoder becomes a general-purpose feature extractor, and the joint geometry enables zero-shot classification and retrieval.

The symmetric InfoNCE loss, line by line

For a batch of N matched pairs, let E_i^a, E_i^t ∈ ℝ⁵¹² be the unit-normalized audio and text embeddings. Because they live on the unit sphere, the inner product E_i^a · E_j^t equals their cosine similarity. The training objective is:

Reading it as code: the first log-term is the audio→text classification (for audio i, the right answer among N candidate captions is caption i); the second is the symmetric text→audio classification. Without the symmetric term the text tower receives only weak gradient; under symmetric training both encoders are updated by the same loss magnitude per pair. τ is a learnable scalar — both papers parameterize it in log-space to keep it positive, with initialization values that differ in a revealing way (covered just below). Projections are L2-normalized before the inner product is taken; without this, the dot product would conflate similarity with vector magnitude and the loss would just push everything toward larger norms.

Why is this the right loss? The mutual-information lower bound

The InfoNCE loss is not arbitrary; it is provably a lower bound on the mutual information between the two views (van den Oord et al., 2018). Specifically:

Pushing the loss down pushes the bound up, so minimizing ℒ provably increases (the bound on) I(audio; text). A consequence with practical bite: the bound is capped by log N. Doubling the batch size raises the ceiling by log 2 ≈ 0.69 nats. This is the formal reason contrastive methods are notoriously batch-size hungry — and why CLAP scales the batch from 768 to 2,304 to 4,608 as more corpora are mixed in. Empirically, doubling N improves zero-shot accuracy by 1–2 points across the board.

Feature fusion: handling variable-length audio  LAION-CLAP only

LAION-CLAP’s training set includes clips that run for minutes, but the audio encoder still expects a fixed-length input. Its solution is the Attention Feature Fusion (AFF) module — an attention-pooled aggregator that preserves both broad temporal context and local detail. (MS-CLAP, for comparison, simply sidestepped the problem by training only on fixed 5-second clips — clean, but discards anything longer.) The AFF algorithm is more involved than a simple chunk-and-pool. For a clip longer than 10 seconds:

The intuition: the global view preserves broad temporal context (a 5-minute song's overall structure) while the three locals preserve fine-grained detail at full resolution; AFF lets the model adaptively weight the two depending on the content. Without fusion, the encoder must choose between coarse resolution (downsample everything) or losing context (random 10s crop). For clips shorter than 10 seconds, the input is simply repeated and padded; no fusion is applied.

Keyword-to-Caption augmentation  LAION-CLAP only

LAION-CLAP scales captioned-data availability by an order of magnitude over MS-CLAP — not by finding more captioned audio, but by inventing captions for the much larger label-only corpora. Much of the available audio data (AudioSet’s 1.9M clips in particular) ships with sparse tag-style labels (a comma-separated list like dog, barking, outdoor) rather than fluent captions. CLAP turns these into training pairs by prompting a pre-trained T5 with the keywords and the instruction generate a caption from these keywords, yielding outputs along the lines of A dog is barking outdoors. A small post-processing step replaces gendered terms (the words man, woman) with the neutral person to mitigate downstream gender bias amplification. The result is roughly 1.9M synthetic (audio, caption) pairs from AudioSet alone, which dwarfs the genuinely captioned pool.

Training data and hyperparameters

LAION-CLAP’s full training corpus reaches approximately 2.5M (audio, caption) pairs across three sources: LAION-Audio-630K (633,526 pairs, ~4,325 hours of in-the-wild web-scraped audio), AudioCaps + Clotho (~55K human-captioned pairs, the only genuinely high-quality captions in the mix), and AudioSet (1.9M label-to-caption pairs via the K2C augmentation above). Training runs for 45 epochs with Adam (β₁=0.99, β₂=0.9), learning rate 1e−4, warmup followed by cosine decay. Batch size is scaled with corpus size: 768 for the smallest mix, 2,304 when LAION is included, 4,608 at full scale — a direct reflection of the log N bound discussed above. MS-CLAP, by contrast, used roughly 20× less data and 40 epochs at lr 1e−3 with batches as small as 32 — an instructive baseline for what the contrastive recipe can do with clean captions before scale enters the picture.

What does CLAP actually let you do?

The reason all this engineering matters is the downstream interface it unlocks: once the encoders are trained, you can describe any sound in natural language and find it (or sounds like it) in any indexed library. No retraining per category.

The downstream utility is broad and is the reason CLAP-style embeddings now appear in almost every recent audio generation system. Zero-shot classification: take the CLAP audio embedding, take the CLAP text embedding of each candidate class name (“dog barking,” “piano,” “rain”), pick the closest. Retrieval: build an index over a sound library and query it with a sentence. Conditioning: feed the CLAP text embedding to a diffusion model so that text prompts steer audio generation. None of these downstream uses require a fresh round of training; the representation does the work.

MERT: Learning Music From Itself

Li et al. (2024) take the position that aligning with text is the wrong starting point for music understanding, for two reasons. First, captions describe what humans choose to say about music, which is heavily biased toward genre, instrument, and mood, and almost never mentions key, harmonic function, or rhythmic structure; a representation aligned with that distribution will inherit its blind spots. Second, large-scale captioned music data is scarce compared to large-scale unlabeled music, and self-supervised approaches scale with raw audio hours rather than caption availability.

The architecture inherits from HuBERT in speech (Hsu et al., 2021). Raw waveform at 24 kHz is fed into a 7-layer convolutional feature extractor that downsamples to 75 Hz (one frame per ~13.3 ms, matching EnCodec’s tokenization rate exactly so the teachers and the encoder share a temporal grid). The frame sequence enters a Transformer encoder (12 layers for MERT-95M-public, 24 layers with Pre-LN for MERT-330M). During training, frames are masked HuBERT-style — the MERT paper inherits the masking hyperparameters from HuBERT rather than restating them, so the numbers here are HuBERT’s: roughly 8% of frames are sampled as starting indices and each is expanded into a 10-frame span, yielding approximately 50% of frames masked in expectation. Masked positions are replaced with a learned mask embedding, and the model must predict properties of the masked frames from the surrounding context. What MERT changes is the target of that prediction.

The acoustic teacher: EnCodec RVQ-VAE

The acoustic teacher is a pre-trained EnCodec residual vector-quantized autoencoder (Défossez et al., 2022). EnCodec processes the original (unmasked) audio and emits at each frame a stack of K = 8 codebook indices, where each codebook has 1024 entries. This factorized representation can express 1024⁸ ≈ 10²⁴ distinct combinations per frame, more than enough to reconstruct the waveform on decoding. MERT trains a per-codebook classification head on top of the transformer encoder. For each masked frame t in the masked set M and each codebook k from 1 to 8, the encoder must predict the right index out of 1024:

where x′ is the masked input sequence, z_t,k is the ground-truth code index from running EnCodec on the unmasked audio at frame t in codebook k, and f_k is the prediction head for codebook k. This is identical in form to HuBERT's loss but with codes drawn from EnCodec rather than k-means on MFCCs. The paper also explored a k-means alternative (300 centroids on log-mel + 200 centroids on chroma, joint cluster space = 60,000 classes); EnCodec proved more effective. Any feature in the encoder that helps minimize ℒ_acoustic must by construction preserve enough information to identify which of 1024 acoustic prototypes — spectral shape, transient pattern, broad timbre — the masked frame belongs to in each of 8 parallel quantizations.

The musical teacher: CQT regression

The musical teacher is the Constant-Q Transform of the original audio. The CQT differs from a mel-spectrogram in that its frequency bins are spaced logarithmically, with one bin per semitone (or a fraction thereof), which means harmonics of any pitch land at fixed, semantically meaningful positions independent of the fundamental. MERT trains a separate prediction head f_cqt to regress the CQT magnitudes at masked positions under an L2 loss:

where z_cqt,t is the CQT bin vector at frame t computed on the unmasked audio. This is regression, not classification: the head outputs a continuous vector and is trained to land near the target rather than choose among quantized categories. The choice matters — quantizing CQT would discard the magnitude information that distinguishes a forte from a piano, and would reintroduce the pitch-blurring that motivated using CQT in the first place.

The two losses are combined with a scalar weight α (the paper sweeps and finds α = 1 optimal):

Scale and training

The two principal model sizes are:

Batch sizes are expressed in audio-hours per step (a useful convention for music SSL): roughly 1.5 hours per batch for the 95M models, 5.5 hours per batch for the 330M. At 75 Hz, 5.5 hours corresponds to approximately 1.5 million frames per batch — large enough to make EnCodec teacher precomputation and on-disk caching non-trivial. Pre-LN was specifically introduced in the 330M variant because the original Post-LN HuBERT recipe diverged at this scale, a hard-won engineering lesson the paper documents in the appendix.

Evaluation: MARBLE and the probing protocol

After pretraining, the encoder is evaluated on the MARBLE benchmark (Yuan et al., 2023) plus separate evaluations covering genre classification, emotion recognition, key detection, beat tracking, source separation, and singer identification. The protocol is probing: the MERT weights are frozen and a small head is trained per task. The head is not always linear — the paper’s default probe is an MLP with one hidden layer of 512 units and 0.25 dropout, trained with batch size 64. Hyperparameter search is limited to a small learning-rate grid spanning 1e−4 through 1e−2. The 330M model achieves state-of-the-art across most tasks at the time of publication, including notably strong results on the pitch-sensitive tasks where prior masked-modeling work without a musical teacher had lagged.

What MERT lets you ask about a clip

Once the encoder is trained, the workflow is: feed a clip through MERT (frozen), grab the resulting frame-level features, and train a tiny task head on top. The same representation supports very different questions in parallel.

How They Connect

CLAP and MERT both seek a general-purpose audio representation, but they make almost-mirrored design decisions, and the resulting embeddings have correspondingly mirrored strengths.

The most useful way to read the pair is to ask, for any given downstream task, which kind of supervisory signal would have been most informative during pretraining? If the task involves matching audio to natural language (“retrieve clips that sound joyful,” “label by genre using a free-text vocabulary”), CLAP’s training distribution maps directly. If the task involves musical structure that captioners rarely mention (chord recognition, key detection, beat tracking, instrument fingerprinting), MERT’s frame-level acoustic supervision captures it more faithfully. Several recent systems combine both, using CLAP for the conditioning interface and MERT for the encoder backbone; the two are complementary in production even though they compete in the literature.

It is also worth noting how this pair maps onto the parallel debate in vision. CLIP (contrastive cross-modal) and DINO / MAE (self-supervised single-modal) emerged at roughly the same time for the same reason, and the field has not so much resolved the debate as accepted that both representations have their place. CLAP and MERT are the audio-side echo. The next several years of work in audio representation learning will likely follow the vision template: hybridization (use both signals together), distillation (compress the bigger model into a smaller deployable one), and progressively more music-specific second modalities (lyric-aligned text, score-aligned MIDI, performance metadata).

Further Reading

CLAP and MERT are entry points into much larger families. If either clicked, here’s how to navigate the literature next.

Beyond CLAP: variants and the music-specific cousins

The contrastive audio-text recipe has been extended in several productive directions. The most important pointer for a music-AI audience is MuLan, which is essentially “CLAP, but music-specific and at 44M-clip scale” — it’s the audio encoder behind Google’s MusicLM and the most direct music-domain analog of what we covered today. MuQ-MuLan is its 2025 successor and currently the SOTA music-CLAP.

Beyond MERT: other paths to music SSL

MERT is one of several approaches to self-supervised music understanding. The most useful contrasts are with MusicFM (which argues you don’t need a learned tokenizer at all — random-projection quantization works fine) and JukeMIR (which argues the right move is to skip SSL entirely and reuse a large generative model’s hidden states as features). MAP-Music2Vec is MERT’s direct predecessor from the same group and shows the lineage. MULE is the strongest non-transformer non-masked baseline.

Papers & Resources