Liuyuan Jiang / Research
Publications & projects

ICASSP 2026Speech representation learning

BiRQ: Bi-Level Self-Labeling Random Quantization for Self-Supervised Speech Recognition

Liuyuan Jiang, Xiaodong Cui, Brian Kingsbury, Tianyi Chen, Lisha Chen

Read the paper Cite

Summary

Self-supervised speech models learn by predicting targets at masked audio frames. BiRQ constructs those targets from the encoder’s own intermediate representations and trains them jointly with the encoder. A second set of targets, obtained by random quantization of raw audio features, retains a fixed reference task. This combines representation-dependent target refinement with the single-loop training structure of BEST-RQ.

  • Intermediate encoder features generate enhanced pseudo-labels, while fixed raw-input labels define an anchoring task. Gumbel-softmax allows gradients to pass through enhanced-target construction. Figure 1
  • The penalty implementation updates one shared encoder using both loss gradients. The minimum anchoring loss is constant with respect to the parameters, so no auxiliary lower-level solve is required. Equation 9 · Algorithm 1
  • For the 155M Conformer with matched 100-epoch LibriSpeech pretraining, test-clean/test-other WER decreases from 6.8%/19.6% with BEST-RQ to 5.0%/12.6% with BiRQ. Table 2
  • For the 275M Conformer pretrained on YODAS, AMI headset/distant-microphone WER decreases from 18.4%/37.2% with BEST-RQ to 16.3%/34.0% with BiRQ. Table 3

Target refinement within a shared speech encoder

From fixed labels to learned representations

BEST-RQ turns unmasked audio features into pseudo-labels using a fixed random projection and codebook, then trains an encoder to predict those labels from masked audio. The targets remain unchanged as the encoder learns. HuBERT-style relabeling uses learned features to refine targets through separate labeling stages.

A target generator inside the prediction model

BiRQ reuses the first part of the acoustic encoder to generate targets from unmasked speech. Normalized intermediate features pass through a fixed random projection and a Gumbel-softmax quantizer. The resulting enhanced targets depend differentiably on the same encoder parameters used for masked prediction.

An anchoring task for joint learning

The raw-input targets are retained as a reference task independent of the encoder parameters. BiRQ minimizes the enhanced-target loss while requiring the anchoring loss to stay near its minimum. The penalty implementation brings both tasks into one training loop, with gradients through target construction as well as prediction.

From the paperView full size
Original BiRQ architecture: a shared speech encoder generates enhanced labels from intermediate features, alongside anchoring labels from raw input.
Figure 1 · Two sources of speech-learning targets

A shared encoder processes unmasked speech to generate enhanced targets from intermediate features through Gumbel-softmax selection. A fixed random quantizer maps raw input to anchoring targets. The masked prediction task learns from both branches, with gradients through enhanced-target generation.

Original paper source

Two inputs, one encoder. The unmasked input supplies the raw-input and intermediate-feature targets. The masked input passes through the full encoder for prediction. Cross-entropy is evaluated at the masked positions.

Fixed and representation-dependent targets. The raw-input branch generates fixed anchoring labels. The intermediate-feature branch uses the first part of the encoder, a fixed random projection and a Gumbel-softmax quantizer to generate enhanced targets.

Shared parameters and gradient paths. Enhanced-target construction and masked prediction share encoder parameters. The enhanced-loss gradient includes both paths, while the anchor targets remain independent of the encoder.

Joint target generation and masked prediction

Generate both targets from unmasked speech. The anchor branch quantizes raw audio features by nearest-neighbor assignment. The enhanced branch quantizes normalized intermediate encoder features with Gumbel-softmax selection. Both branches use fixed random projections and a fixed codebook.

Predict the targets at masked frames. A masked version of the input passes through the full encoder and its prediction head. Enhanced-label and anchoring-label cross-entropy are evaluated at the masked positions.

Update the shared encoder. Algorithm 1 combines the two loss gradients. The enhanced-loss gradient passes through both the masked prediction and the unmasked target-generation path. The experiments set the enhanced and anchor loss weights to 0.1 and 2.4, respectively, and the Gumbel-softmax temperature to 0.5.

The anchor optimum is constant in this formulation

Both objectives optimize the same encoder parameters. The minimum anchoring loss in Equation 9 is a constant rather than a value function of a separate outer variable. Removing that constant leaves the weighted loss used in Algorithm 1 and avoids an auxiliary lower-level solve.

The anchor-constrained objective and its penalty implementation

Notation

x,  x~x,\;\tilde x
Unmasked normalized speech-feature sequence and its masked version.
θ\theta
Parameters of the shared acoustic encoder.
k,  Kk,\;K
Layer used to generate enhanced targets and total encoder depth.
z(k)(θ;x)z^{(k)}(\theta;x)
Normalized intermediate representation from the first k encoder layers.
y(x)y(x)
Fixed raw-input anchoring target generated by nearest-neighbor quantization in Equation 2.
y(k)(θ;x)y^{(k)}(\theta;x)
Encoder-dependent enhanced target generated by Gumbel-softmax selection in Equation 4.
F(θ),  G(θ)F(\theta),\;G(\theta)
Enhanced-target and anchoring-target cross-entropy losses, averaged over training utterances and evaluated at masked frames.
Sδ(θ),  δS_\delta(\theta),\;\delta
The feasible encoder parameters in Equation 8 and their allowed anchoring-loss gap above its minimum. θ′\theta^{\prime} is the variable in that minimum.
w1,  w2w_1,\;w_2
Weights on enhanced and anchoring losses. The ratio w2/w1w_2/w_1 is the penalty constant in Equation 9.
η\eta
Learning rate in the Algorithm 1 parameter update.
Equation 8 · Enhanced-target objective
min⁡θ∈Sδ(θ)F(θ)\min_{\theta\in S_\delta(\theta)}F(\theta)
Equation 8 · Anchor-loss feasible set
Sδ(θ)={θ:G(θ)−min⁡θ′G(θ′)≤δ}S_\delta(\theta)=\left\{\theta:G(\theta)-\min_{\theta^\prime}G(\theta^\prime)\leq\delta\right\}

In Equation 8, F(θ)F(\theta) is cross-entropy against enhanced targets and G(θ)G(\theta) is cross-entropy against fixed anchor targets. The allowed anchor-loss gap δ\delta defines Sδ(θ)S_\delta(\theta). Equation 9 penalizes that gap. Its minimum anchor-loss term is constant with respect to θ\theta, so Algorithm 1 uses the weighted gradients of F(θ)F(\theta) and G(θ)G(\theta) without computing the minimum.

Equation 9 · Penalized anchor-constrained objective
min⁡θw1F(θ)+w2(G(θ)−min⁡θ′G(θ′))\min_{\theta} w_1F(\theta)+w_2\bigl(G(\theta)-\min_{\theta^\prime}G(\theta^\prime)\bigr)
Algorithm 1 · Weighted-gradient update
θ←θ−η(w1∇F(θ)+w2∇G(θ))\theta\leftarrow\theta-\eta\bigl(w_1\nabla F(\theta)+w_2\nabla G(\theta)\bigr)

Equation 8

Enhanced-target learning constrained by the anchor task

The objective minimizes enhanced-label cross-entropy over encoder parameters whose anchoring loss lies within a prescribed gap of its minimum. Both losses use the shared encoder and masked-frame prediction task.

Equation 8 in the paper

Lemma 1

A conditional link between penalty and constrained objectives

With bounded enhanced-loss subgradients on exact anchor minimizers and a Hölderian error bound for the anchor-loss gap, a sufficiently large loss-weight ratio transfers approximate optimality from the penalty problem to the anchor-constrained problem.

Lemma 1 in the paper

Differentiating through the enhanced labels

The enhanced targets depend on the encoder through the unmasked intermediate representations. Gumbel-softmax provides a differentiable distribution over codebook entries, so the enhanced-loss gradient includes the effect of changing the targets as well as changing the predictions.

From the anchor constraint to the penalty objective

Equation 8 constrains the anchoring loss to be within a prescribed gap of its minimum. Equation 9 adds that gap to the enhanced loss with a penalty controlled by the ratio of the loss weights. The anchor minimum contributes no parameter gradient.

The role of the error-bound condition

Lemma 1 links the penalty objective to the anchor-constrained problem under bounded enhanced-loss subgradients and a Hölderian error bound for the anchor-loss gap. The condition determines how the loss-weight ratio must grow as the allowed gap shrinks.

Assumptions behind the analysis

Lemma 1 assumes bounded enhanced-loss subgradients on exact anchor minimizers and a Hölderian error bound for the anchor-loss gap, with exponent greater than one. A sufficiently large loss-weight ratio, chosen according to the error-bound exponent and allowed anchor gap, then transfers approximate optimality from the penalty problem in Equation 9 to the constrained problem in Equation 8. Algorithm 1 specifies the gradient implementation.

Section 3 · Equations 8–9, Lemma 1 and Algorithm 1 in the paper

Further details

How is the target-generating layer chosen?

The paper uses a pilot heuristic near 70% of encoder depth and selects layer 7 for the 10-layer experiments. The 5-layer ablation in Table 1 compares layers 2, 3 and 4. At 100 epochs, layer 4 yields the lowest WER among those three choices.

How does online refinement compare with staged relabeling?

Table 1 includes a 137M Conformer comparison with offline iterative relabeling, where each retraining stage uses targets from a previously trained model. The first 100-epoch relabeling stage yields 18.6% test-other WER, compared with 19.1% for the 100-epoch layer-3 BiRQ setting. BiRQ reaches 17.2% after 300 epochs. Interpreting these rows requires accounting for the preceding label-encoder training as well as the current training stage.

Related projects

BiRQ applies a penalty formulation to joint target construction and representation learning. PBGD-Free studies a different setting, where a lower-level optimum depends on an outer decision, and identifies flatness conditions for removing its value-function gradient.

PBGD-Free

Illustration

Information paths for the two target sources

Information flow / schematic

Two target-generation branches

Audio featuresObserved input
Raw-input quantizationAnchor labels
Intermediate representationEnhanced-label source
Target predictionLearning with both label sources

Both target sources feed the shared masked-frame prediction task.

Self-supervised pretraining for recognition

The encoder predicts pseudo-labels from unlabeled audio during pretraining. It is then fine-tuned with transcripts using CTC and evaluated by word error rate. The LibriSpeech and AMI experiments measure whether joint target refinement improves this downstream recognition task.

Recognition accuracy after self-supervised pretraining

LibriSpeech with a 155M Conformer

Table 2 · C2 · word error rate (%) ↓
Methodtest-cleantest-other
Supervised baseline7.420.5
BEST-RQ6.819.6
BiRQ5.012.6
Result

BiRQ reduces test-clean WER by 1.8 percentage points and test-other WER by 7.0 percentage points. The latter corresponds to a 35.7% relative reduction from the reported BEST-RQ value. Both methods use the same C2 encoder and 100-epoch pretraining budget.

Experiment setup and comparison details

C2 is a 10-layer, 155M-parameter Conformer with full attention. BEST-RQ and BiRQ are pretrained for 100 epochs on 960h LibriSpeech. BiRQ uses layer 7 to generate enhanced targets. Pretraining uses AdamW, batch size 64 and learning rate 1×10⁻⁴. Fine-tuning uses CTC on 100h labeled LibriSpeech for a 30-epoch warmup, hold and annealing schedule. Decoding uses the LibriSpeech 4-gram language model. Table 2 reports point estimates.

Read the original result

YODAS pretraining and AMI meeting recognition

Table 3 · C3 · word error rate (%) ↓
MethodAMI IHMAMI SDM
Supervised baseline25.747.0
BEST-RQ18.437.2
BiRQ16.334.0
Result

BiRQ reduces WER in both microphone conditions: by 2.1 percentage points for individual headset microphones and 3.2 percentage points for the single distant microphone. This comparison uses the same C3 encoder and pretraining corpus for both methods.

Experiment setup and comparison details

C3 is a 10-layer, 275M-parameter Conformer with an attention window of 200. BEST-RQ and BiRQ are pretrained for 100 epochs on 5k hours of YODAS. BiRQ uses layer 7 to generate enhanced targets. Pretraining uses AdamW, batch size 128 and learning rate 1×10⁻⁴. Fine-tuning uses CTC on 150h labeled AMI with the same 30-epoch schedule as LibriSpeech. Decoding uses an in-house interpolated 4-gram language model. IHM denotes individual headset microphones and SDM denotes a single distant microphone. Table 3 reports point estimates.

Read the original result

Evaluation protocol

Tables 2 and 3 evaluate the pretrained representations after CTC fine-tuning and 4-gram language-model decoding. They report point estimates of word error rate for the specified data and architecture configurations.

Cite this work

@inproceedings{jiang2026birq,
  title = {BiRQ: Bi-Level Self-Labeling Random Quantization for Self-Supervised Speech Recognition},
  author = {Liuyuan Jiang and Xiaodong Cui and Brian Kingsbury and Tianyi Chen and Lisha Chen},
  booktitle = {2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year = {2026}
}

Preprint first released in 2025.

Research sources