Activation Steering for Accent Adaptation in Speech Foundation Models

Activation Steering for Accent Adaptation in Speech Foundation Models

Jinuo Sun, Yang Xiao, Sung Kyun Chung, Qiuchi Hu, Gongping Huang, Eun-Jung Holden, Ting Dang

Submitted to Interspeech 2026

中文版 README

Overview

Accent variability remains a major challenge in automatic speech recognition (ASR). Most adaptation methods rely on parameter fine-tuning without understanding where accent information is encoded.

This repository provides the official implementation of our paper, which:

1. Locates accent-sensitive layers in the audio encoder of Qwen2-Audio-7B through layer-wise activation patching and the proposed Accent Alignment Score (AAS).
2. Steers accented speech representations at inference time by injecting mean-shift vectors into identified layers — a parameter-free approach requiring no weight updates.
3. Compares steering against LoRA fine-tuning, showing that steering excels in data-scarce scenarios (< 100 training pairs) while LoRA performs better with abundant data.

Key Results

Accent	Baseline WER	Steering WER	LoRA WER	Steering ΔWER	LoRA ΔWER
Scottish	26.72%	6.80%	9.25%	-19.92%	-17.47%
South African	29.86%	4.35%	27.10%	-25.51%	-2.76%
Canadian	37.27%	3.47%	32.60%	-33.80%	-4.67%
Northern Irish	36.27%	6.64%	31.57%	-29.63%	-4.70%
Irish	31.91%	6.41%	30.28%	-25.50%	-1.63%
Arabic	18.13%	10.07%	7.20%	-8.06%	-10.93%
Hindi	14.26%	10.22%	7.82%	-4.04%	-6.44%
Spanish	15.31%	9.39%	8.61%	-5.92%	-6.70%

Project Structure

.
├── dataset/                  # Dataset construction
│   ├── vctk_dataset.py           # Build VCTK paired dataset
│   ├── l2arctic_dataset.py       # Build L2-ARCTIC + CMU-ARCTIC paired dataset
│   └── download_scripts/
│       └── download_vctk.sh
│
├── locate/                   # Experiment 1: Layer-wise Accent Subspace Analysis
│   ├── locate_experiment.py      # Main entry: 32-layer AAS sweep
│   ├── preprocess_audio.py       # Audio preprocessing & caching
│   ├── core/
│   │   ├── model_io.py               # Qwen2-Audio I/O (audio loading, prompt, inference)
│   │   └── patching.py               # Activation patching engine (probe/residual/full)
│   ├── utils/
│   │   ├── data_loader.py            # Data loader for paired JSON datasets
│   │   ├── cache_utils.py            # Activation & audio feature caching
│   │   └── eval_utils.py             # Metrics (log-likelihood, perplexity, WER, sensitivity)
│   ├── improvements/
│   │   ├── alignment_evaluator.py    # Accent Alignment Score (AAS)
│   │   ├── enhanced_metrics.py       # Normalized CE, probe classifiers
│   │   ├── analysis_utils.py         # Bootstrap CI, significance tests, Cohen's d
│   │   └── generation_eval.py        # Generation-based eval (WER, delta-WER)
│   ├── scripts/                      # Convenience shell runners
│   └── config/                       # YAML configs per accent
│
├── steering/                 # Experiment 2: Inference-Time Accent Steering
│   ├── compute_mean_shift.py     # Compute mean-shift vectors (standard − accent)
│   ├── generate_splits_vctk.py   # Generate leave-out splits for VCTK
│   ├── generate_splits_l2arctic.py
│   ├── run_leaveout.py           # Full leave-out experiment orchestrator
│   ├── leaveout_eval_single_sweep.py  # Per-layer steering evaluation
│   └── leaveout_eval_all_layer.py     # All-layer simultaneous steering
│
├── finetune/                 # Experiment 3: LoRA Fine-tuning Baseline
│   ├── train.py                  # Training entry (Seq2SeqTrainer + PEFT LoRA)
│   ├── evaluate.py               # WER evaluation
│   ├── data_utils.py             # Speaker-disjoint splitting, Dataset construction
│   ├── lora_utils.py             # LoRA config (audio_tower only, layer-range control)
│   ├── prepare_data.py           # Generate train/val/test splits
│   ├── prepare_steering_aligned_data.py  # Steering-aligned data splits
│   ├── evaluate_strict_aligned_steering.py  # Evaluate on identical steering test set
│   ├── scripts/                      # Convenience shell runners
│   └── config/
│       ├── l2arctic_config.yaml
│       └── vctk_config.yaml
│
├── tools/
│   └── cache_manager.py          # Cache inspection & cleanup
│
├── environment.yml           # Conda environment specification
├── requirements.txt          # Pip dependencies
└── experiment_outcome/       # Output directory (gitignored)

Setup

Prerequisites

• Python 3.11+
• CUDA-compatible GPU with ≥ 24 GB VRAM (for Qwen2-Audio-7B in bfloat16)
• Conda or pip

Note: The Qwen2-Audio-7B model (~16 GB) is downloaded automatically from HuggingFace on the first run. Ensure you have internet access and sufficient disk space.

Environment

conda env create -f environment.yml
conda activate research

Or install via pip:

pip install -r requirements.txt

Data Preparation

VCTK Corpus (native accents):

bash dataset/download_scripts/download_vctk.sh

This downloads VCTK Corpus 0.92 into data/vctk/.

L2-ARCTIC Corpus (non-native accents):

Download from the L2-ARCTIC website and place it in data/l2arctic/.

CMU-ARCTIC (standard reference for L2-ARCTIC):

Download from CMU-ARCTIC and place it in data/cmu_arctic/.

Build Paired Datasets

python dataset/vctk_dataset.py
python dataset/l2arctic_dataset.py

This generates JSON dataset files (data/vctk_locate_dataset.json, data/l2arctic_locate_dataset.json) containing text-matched utterance pairs.

Usage

Experiment dependencies: Experiments 1–3 all require the paired datasets from the Build Paired Datasets step. Experiment 2 (steering) and Experiment 3 (LoRA) are independent of Experiment 1 (locate), but Experiment 3 requires the steering splits from Experiment 2 Step 1 if you want strictly aligned comparison.

Experiment 1: Layer-wise Accent Subspace Analysis

Identify which encoder layers are most sensitive to accent-induced representation shifts using the Accent Alignment Score (AAS).

python locate/locate_experiment.py \
    --config locate/config/locate_config_vctk_scottish.yaml

Config files are provided for each accent under locate/config/. The experiment:

• Extracts layer-wise encoder activations for matched standard–accent pairs
• Computes AAS via mean-shift perturbation at each of the 32 layers
• Runs within-accent control tests to compute specificity scores
• Outputs sensitivity profiles and statistical analysis

Experiment 2: Inference-Time Accent Steering

Apply mean-shift steering vectors at inference time to reduce accent-induced WER.

Step 1: Generate data splits

# VCTK
python steering/generate_splits_vctk.py \
    --dataset data/vctk_locate_dataset.json \
    --vctk-root data/vctk \
    --output experiment_outcome/steering_vctk/splits.json

# L2-ARCTIC
python steering/generate_splits_l2arctic.py \
    --dataset data/l2arctic_locate_dataset.json \
    --output experiment_outcome/steering_l2arctic/splits.json

Step 2: Run the full leave-out pipeline

# Single-layer sweep (per-layer steering)
python steering/run_leaveout.py \
    --run-all --accent scottish

# All-layer simultaneous steering
python steering/run_leaveout.py \
    --run-all --accent scottish \
    --eval-script leaveout_eval_all_layer.py \
    --output-base experiment_outcome/steering_vctk/all_layer

This automatically: computes mean-shift vectors → runs per-layer or all-layer evaluation → reports ΔWER.

Or run individual steps:

# Compute mean-shift vectors
python steering/compute_mean_shift.py \
    --splits experiment_outcome/steering_vctk/splits.json \
    --dataset data/vctk_locate_dataset.json \
    --accent scottish \
    --split-name leave_both \
    --output-dir experiment_outcome/steering_vctk/single_sweep/scottish/leave_both

# Evaluate single-layer sweep
python steering/leaveout_eval_single_sweep.py \
    --splits experiment_outcome/steering_vctk/splits.json \
    --dataset data/vctk_locate_dataset.json \
    --accent scottish \
    --split-name leave_both \
    --split-dir experiment_outcome/steering_vctk/single_sweep/scottish/leave_both \
    --target-samples 200

Experiment 3: LoRA Fine-tuning Baseline

Fine-tune the audio encoder with LoRA for comparison.

Step 1: Prepare data splits (aligned with steering for fair comparison)

# For L2-ARCTIC
python finetune/prepare_steering_aligned_data.py \
    --dataset data/l2arctic_locate_dataset.json \
    --splits experiment_outcome/steering_l2arctic/splits.json \
    --output finetune/data/l2arctic_strict_aligned_splits.json \
    --accents arabic,hindi,spanish

# For VCTK
python finetune/prepare_steering_aligned_data.py \
    --dataset data/vctk_locate_dataset.json \
    --splits experiment_outcome/steering_vctk/splits.json \
    --output finetune/data/vctk_strict_aligned_splits.json \
    --accents scottish,southafrican,canadian,northernirish,irish

Step 2: Train

python finetune/train.py \
    --config finetune/config/l2arctic_config.yaml \
    --accent arabic

Step 3: Evaluate

# Standard evaluation
python finetune/evaluate.py \
    --config finetune/config/l2arctic_config.yaml \
    --accent arabic \
    --adapter_path path/to/checkpoint

# Evaluate on exact steering test set for fair comparison
python finetune/evaluate_strict_aligned_steering.py \
    --config finetune/config/l2arctic_config.yaml \
    --accent arabic \
    --adapter_path path/to/checkpoint

Method Summary

1. Accent Subspace Analysis

We construct text-matched utterance pairs (standard English vs. accented speech) and compute per-layer mean-shift directions:

$$\mathbf{d}_{s \to a}^{(l)} = \frac{1}{|G_s|}\sum_{j \in G_s} \bar{\mathbf{h}}_j^{(l)} - \frac{1}{|G_a|}\sum_{i \in G_a} \bar{\mathbf{h}}_i^{(l)}$$

The Accent Alignment Score (AAS) measures how injecting this direction at layer $l$ changes the cosine similarity between accented and standard representations in the projector space.

2. Inference-Time Steering

At inference, we inject the normalized mean-shift vector into the identified sensitive layers:

$$\tilde{\mathbf{H}}^{(l)} = \mathbf{H}^{(l)} + \alpha \cdot \hat{\mathbf{d}}_{s \to a}^{(l)}$$

where $\alpha$ controls steering strength. This is implemented via forward hooks — no model parameters are modified.

Findings

• Early layers (0–14): Low accent sensitivity; process low-level acoustic features
• Middle layers (15–19): Optimal steering window; consistent WER improvements across all accents
• Late layers (20–30): High sensitivity but poor controllability; risk of representation collapse
• Layer 31: Directly precedes the multi-modal projector; always unsuitable for injection

Supported Accents

Dataset	Accents	Reference Group
VCTK	Scottish, South African, Canadian, Irish, Northern Irish	Standard English
L2-ARCTIC	Arabic, Hindi, Spanish	CMU-ARCTIC (Standard American English)

Citation

@article{sun2026activation,
    title={Activation Steering for Accent Adaptation in Speech Foundation Models},
    author={Sun, Jinuo and Xiao, Yang and Chung, Sung Kyun and Hu, Qiuchi and Huang, Gongping and Holden, Eun-Jung and Dang, Ting},
    journal={arXiv preprint arXiv:2603.05813},
    year={2026}
}

License

This project is licensed under the MIT License. See LICENSE for details.

The datasets used in this work have their own licenses:

• VCTK Corpus: Creative Commons Attribution 4.0
• L2-ARCTIC: See the L2-ARCTIC website
• CMU-ARCTIC: See the CMU-ARCTIC page

Acknowledgements

This work utilizes the Qwen2-Audio-7B model. We thank the Qwen team for releasing the model weights.