On‑Device Real‑Time Audio Source Separation in the Browser with JavaScript

Why Separate Audio Sources Directly in the Browser?

Imagine a music‑learning app that can isolate a vocalist from a full‑band track without sending any data to a remote server. The user uploads a song, presses a button, and instantly hears the vocal track alone. This experience is possible today because modern browsers expose low‑level graphics and audio APIs that run entirely on the client device.

Recent benchmarks from Mozilla show that WebGPU can deliver more than 30 TFLOPs on high‑end GPUs, while the WebAudio API already processes 48 kHz streams with sub‑millisecond latency. Combining these capabilities with a neural separator such as Demucs creates a feasible pipeline for real‑time, on‑device source separation.

Architecture Overview

The core pipeline consists of four stages:

1. Load an audio file into an AudioBuffer using the WebAudio API.
2. Transform the time‑domain signal into a spectrogram (STFT) on the GPU via WebGPU.
3. Run the Demucs model on the spectrogram to predict source masks.
4. Apply the masks, inverse‑transform the result, and play each source through separate AudioNode chains.

All stages stay in memory; no network request is required after the initial script load. The heavy lifting—matrix multiplications and convolutional layers—happens on the GPU, while the audio routing remains in the WebAudio graph.

Setting Up WebAudio for Real‑Time Playback

First, create an AudioContext and a MediaStreamAudioDestinationNode for each source you want to output. The following snippet demonstrates a minimal setup for vocals and accompaniment:

const audioCtx = new (window.AudioContext || window.webkitAudioContext)();
const vocalGain = audioCtx.createGain();
const accompanimentGain = audioCtx.createGain();

vocalGain.connect(audioCtx.destination);
accompanimentGain.connect(audioCtx.destination);

// Later we will feed Float32Array buffers into these gains via AudioBufferSourceNode.

When a new audio buffer arrives, instantiate an AudioBufferSourceNode, set its buffer, and connect it to the appropriate gain node. The start() method respects the current currentTime, ensuring tight synchronization between multiple sources.

Leveraging WebGPU to Run Demucs

Demucs is a convolutional encoder‑decoder architecture originally written in PyTorch. To run it in the browser, we export the model to ONNX and then to a WebGPU‑compatible format using tfjs‑converter. The resulting .wgsl shaders perform the convolution and activation steps directly on the GPU.

Below is a concise example that creates a GPU buffer, uploads a spectrogram, and dispatches the Demucs compute pipeline:

async function runDemucs(spectrogram) {
  const adapter = await navigator.gpu.requestAdapter();
  const device = await adapter.requestDevice();
  const shaderModule = device.createShaderModule({code: demucsWGSL});
  const pipeline = device.createComputePipeline({compute: {module: shaderModule, entryPoint: "main"}});

  const inputBuffer = device.createBuffer({
    size: spectrogram.byteLength,
    usage: GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST,
  });
  device.queue.writeBuffer(inputBuffer, 0, spectrogram);

  const outputBuffer = device.createBuffer({
    size: spectrogram.byteLength,
    usage: GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_SRC,
  });

  const bindGroup = device.createBindGroup({
    layout: pipeline.getBindGroupLayout(0),
    entries: [
      {binding: 0, resource: {buffer: inputBuffer}},
      {binding: 1, resource: {buffer: outputBuffer}},
    ],
  });

  const commandEncoder = device.createCommandEncoder();
  const passEncoder = commandEncoder.beginComputePass();
  passEncoder.setPipeline(pipeline);
  passEncoder.setBindGroup(0, bindGroup);
  passEncoder.dispatchWorkgroups(Math.ceil(spectrogram.length / 256));
  passEncoder.end();
  device.queue.submit([commandEncoder.finish()]);

  // Read back the masked spectrogram
  const readBuffer = device.createBuffer({
    size: spectrogram.byteLength,
    usage: GPUBufferUsage.COPY_DST | GPUBufferUsage.MAP_READ,
  });
  const copyEncoder = device.createCommandEncoder();
  copyEncoder.copyBufferToBuffer(outputBuffer, 0, readBuffer, 0, spectrogram.byteLength);
  device.queue.submit([copyEncoder.finish()]);
  await readBuffer.mapAsync(GPUMapMode.READ);
  return readBuffer.getMappedRange();
}

Because the GPU operates asynchronously, you should pipeline the audio frames: while one frame is being processed, the next frame is already queued for STFT. This overlap reduces end‑to‑end latency to under 100 ms on a typical laptop GPU.

Real‑Time Processing Loop

The main loop pulls chunks from the AudioBuffer, converts them to frequency domain, runs Demucs, and writes the inverse‑transformed audio back to the WebAudio graph. A simplified version looks like this:

const CHUNK_SIZE = 16384; // ~371 ms at 44.1 kHz
let readPos = 0;
function processChunk() {
  if (readPos + CHUNK_SIZE > audioBuffer.length) return;
  const chunk = audioBuffer.getChannelData(0).subarray(readPos, readPos + CHUNK_SIZE);
  const spectrogram = stft(chunk); // custom STFT on CPU or GPU
  runDemucs(spectrogram).then(masked => {
    const separated = istft(masked);
    const sourceNode = audioCtx.createBufferSource();
    const buf = audioCtx.createBuffer(1, separated.length, audioCtx.sampleRate);
    buf.copyToChannel(separated, 0);
    sourceNode.buffer = buf;
    sourceNode.connect(vocalGain); // or accompanimentGain based on mask
    sourceNode.start();
    readPos += CHUNK_SIZE;
    requestAnimationFrame(processChunk);
  });
}
processChunk();

Using requestAnimationFrame keeps the UI responsive, and the incremental readPos guarantees that playback stays in sync with the original timeline.

Practical Tips and Performance Tweaks

1. **GPU Memory Management** – Reuse buffers instead of allocating new ones each frame. Allocate a pool of GPUBuffer objects at startup and rotate them.

2. **Quantization** – Converting the Demucs weights to 8‑bit integers reduces memory bandwidth by 75 % with less than 1 dB SNR loss. The tfjs‑quantization tool automates this step.

3. **Sample Rate Matching** – Most pre‑trained Demucs models expect 44.1 kHz input. If the source is 48 kHz, downsample with AudioContext.createBiquadFilter before STFT to avoid aliasing.

4. **Fallback Path** – Not all browsers support WebGPU yet. Detect support with if (!('gpu' in navigator)) and fall back to a WebAssembly‑based Demucs implementation, which still runs faster than pure JavaScript.

Conclusion

Real‑time, on‑device audio source separation is no longer a research prototype; it is a practical feature you can embed in any JavaScript‑driven web app. By orchestrating WebAudio for low‑latency routing, WebGPU for heavyweight tensor math, and a converted Demucs model for accurate mask prediction, developers can deliver studio‑grade vocal isolation, drum removal, or instrument extraction directly in the browser. The key is to keep the pipeline streaming, reuse GPU resources, and provide a graceful fallback for browsers that lack WebGPU.

When you combine these building blocks, the user experience becomes instant, private, and platform‑agnostic—exactly the kind of interaction modern web applications strive for.

Sources

Mozilla WebGPU Documentation; TensorFlow.js Model Converter Guide; Demucs Official GitHub Repository

Author: Mahmut Sarıkaya — sarikayadev.com