Audiostream Mixer

This project is work in progress.

When the DAF - Dynamisch Akustische Forschung searched for a way to organize an algorave over the internet during the pandemic, I started to work on an audiostream mixer that would allow multiple people to stream high quality audio with low latency over the internet, directly in the browser.

There would be multiple senders, people who would code music on their machine and send it via a WebRTC audio stream. One (or more people) would act as a mixer, mixing all incoming streams and sending one final mix to every sender. Every sender could listen to the final mix and their own audiostream at the same time.

There is some delay between the audio streams because internet, so this would not be suitable for performing traditional music where synchronicity really matters. But for the type of music the DAF produces at their algorave it would work.

Here are some screenshots of the development version:

a basic first prototype, using HTML audio elements for playback#

the first version using the web audio api; there was no sender/mixer concept yet, everyone would receive all audio streams and could do their own mix#

the start screen with lots of options so I could test which combination works best for music streaming#

the sender just sees their own stream and the final mix#

the mixer can manipulate and mix all streams; they also send out the final mix to all senders#

I started by using this WebRTC example by anoek. It provides basic functionality to send audio streams over WebRTC and play them via the HTML audio element. I then replaced the audio elements with the web audio api and custom designed controls so that I could add high/mid/low sliders and other things.

To get better audio quality, I disabled some speech enhancements that getUserMedia automatically adds (which make sense for speech streaming, but reduces audio quality for music):

navigator.mediaDevices.getUserMedia({
    audio: {
        autoGainControl: false,
        echoCancellation: false,
        noiseSuppression: false,
        volume: 1.0,
        # ...
    },
    video: false
})...

I also experimented with playoutDelayHint and jitterBufferDelayHint (both Chrome only) in WebRTC audio streams to maybe add some buffer to the audio stream and get better streaming quality (with the drawback of more delay), but that did not work. I also tried to use TCP instead of UDP to eliminate packet loss but couldn't get WebRTC to only use TCP connections.

It is also possible to set the maxaveragebitrate in the SDP description to a higher value:

let answer = await peer.conn.createAnswer(offerOptions);
answer.sdp = answer.sdp.replace('useinbandfec=1', 'useinbandfec=1; stereo=1; maxaveragebitrate=510000');
await peer.conn.setLocalDescription(answer);

and manually set the audio codec to Opus (which should yield the highest audio quality while maintaining a relatively low latency).

In the end, I couldn't get the audio quality high enough for this to really work, so the DAF ended up using audiomovers in their live performance (which you can watch on the BR KulturBühne).

Nevertheless, I learned a lot about the Web Audio API and WebRTC. I have plans to try and use RTCDataChannels for the audio streams so I can build and control my own audio buffer. There are also some proposals for WebRTC that would allow the control of the audio buffer so one could balance audio quality and latency but it may take some time for this to get implemented in the browsers.