Motivation

BRSTM

• File format
• Audio data, with loop point!
• Used in Nintendo Wii
• Popular in Nintendo modding community

Why recreate it on web?

How to play audio on web?

<audio />

HTML's <audio /> cannot play BRSTM!

Let's start an audio player project

But first, we need a name

Neeku Vozo

Nikku

Idea (1)

ffmpeg

Big idea

1. ffmpeg converts BRSTM to MP3
2. Play using <audio src="music.mp3" />

Solved?

No, it's not able to loop

Idea (2)

Live stream

Big idea

• Server serves the latest audio chunk continuously
• Client listens like listening online radio

Live streaming protocols

• HLS: HTTP Live Streaming (old Safari/Edge)
• DASH: Dynamic Adaptive Streaming over HTTP (all other browsers)

The idea

• Server implements DASH protocol
• Client listens using DASH client library

Is the complexity getting out of hand?

What's the user flow like?

Two big ideas

Two main problems

• How to decode BRSTM
• How to play audio using Web Audio API

<crash-course>

in Digital Audio

Analog to Digital

Pulse-code modulation (PCM)

• Analog sound wave
• Sampling
• Quantization
• Bit depth
• Sample rate

Audio channels

4-minute track from a CD 💿

• Sample rate: 44100 per second
• Channels: 2
• Bit depth: 16 bits per channel
• Length: 240 seconds
• 44100 /s × 2 × 16 bits × 240 s
  = 338,688,000‬ bits! ≈ 42.336 MB

Why your 4-minute audio file does not take up 42 MB of space?

Because compression!

Compression

• Coding format: MP3, AAC, FLAC
• Codec: encoder & decoder program
• Container format: .mp3, .m4a, .flac

</crash-course>

Extracting BRSTM

For real

• WiiBrew
• Custom Mario Kart Wii Wiki

Overview

• Container Format: BRSTM
• Coding Format: PCM or ADPCM
• Other metdata specified in file

So I started a simple page that accepts a BRSTM file and read its metadata

It works. What's next?

Audio Player

Web Audio API

W3C - Web Audio API specs

Buffer Source node → Audio Context destination node

                const audioContext = new AudioContext();
                const bufferSource = audioContext.createBufferSource();
                bufferSource.connect(audioContext.destination);
            

Get PCM samples from BRSTM

              const pcm = brstm.getAllSamples(); // TODO: Implement
            

Transform PCM into audio buffer

              const audioBuffer = audioContext.createBuffer(
                1, // number of channels
                pcm.length,
                metadata.sampleRate
              );
              audioBuffer.getChannelData(0).set(pcm);
            

Write into source node, and plays it

              bufferSource.buffer = audioBuffer;
              bufferSource.start(0); // start immediately
            

W3C - Web Audio API specs

Decoding ADPCM

This is the coding format

There should be a NPM package for this

😱

and the input/output looks as expected!

🕵️‍

ADPCM

MultimediaWiki

Decoding Nintendo ADPCM

That means I can't use that npm package

How did they do it?

How did they do it?

Attach debugger!

Eureka!

Is it finished?

No

Audio Player

Web Audio API Gotchas

Audio Player

• Play/pause
• Display current playback time
• Seek
• Loop

Looping

Easy

              // audioBuffer contains raw audio data (PCM)
              audioBuffer.loop = true
              audioBuffer.loopStart = … // in seconds
              audioBuffer.loopEnd = … // in seconds
            

Play/pause

Easy

              audioContext.suspend(); // pause
              audioContext.resume(); // resume
            

Seek

              // in <audio />
              audioElement.time = … // in seconds
            

Seek (in Web Audio)

• Destroy audio buffer
• Recreate the whole audio node connect thing
• Start playing at certain time

  
                      bufferSource.start(0, soughtTimeInSeconds);
                    

Playback time display

Impossible

              audioContext.currentTime
            

Playback time display (workaround)

Reimplement timer

Finally

Marketing

Lessons