Soundscape was born of a passion for music visualizers and browser APIs, combined with a desire to experiment with new forms of interactive music production.

The app relies on THREE.js, React, and Anime.js to bring it to life. An earlier version of the app also used Tone.js as a wrapper for the WebAudio API, but I found the library to be over-engineered and ultimately went with a slimmer, custom-tailored approach to leveraging the capabilities of WebAudio. I wrote and recorded all of the music in Ableton Live using my Yamaha P-125 and various software synths and samples that I've collected over the years.

Screenshots

"Moonrise" was the first experience I added to Soundscape. This was before I learned Blender, so all of the 3D models were created algorithmically

After "Moonrise" I wanted to go with something with a lighter theme, so I wrote "Mornings". This was my first foray into 3D modeling. It turned out alright, but it was tough to learn.

"Swamp" was by far the most complex experience to build. I worked with Alana Guidry on the 3D modeling this time around, which freed me up to focus on the animation and music.

Learnings

Here are a few of my key learnings since the inception of the project.

Software

1. The end user's audio clock can be leveraged to schedule events with far greater precision than any other technique. By using the onended callback of the AudioScheduledSourceNode, events can be scheduled with sample-level (e.g. 1/44100s) precision, which is crucial for audio sequencing.

2. Initially, I went for a tightly coupled approach to integrating 3rd party libraries with React. That was a mistake. Constant re-renders put a strain on browser resources when they also trigger expensive graphic operations or the creation of audio nodes as side effects. I learned to remove extraneous logic from the React data flow and to try to limit React to managing UI state only. Only then did I begin to optimize with memoization.

3. I learned not to initialize new functions or arrays in render loops. Reusing arrays is the way to go.

4. I removed references to unused objects to allow them to be garbage collected. Also, I learned that the contents of an array can be removed from memory using array.length = 0.

5. I always wondered how Netlify made money until I got smacked in the face with a $60 charge for bandwidth usage. And that's how I learned not to serve audio files or 3D models from Netlify, instead opting to move them to an S3 bucket behind Cloudfront (see next).

6. Instead of grabbing assets directly from S3, I set up a CloudFront CDN, which significantly improved load times, especially in international regions. I'm a big fan of CloudFront, and look for opportunities to use it whenever I can. The ability to use a CDN like CloudFront is the main reason why I avoid SSR solutions when possible. Flat files, JS, and static assets in a bucket...so clean.

Audio

Musically, I learned a lot with this.

1. I learned not to deviate from one or two chords when composing free-form looping music. When you cede control of the music over to the user, there becomes a tradeoff between freedom and simplicity. The freer the user is to control the sequence of sounds, the simpler they must be. This led me to consider modes as the primary basis for songs over chord progressions (and I mean yes, I've heard Kind of Blue, but sometimes you have to see it to believe it).

2. Each take that I included in the final project had to be really good, which meant that I needed to play the same thing over and over again so that I could have a good selection of takes to choose from. I noticed that the first few times playing a sequence, I would feel a little tight, but gradually I would get "lost" and stop thinking about playing it perfectly. That's when my playing began to feel more loose, and as a consequence, I was able to play slightly off the metronome and with more expression. Those were the best takes.

3. I learned how powerful a theme can be for composition. For example, knowing that I wanted to create a world called "Swamp" allowed me to hone in on a specific sound. A lot of electronic music is abstract. It may have a theme, but it's usually not so literal. As a result, I also gained a new appreciation for video game soundtracks, which have their theme and settings front and center.