Dev:SynthJS

From LQ's wiki
Revision as of 22:40, 5 February 2014 by Changtau2005 (Talk | contribs)

Jump to: navigation, search
Icon-github.pnghttps://github.com/lqkhoo/SynthJS
Hosted deployment (coming soon)
Immediate development goals
  • Photoshop a logo - SynthJS - Music made quick
  • Tabify top menus
  • Get events capture layer working
  • Enable add instrument

SynthJS

I'll just stick the latest screenshot up here so it doesn't look too boring

SynthJS is an application to let users to quickly synthesize their favorite tune within the browser with little to no musical knowledge, and let them turn their new creation into a rhythm game. I want to build something that people can both use to do something worthwhile and to play with (if they want to), and the interesting thing about it is they get to play with their own creation; the application is a constant but their experience is as individual as they want to make it.

This is still very early in development - most of the features and parts of the GUI haven't even been implemented yet!

I've had this idea of a music synthesizer / rhythm game for a while, but I wasn't confident enough in execution (JavaScript, mainly) to start it yet, plus I'd had thought there would be something like this out there already. There's tons of HTML5 audio projects out there - this article lists more than a few, but I couldn't find anything like what I wanted to make (SynthJS). So, I thought I might as well start now and throw myself into the deep end and see how far I can take the idea and hopefully learn a few things along the way.

Currently it's built on top of libraries like Timbre.js and Music.js, with Backbone.js for models and jQuery for the DOM, but this may well change since this is very early in its development. This is especially the case for the music libraries as they're all fairly new.

Roadmap

The first phase is to get the synthesizer to a reasonable state as that's the original core idea. If I don't run into performance or programming brick walls then I'll start looking at serializing the data (should be straightforward) and making a compatible rhythm game out of it. Getting SynthJS to output audio files is possible (there's this project doing it already), but not a priority.

Synthesizer / sequencer | SynthJS - Music made quick

  • Allow add instruments
  • Implement volume controls, instrument deletion
  • Add additional instrument sounds
  • Add note highlighting based on selected scale
  • Add sound preview for instrument
  • Implement undo stack + associated command classes
  • Add waveform editor (ADSR envelope only)
  • Add serialize-to-JSON capability
  • Add load-from-JSON

Game | SynthJS - Music played quick

  • We'll see


Similar projects

  • Theres loads of HTML audio stuff going on. I'll list the interesting ones here when I get time.

Design decisions / considerations

The application is built upon several core models:

Orchestra The top level singleton model handling all the playback and containing all other models concerned with music-playing
Instrument The model for an instrument. Contains a number of Beat instances as well as other parameters
Beat The model to handle each time value. Contains an array of bools for each pitch (88 of them), as well as other parameters

Design consideration 01 - The grid

After I got the grid functionally working, it's time to optimize its performance. Initially I misidentified the cause of the performance issues as my models, but it turns out it's going to be the DOM that's causing the most problems. You see, the application is essentially a giant grid.

  • Let m be the number of instruments
  • Let n be the number of beats in the piece of music.

The grid is 88 columns wide (there are 88 keys on a piano), n rows down, and m instruments (layers) deep. We're looking at 88 * n * m divs minimum if we generate the grid naively, which is what the implementation was doing.

The implementation at this time of writing is not using HTML5 web workers to generate the audio output (if it comes to that because of performance issues later then I'll do it). Why did I initially guess the performance bottleneck to be with my models? When the user hits play, the application scans all 88 frequencies per instrument to decide what to play. It then generates Timbre.js objects on the fly, passes them through a Timbre.js Adder (These perform an arithmetic sum on the generated sound oscillators + envelopes).

That's the initial performance concern. I was tweaking the grid headers as well - initially they were generated along with the grid, so if the user scrolls away from the initial position, they become hidden. I thought of a 'floating' them (CSS sticky when it finally gets implemented) via JS but bearing in mind the event handlers in place, I wanted to just have one set of headers for all instrument layers, so I moved them out. After solving CSS problems (The top bar has to scroll with the grid but stay static), it looks all nice and works the same, but I still haven't solved the performance issues yet.

I had an idea of letting users limit the instruments' range to reduce the 88 * m * n space since an instrument would probably use less than 33% of the 88 frequencies, usually -- most HTML5 audio projects I've seen out there let the user play with between a dozen to two dozen frequencies at most -- and I'd just let this be set on individual instruments. All it needs is an additional field per instrument masking out those frequencies - during playback, before creating the Timbre objects, perform a boolean AND on the mask and the registered frequency to decide whether to play it or not. That's to target the potential bottleneck being generating those objects. Seems like a good direction to optimize (it really does shave off a lot of that space) but turns out this wasn't necessary.

There's another way, which is to cache which frequencies are actually active at any time and only scan those values -- that's if the bottleneck is fetching all those Backbone Beat instances and arrays, although I thought this is pretty unlikely to be the issue.

Halfway through solving the CSS issues, I decided to hide parts the grid to troubleshoot z-indices and layering problems, but I had some notes registered so I hit play anyway. Funnily enough, there were no problems playing about a hundred notes simultaneously. So I had a look at the grid and decided I might as well streamline it now since I pinpointed my bottleneck.

I made sure of one thing - I tried setting the elements to display: none to remove them from the DOM and stop the browser rendering them, and this doesn't cause any problems even when there's tons of them on the DOM, so this is going to be my approach - hide the squares until they become active. To remove the grid, first I redefined my CSS for the squares to be absolutely-positioned based on their frequency (They were floated left before -- yeah, I know). Since there are 88 frequencies max, I just created 88 CSS definitions.

I settled on an in-between event catcher layer. Initially I opted for a blank div to capture mouse click coordinates (like thus), and then map them down to the active instrument, but then I realized, without the squares, there's no grid!. So, I'm going to populate this in-between layer with two layers - one with 88 divs going down, one with m rows Backbone-mapped to the number of beats tracked by the Orchestra singleton. This way, I get a grid for 88 + n divs instead of 88 * m * n. I'll just have to define an opacity filter and/or color difference for non-active instruments to differentiate between active and inactive instruments. I checked for a way to trigger events on overlapping items so I don't have to calculate coordinates, but since these neither of these divs are parents nor children of each other, it's not possible to make use of event bubbling /propagation.

{Edit}: Actually, I just thought about the fact that I could get a grid by simply pre-rendering it as a single-square PNG and setting it to background-repeat! Do I do it? Nah, it's cheating =p Maybe at the end to eke out that last drop of performance.

Diary

Feb 1 - 3

Screenshot

I'm writing this as a back post since I'm late like that (actually I spent most of the day trying to deploy Jenkins onto my Ubuntu instance on Amazon EC2 but the packages don't want to work, so I decided to do this instead and forgo CI / automatic deployment for the time being as I don't see a straightforward solution).

The project started before this but it was as a local hard drive copy. Ha. These few days were mostly initial project setup and the like. I looked through audio libraries to use and settled on Timbre.js, and used Music.js to generate the proper frequencies for the notes easily. Music.js is using the circle of fifths to do this - this would make sense to people familiar with music theory. Got started on the initial page layouts and used Backbone models. Main initial goal is to get the note grid functional since, you know, having it look real nice but only stare at you and not work makes one crazy. Okay, maybe just me.

The basic layout of the application is thus: There are a series of menus and elements, all elementId labeled for fast DOM lookup, and all of them hug that grid in the middle which is the centerstage. The user picks which note he/she wants to play at what time, and clicks on the appropriate square to register it. When he hits play, then the application plays it back.

Initially this grid is generated naively - one div per square, since I'm still having a refresher on Backbone.js etc. and I wasn't that concerned about performance considerations at that point although I'm aware that it wasn't performing as well as I'd like it to.

So, summary time, functionality-wise, up to Feb 3, I got the playback system working with rudimentary add/remove beats from the grid, and actually getting the grid to work.

Feb 4

Most of today is concerned with addressing performance and usability concerns. The easiest thing done today is getting Orchestra to automatically select an instrument when it has none whenever a new one is added. That took under 2 minutes. Done, boom, yay, whatever. The rest took a lot longer. Primarily this concerns the grid behavior. This is important enough to be covered at the top. See #Design consideration 01 - The grid. Also, Jenkins is feeling uncooperative and I ended up throwing away several hours because the package for Ubuntu (actually they have Debian/Redhat only) refuses to deploy. Has something to do with Winstone. Anyways I spent enough time on Google and troubleshooting already so I decided to write this up. Most of today's work is covered in the given link.

Feb 5

Some UI improvements

Spent most of today implementing the planned capture layer and DOM optimizations. It was way more troublesome than I thought it would be, but the end result is more than worth it - everything is now buttery smooth. No jitters or delays whatsoever when doing most things I could think of, other than saturating the whole screen of multiple instruments with notes. I didn't know event.offset was optional -- firefox doesn't have it, so I had to use a fallback called event.originalEvent.layerX/Y. When I was doing all that, Opera started refusing to work for whatever reason. I spent a while trying to fix it, but seeing that Timbre.js uses Webkit/Firefox's audio API, I decided to leave it for the time being. But guess what, after everything's finished and the grid's polished up, Opera decided to work with the implementation again. Oh well, guess it doesn't like being left out =) Sure, it still doesn't play anything but I like to get everything in working order in case I want to port to the HTML5 audio framework later on -- I try to maintain quite a clean line of separation between my models and the audio generation suite.

The biggest problem was getting the listeners for each beat to rebind properly when users delete beats -- it took me several hours to figure that out but I've finally solved it. For all that work, the performance improvement is worth it.