Dev:SynthJS

From LQ's wiki
Revision as of 00:08, 12 February 2014 by Changtau2005 (Talk | contribs)

Jump to: navigation, search
Synthjs-scrshot-03.png
Icon-github.pnghttps://github.com/lqkhoo/SynthJS
Hosted deployment (coming soon)

SynthJS

I'll just stick the latest screenshot up here so it doesn't look too boring

SynthJS is an application to let users to quickly synthesize their favorite tune within the browser with little to no musical knowledge, and let them turn their new creation into a rhythm game. I want to build something that people can both use to do something worthwhile and to play with (if they want to), and the interesting thing about it is they get to play with their own creation; the application is a constant but their experience is as individual as they want to make it.

This is still very early in development - most of the features and parts of the GUI haven't even been implemented yet!

I've had this idea of a music synthesizer / rhythm game for a while, but I wasn't confident enough in execution (JavaScript, mainly) to start it yet, plus I'd had thought there would be something like this out there already. There's tons of HTML5 audio projects out there - this article lists more than a few, but I couldn't find anything like what I wanted to make (SynthJS). So, I thought I might as well start now and throw myself into the deep end and see how far I can take the idea and hopefully learn a few things along the way.

Currently it's built on top of libraries like Timbre.js and Music.js, with Backbone.js for models and jQuery for the DOM, but this may well change since this is very early in its development. This is especially the case for the music libraries as they're all fairly new.

Roadmap

The first phase is to get the synthesizer to a reasonable state as that's the original core idea. If I don't run into performance or programming brick walls then I'll start looking at serializing the data (should be straightforward) and making a compatible rhythm game out of it. Getting SynthJS to output audio files is possible (there's this project doing it already), but not a priority.

Synthesizer / sequencer | SynthJS - Music made quick

  • Allow add instruments
  • Implement volume controls, instrument deletion
  • Add additional instrument sounds
  • Add note highlighting based on selected scale
  • Add sound preview for instrument
  • Implement undo stack + associated command classes
  • Add waveform editor (ADSR envelope only)
  • Add serialize-to-JSON capability
  • Add load-from-JSON

Other optional but useful requirements I can think of I'll add below:

  • Double / halve beats per bar
  • Copy squares / rows / columns of grid
  • Have a way of specifying play duration per note. Currently they're all played staccato

Game | SynthJS - Music played quick

  • We'll see


Similar projects

  • Theres loads of HTML audio stuff going on. I'll list the interesting ones here when I get time.

Design decisions / considerations

The application is built upon several core models:

Orchestra The top level singleton model handling all the playback and containing all other models concerned with music-playing
Instrument The model for an instrument. Contains a number of Beat instances as well as other parameters
Beat The model to handle each time value. Contains an array of bools for each pitch (88 of them), as well as other parameters

Design consideration 01 - The grid

After I got the grid functionally working, it's time to optimize its performance. Initially I misidentified the cause of the performance issues as my models, but it turns out it's going to be the DOM that's causing the most problems. You see, the application is essentially a giant grid.

  • Let m be the number of instruments
  • Let n be the number of beats in the piece of music.

The grid is 88 columns wide (there are 88 keys on a piano), n rows down, and m instruments (layers) deep. We're looking at 88 * n * m divs minimum if we generate the grid naively, which is what the implementation was doing.

The implementation at this time of writing is not using HTML5 web workers to generate the audio output (if it comes to that because of performance issues later then I'll do it). Why did I initially guess the performance bottleneck to be with my models? When the user hits play, the application scans all 88 frequencies per instrument to decide what to play. It then generates Timbre.js objects on the fly, passes them through a Timbre.js Adder (These perform an arithmetic sum on the generated sound oscillators + envelopes).

That's the initial performance concern. I was tweaking the grid headers as well - initially they were generated along with the grid, so if the user scrolls away from the initial position, they become hidden. I thought of a 'floating' them (CSS sticky when it finally gets implemented) via JS but bearing in mind the event handlers in place, I wanted to just have one set of headers for all instrument layers, so I moved them out. After solving CSS problems (The top bar has to scroll with the grid but stay static), it looks all nice and works the same, but I still haven't solved the performance issues yet.

I had an idea of letting users limit the instruments' range to reduce the 88 * m * n space since an instrument would probably use less than 33% of the 88 frequencies, usually -- most HTML5 audio projects I've seen out there let the user play with between a dozen to two dozen frequencies at most -- and I'd just let this be set on individual instruments. All it needs is an additional field per instrument masking out those frequencies - during playback, before creating the Timbre objects, perform a boolean AND on the mask and the registered frequency to decide whether to play it or not. That's to target the potential bottleneck being generating those objects. Seems like a good direction to optimize (it really does shave off a lot of that space) but turns out this wasn't necessary.

There's another way, which is to cache which frequencies are actually active at any time and only scan those values -- that's if the bottleneck is fetching all those Backbone Beat instances and arrays, although I thought this is pretty unlikely to be the issue.

Halfway through solving the CSS issues, I decided to hide parts the grid to troubleshoot z-indices and layering problems, but I had some notes registered so I hit play anyway. Funnily enough, there were no problems playing about a hundred notes simultaneously. So I had a look at the grid and decided I might as well streamline it now since I pinpointed my bottleneck.

I made sure of one thing - I tried setting the elements to display: none to remove them from the DOM and stop the browser rendering them, and this doesn't cause any problems even when there's tons of them on the DOM, so this is going to be my approach - hide the squares until they become active. To remove the grid, first I redefined my CSS for the squares to be absolutely-positioned based on their frequency (They were floated left before -- yeah, I know). Since there are 88 frequencies max, I just created 88 CSS definitions.

I settled on an in-between event catcher layer. Initially I opted for a blank div to capture mouse click coordinates (like thus), and then map them down to the active instrument, but then I realized, without the squares, there's no grid!. So, I'm going to populate this in-between layer with two layers - one with 88 divs going down, one with m rows Backbone-mapped to the number of beats tracked by the Orchestra singleton. This way, I get a grid for 88 + n divs instead of 88 * m * n. I'll just have to define an opacity filter and/or color difference for non-active instruments to differentiate between active and inactive instruments. I checked for a way to trigger events on overlapping items so I don't have to calculate coordinates, but since these neither of these divs are parents nor children of each other, it's not possible to make use of event bubbling /propagation.

{Edit}: Actually, I just thought about the fact that I could get a grid by simply pre-rendering it as a single-square PNG and setting it to background-repeat! Do I do it? Nah, it's cheating =p Maybe at the end to eke out that last drop of performance.

Design consideration 02 - The music model

Up to this point the project is essentially a huge spike - a lot of the work is to discover performance bottlenecks and figuring out how to do things. Now, with most of those out of the way, it's time to tackle the main problem how to represent the music.

Right now the model is what I call the music box model. The music is divided into a number of beats, and an instrument either plays a particular note on that pulse or not - much like the rotating drum and pins of a music box. What it does not let me specify is how long the note needs to be played. While this is perfectly fine for percussive / plucked / struck instruments like the guitar or to a certain extent the piano, this will not do for woodwind / brass / strings / vocals at all.

Now, if I want any decent chance at making it into a rhythm game, or have a way of representing a good number of pieces reasonably accurately, this needs to change, and since this is the core of the application, the change needs to be now. It's stopping me from working on the actual sound-producing features like instruments and the envelope editor because if I were to change this later it would be throwing all that work away. Even now, it means rewriting the whole grid, so this is going to take a good few days.

Instead of an array of Beats, Instruments are going to hold a fixed 88 instances of Backbone collections of Pitch(es), and each one of these would have any number of Note(s). These notes would have their id property as their starting time, and a duration property in terms of the number of smallest time intervals. Model-wise this needs another element to work - a boolean array specifying which time intervals are open for that frequency - you can't play the same note if again if the previous one is still sounding. If it's closed then the proper Note to edit can be found by walking backwards until the first Note is reached - this is the note which is playing more than one unit of time. Presentation-wise Backbone needs to sync absolutely positioned divs - their left property, and their width property. It's the same when deleting time intervals. However, if the root note is deleted, then the array needs to open up the time intervals. When playing, the loop gets all 88 collections and tries to fetch the notes by the id of the current time interval. If there's no note, that frequency is simply not played.

While I'm at this I'll take the chance to transpose the grid - time goes right, pitch goes down, and I'll add a checkbox for each frequency to let users disable them, and these objects are not scanned during play - it's very unlikely for an instrument to play more than 3 - 4 octaves.

That's the current plan. Will update if anything changes or if I overlooked anything.

{Edit} Not transposing grid. It's less intuitive and less aesthetically pleasing.

Diary

End of Jan - Feb 3

Screenshot

I'm writing this as a back post since I'm late like that (actually I spent most of the day trying to deploy Jenkins onto my Ubuntu instance on Amazon EC2 but the packages don't want to work, so I decided to do this instead and forgo CI / automatic deployment for the time being as I don't see a straightforward solution).

The project started before this but it was as a local hard drive copy. Ha. These few days were mostly initial project setup and the like. I looked through audio libraries to use and settled on Timbre.js, and used Music.js to generate the proper frequencies for the notes easily. Music.js is using the circle of fifths to do this - this would make sense to people familiar with music theory. Got started on the initial page layouts and used Backbone models. Main initial goal is to get the note grid functional since, you know, having it look real nice but only stare at you and not work makes one crazy. Okay, maybe just me.

The basic layout of the application is thus: There are a series of menus and elements, all elementId labeled for fast DOM lookup, and all of them hug that grid in the middle which is the centerstage. The user picks which note he/she wants to play at what time, and clicks on the appropriate square to register it. When he hits play, then the application plays it back.

Initially this grid is generated naively - one div per square, since I'm still having a refresher on Backbone.js etc. and I wasn't that concerned about performance considerations at that point although I'm aware that it wasn't performing as well as I'd like it to.

So, summary time, functionality-wise, up to Feb 3, I got the playback system working with rudimentary add/remove beats from the grid, and actually getting the grid to work.

Feb 4

Most of today is concerned with addressing performance and usability concerns. The easiest thing done today is getting Orchestra to automatically select an instrument when it has none whenever a new one is added. That took under 2 minutes. Done, boom, yay, whatever. The rest took a lot longer. Primarily this concerns the grid behavior. This is important enough to be covered at the top. See #Design consideration 01 - The grid. Also, Jenkins is feeling uncooperative and I ended up throwing away several hours because the package for Ubuntu (actually they have Debian/Redhat only) refuses to deploy. Has something to do with Winstone. Anyways I spent enough time on Google and troubleshooting already so I decided to write this up. Most of today's work is covered in the given link.

Feb 5

Some UI improvements

Spent most of today implementing the planned capture layer and DOM optimizations. It was way more troublesome than I thought it would be, but the end result is more than worth it - everything is now buttery smooth. No jitters or delays whatsoever when doing most things I could think of, other than saturating the whole screen of multiple instruments with notes. I didn't know event.offset was optional -- firefox doesn't have it, so I had to use a fallback called event.originalEvent.layerX/Y. When I was doing all that, Opera started refusing to work for whatever reason. I spent a while trying to fix it, but seeing that Timbre.js uses Webkit/Firefox's audio API, I decided to leave it for the time being. But guess what, after everything's finished and the grid's polished up, Opera decided to work with the implementation again. Oh well, guess it doesn't like being left out =) Sure, it still doesn't play anything but I like to get everything in working order in case I want to port to the HTML5 audio framework later on -- I try to maintain quite a clean line of separation between my models and the audio generation suite.

The biggest problem was getting the listeners for each beat to rebind properly when users delete beats -- it took me several hours to figure that out but I've finally solved it. For all that work, the performance improvement is worth it.

Feb 6

Midway through refactor

Had to take time out for a university research project of mine involving clustering queries in search engine logs. Anyway, regarding SynthJS, I looked at Grunt and other JS and standard build tools (Ant, Maven, Yeoman, etc.) to see if they had a one-click minify-concat etc. build option using a config file from within Eclipse, but came away disappointed that there isn't (or at least I couldn't make one work after an afternoon). At least I finally figured out how to make Eclipse outline the code correctly from a Stack Overflow post. Apparently the outermost anonymous function cannot self-execute. One must declare a {} prototype and execute the outermost function on a separate line -.-' mm.. Ok.., and apparently it doesn't work in Kepler, but I'm on Eclipse Juno.

I should probably start testing my models in QUnit before there's too many, although I have little to no idea about how to test the event bindings declared in Backbone Views. Actual work done, I whipped up a logo in CSS3 which should serve its purpose well enough for now.

Feb 7

Got the undo/redo stack done. There's a more pressing concern, however as the project is at a crossroads of sorts. See #Design consideration 02 - The music model

Feb 8

Midway through refactor

Everything's broken to bits since I'm halfway through refactoring as per yesterday's plan, but the models and views are shaping up nicely. I've modified the logo a bit as the relative positioning is preventing elements from being clicked on. Test-wise I've written unit tests for half the classes. Hopefully most of this'll be done by tomorrow or the day after. I've decided to not transpose the grid, since once I did the layout is much less intuitive - at least to me as a piano player. The main motivation for flipping it is so the user has more space to work with, but it's not worth the tradeoff as far as it stands now.

Oh yeah, almost forgot. I tried seeing if Epoxy.js is a better alternative than ModelBinder - and got mixed results. Mainly I did this because of the irksome problem in that ModelBinder requires an extra element when binding collections - you can't target whatever you insert as a template - its direct descendant is the highest level you can bind to. This means whatever I insert in the grid would have a dangling div somewhere. Hidden away, of course, but still, it's on the DOM. In a way, Epoxy's syntax is more terse binding models to views, but its main drawback is that if you want to bind to collections, it needs a one to one relationship between the collection and the view. This makes it an immediate no-go, as even just for the instruments, I need to bind it to both the grid and the instrument panel. Hence, back to ModelBinder it is.

Feb 9

Got the grid display back up, but event bindings are not fully done yet, specifically those concerning playback and the note grid. First I have to get the pitch-based model right. Spent some time today tidying up the files and spiking a Java project to minify + concat, using the Closure compiler and YUI css minifier components. Looks promising enough, but I didn't proceed further as this is more important. Looks like the reconstruction's going to take at least one more day, but I have to take time out tomorrow to get some work done on my research project, so we'll see.

Feb 10

Made some good progress on my research project. On this one, in what time I have left I searched around for good alternatives to Timbre.js (not that Timbre is bad - it's on the contrary, really), but I just wanted to have another look around to see what's out there. Found MIDI.js with a damn impressive demo, and a good quality soundfont library already encoded as a JSON base64 arrays that I'll try out tomorrow. There's so much potential for this tech. Still have to get that grid back up first though. Grah.

Feb 11

Grid note ops and handlers back up, on the new model

Got the note operations (add / delete / edit) on the grid fully working, all as invocations and with logical reverses (they can be fully undone/redone using the established stacks with no limitations). Had to fix a couple of bugs in the end, visual and otherwise, but it's done. Next stage is to get the grid ops fully working (add / delete rows), and getting the instrument models fully finished. Currently the instruments are merely skeletons of values without a sound generator unit, and I still need to give them each a color selector - right now their notes all default to fuschia / maroon. Overall, very satisfied with today's progress. Basic playback implementation in MIDI.js should be able to be completed within the week, next by the very latest if I decide to prioritize other things or if some other project comes up on my radar.