Jukebox: here
Added a basic equalizer to the Jukebox. Not 100% sure that it's doing everything correctly, but it's much too late to go back and double check.
Taking a small break from audio. I will come back in a few days and unravel the AudioContext object and how it gets hooked into the Canvas tag to visualize each sound's equalization. But for now, just enjoy the pretty color gradient while you play your music...
Thursday, March 7, 2013
Wednesday, March 6, 2013
Client Storage and the Jukebox
Jukebox Demo
After playing around with my jukebox every day (my son likes it when I play songs on it for him), I found it a little annoying that every time I want to use the jukebox, I need to drag and drop the audio tracks into it. Par for the course when you use a web app? That was probably correct 4 years ago, but today, HTML5 gives us options! HTML5 offers a small veriety of different options for storing data in the browser (client-side) - local storage, session storage, indexed database, and the file system. Session storage is useful only to store data for that session. (It's not very clear to be where you'd really want to use session storage). Local storage is useful for storing data from invocation to invocation but there are limits on the amount of data the browser allows (typically 2-5MB). An indexed database allows for asynchronous access and storage of binary data, coupled with indexing for easier searching. The file system is a sandboxed, local file system but the spec is still early and support is sparse.
For this feature, we'll use the indexed database to store the audio blobs and track info. Insertion order, for now, is dependent on when you inserted the files. Later, we'll go back using local storage or additional indexed database stores to add in playlists with ordering. In an ideal scenario, we'd use local storage to store jukebox configuration settings, the indexed database to store playlists, track categories, and track trivia, and the actual tracks would get stored in a sandboxed file system. We'll take a small hop forward before we just head-first into all the possibilities of client-side storage, and tackle the indexed database first.
So, let's give the indexed database a spin. All we want to do is store our audio blobs when they're added to the jukebox, load everything that was stored automatically when the site is loaded, and of course, allow the jukebox to play the tracks.
Right off the bat, storing a Blob in an IndexedDB in Chrome will not work. There is a bug that currently prevents Blobs from being stored (but it's recognized as a bug, and judging by the amount of activity in the thread, I'd guess that it stands a good chance at being fixed, since this is supported in every other Javascript engine). So, we need to use a polyfill to stringify our blobs for storage in the IndexedDB, and convert it back to a Blob when we pull it out. In this case, I attached a FileReader object, and read in the file as a binary string, then assigned the binary string as a key/value pair to the hash storing all the tracks. Here's the code snippet:
var files = e.dataTransfer.files;
for(var i = 0; i < files.length; i++) {
var f = files[i];
var reader = new FileReader();
reader.onload = (function(filePtr) {
return function(e) {
var str = e.target.result;
addTrack(filePtr.name, e.target.result);
};
})(f);
...
Important things to remember here: FileReader is an asynchronous API, meaning -
1. Files are read in parallel to the main thread executing. A call to "readAs...()" fires off that thread and the main thread continues execution. As a result, callbacks are needed to obtain the result (in this case, it's the 'onload' callback) and do the appropriate handling of the data.
2. Because the main thread continues execution, if there is stateful information that is needed when the callback gets executed, you need to pass that information in to the callback. Otherwise, your data will be out of sync. This is particularly true if the above code snippet actually had 2 or more files to deal with. For large files, the file name would not be correct.
As we can see in the following code, we simply write the binary string into the indexed database, and there's no issues:
function addTrack(name, blob) {
var transaction = db.transaction(["tracks"], "readwrite");
var store = transaction.objectStore("tracks");
var request = store.put({ "name": name, "blob": blob, "timeStamp": new Date().getTime() });
}
So, insertion's pretty easy since we take advantage of some native API functions to return us the type of data that's friendly to the indexed database. Pulling data from the database though... well, here we go...
The Jukebox does a single pull of data from the data - right at the very beginning of the script loading. We start a transaction to open a database handle, and upon loading, load up all the data.
1. var request = window.webkitIndexedDB.open("jukebox", 1);
2. request.onerror = error;
3. request.onsuccess = function(e) {
4. db = e.target.result;
5.
6. var transaction = db.transaction(["tracks"], "readwrite");
7. var store = transaction.objectStore("tracks");
8.
9. var keyRange = window.webkitIDBKeyRange.lowerBound(0);
10. var cursorRequest = store.openCursor(keyRange);
11. cursorRequest.onerror = error;
12. cursorRequest.onsuccess = function(e) {
13. var result = e.target.result;
14.
15. if (!!result == false)
16. return;
17.
18. var arrayBuffer = new Uint8Array(new ArrayBuffer(result.value.blob.length));
19. for(var i = 0; i < result.value.blob.length; i++)
20. arrayBuffer[i] = result.value.blob.charCodeAt(i);
21.
22. var blob = new Blob([arrayBuffer]);
23. fileBlobs[result.value.name] = window.URL.createObjectURL(blob);
24.
25. var option = document.createElement("option");
26. option.innerHTML = result.value.name;
27. option.id = result.value.name;
28. option.addEventListener('dblclick', handleDoubleClick, false);
29. option.addEventListener('click', handleClick, false);
30.
31. playlistElement.appendChild(option);
32.
33. result.continue();
34. };
35. }
I'm sure all of this code makes sense. Line 1, we send the request to open the database 'jukebox'. Because the indexed database is an asynchronous database, we don't know when the open will complete, so the only way to correctly handle errors, success, etc. is to assign callbacks, which are done after the call is made. It has to be done this way. Prior to the open request, we don't have a handle to the transaction, so we attach the callbacks on, and wait. We assign an error handler in line 2 ('error' is a function reference), and a success handler in line 3.
When we have successfully opened the database, this is when we want to grab all our track data. All the track data is stored in the 'tracks' store, so we open up a transaction to that store in lines 6 and 7. Line 9 allows us to specify a key range for our search in the the 'tracks' store. Data is stored based on timestamp, so we set our min range to 0, in order to grab everything. Line 10 opens a new request on the 'tracks' store to return all the data that matches the key range in the request.
At this point, we just need to wait for the data to be returned to us, then we'll do the appropriate thing. 2 indexed database concepts to be aware of. First, we originally passed in a hash, so the returned data (e.target.result in line 13) is a reference to the hash, along with additional members. Lastly, only 1 result is returned at a time in our success callback. For a request with multiple matches, we'll have multiple calls of the success callback - each returning a single set of data that matches the key range. However, we must call continue() from the returned result (line 33). This ensures that we'll get the entire data set.
Lines 18-22 do the conversion of the track data into something that our native <audio> tag can understand. First (line 18), we create an array buffer with a byte-level view that is the same size as the number of bytes in the track data. Lines 19-20, we iterate and assign the elements of the array buffer. Finally, in line 22, we can create our Blob object. This is by no means the fastest code in the world. There is probably a lot that could be done, using a Web Worker, to parallelize this work, or even a different approach to do a faster conversion.
And that's how we load and store data in the indexed database. It's a little different that the traditional, iterative way of programming where function calls block until the function returns. Think of it more like an implementation of hardware interrupts and handlers, but in the world of Javascript. The good thing? No x86 triple-fault shutdown. (The history and modern implementation of that... REALLY interesting)
After playing around with my jukebox every day (my son likes it when I play songs on it for him), I found it a little annoying that every time I want to use the jukebox, I need to drag and drop the audio tracks into it. Par for the course when you use a web app? That was probably correct 4 years ago, but today, HTML5 gives us options! HTML5 offers a small veriety of different options for storing data in the browser (client-side) - local storage, session storage, indexed database, and the file system. Session storage is useful only to store data for that session. (It's not very clear to be where you'd really want to use session storage). Local storage is useful for storing data from invocation to invocation but there are limits on the amount of data the browser allows (typically 2-5MB). An indexed database allows for asynchronous access and storage of binary data, coupled with indexing for easier searching. The file system is a sandboxed, local file system but the spec is still early and support is sparse.
For this feature, we'll use the indexed database to store the audio blobs and track info. Insertion order, for now, is dependent on when you inserted the files. Later, we'll go back using local storage or additional indexed database stores to add in playlists with ordering. In an ideal scenario, we'd use local storage to store jukebox configuration settings, the indexed database to store playlists, track categories, and track trivia, and the actual tracks would get stored in a sandboxed file system. We'll take a small hop forward before we just head-first into all the possibilities of client-side storage, and tackle the indexed database first.
So, let's give the indexed database a spin. All we want to do is store our audio blobs when they're added to the jukebox, load everything that was stored automatically when the site is loaded, and of course, allow the jukebox to play the tracks.
Right off the bat, storing a Blob in an IndexedDB in Chrome will not work. There is a bug that currently prevents Blobs from being stored (but it's recognized as a bug, and judging by the amount of activity in the thread, I'd guess that it stands a good chance at being fixed, since this is supported in every other Javascript engine). So, we need to use a polyfill to stringify our blobs for storage in the IndexedDB, and convert it back to a Blob when we pull it out. In this case, I attached a FileReader object, and read in the file as a binary string, then assigned the binary string as a key/value pair to the hash storing all the tracks. Here's the code snippet:
var files = e.dataTransfer.files;
for(var i = 0; i < files.length; i++) {
var f = files[i];
var reader = new FileReader();
reader.onload = (function(filePtr) {
return function(e) {
var str = e.target.result;
addTrack(filePtr.name, e.target.result);
};
})(f);
...
Important things to remember here: FileReader is an asynchronous API, meaning -
1. Files are read in parallel to the main thread executing. A call to "readAs...()" fires off that thread and the main thread continues execution. As a result, callbacks are needed to obtain the result (in this case, it's the 'onload' callback) and do the appropriate handling of the data.
2. Because the main thread continues execution, if there is stateful information that is needed when the callback gets executed, you need to pass that information in to the callback. Otherwise, your data will be out of sync. This is particularly true if the above code snippet actually had 2 or more files to deal with. For large files, the file name would not be correct.
As we can see in the following code, we simply write the binary string into the indexed database, and there's no issues:
function addTrack(name, blob) {
var transaction = db.transaction(["tracks"], "readwrite");
var store = transaction.objectStore("tracks");
var request = store.put({ "name": name, "blob": blob, "timeStamp": new Date().getTime() });
}
So, insertion's pretty easy since we take advantage of some native API functions to return us the type of data that's friendly to the indexed database. Pulling data from the database though... well, here we go...
The Jukebox does a single pull of data from the data - right at the very beginning of the script loading. We start a transaction to open a database handle, and upon loading, load up all the data.
1. var request = window.webkitIndexedDB.open("jukebox", 1);
2. request.onerror = error;
3. request.onsuccess = function(e) {
4. db = e.target.result;
5.
6. var transaction = db.transaction(["tracks"], "readwrite");
7. var store = transaction.objectStore("tracks");
8.
9. var keyRange = window.webkitIDBKeyRange.lowerBound(0);
10. var cursorRequest = store.openCursor(keyRange);
11. cursorRequest.onerror = error;
12. cursorRequest.onsuccess = function(e) {
13. var result = e.target.result;
14.
15. if (!!result == false)
16. return;
17.
18. var arrayBuffer = new Uint8Array(new ArrayBuffer(result.value.blob.length));
19. for(var i = 0; i < result.value.blob.length; i++)
20. arrayBuffer[i] = result.value.blob.charCodeAt(i);
21.
22. var blob = new Blob([arrayBuffer]);
23. fileBlobs[result.value.name] = window.URL.createObjectURL(blob);
24.
25. var option = document.createElement("option");
26. option.innerHTML = result.value.name;
27. option.id = result.value.name;
28. option.addEventListener('dblclick', handleDoubleClick, false);
29. option.addEventListener('click', handleClick, false);
30.
31. playlistElement.appendChild(option);
32.
33. result.continue();
34. };
35. }
I'm sure all of this code makes sense. Line 1, we send the request to open the database 'jukebox'. Because the indexed database is an asynchronous database, we don't know when the open will complete, so the only way to correctly handle errors, success, etc. is to assign callbacks, which are done after the call is made. It has to be done this way. Prior to the open request, we don't have a handle to the transaction, so we attach the callbacks on, and wait. We assign an error handler in line 2 ('error' is a function reference), and a success handler in line 3.
When we have successfully opened the database, this is when we want to grab all our track data. All the track data is stored in the 'tracks' store, so we open up a transaction to that store in lines 6 and 7. Line 9 allows us to specify a key range for our search in the the 'tracks' store. Data is stored based on timestamp, so we set our min range to 0, in order to grab everything. Line 10 opens a new request on the 'tracks' store to return all the data that matches the key range in the request.
At this point, we just need to wait for the data to be returned to us, then we'll do the appropriate thing. 2 indexed database concepts to be aware of. First, we originally passed in a hash, so the returned data (e.target.result in line 13) is a reference to the hash, along with additional members. Lastly, only 1 result is returned at a time in our success callback. For a request with multiple matches, we'll have multiple calls of the success callback - each returning a single set of data that matches the key range. However, we must call continue() from the returned result (line 33). This ensures that we'll get the entire data set.
Lines 18-22 do the conversion of the track data into something that our native <audio> tag can understand. First (line 18), we create an array buffer with a byte-level view that is the same size as the number of bytes in the track data. Lines 19-20, we iterate and assign the elements of the array buffer. Finally, in line 22, we can create our Blob object. This is by no means the fastest code in the world. There is probably a lot that could be done, using a Web Worker, to parallelize this work, or even a different approach to do a faster conversion.
And that's how we load and store data in the indexed database. It's a little different that the traditional, iterative way of programming where function calls block until the function returns. Think of it more like an implementation of hardware interrupts and handlers, but in the world of Javascript. The good thing? No x86 triple-fault shutdown. (The history and modern implementation of that... REALLY interesting)
Jukebox
Jukebox (for Google Chrome - not intended to run on any other browsers for now):
Version History:
v0.1 - Demo
v0.1a - Bug fix in looping the playlist
v0.2 - Added support for saving songs using an indexed database backend
v0.25 - Added an equalizer
|
Now Playing:
Length:
|
Playlist (drop files to add)
|
Version History:
v0.1 - Demo
v0.1a - Bug fix in looping the playlist
v0.2 - Added support for saving songs using an indexed database backend
v0.25 - Added an equalizer
Stripping .mp4 files...
On a whim, I tried dropping an mp4 file, with video, in to the jukebox on Chrome, just to see what would happen. Turns out... Chrome automatically strips off the video stream, and plays the audio just fine!
MP4 files... if you don't know much/anything about this file format, MP4 is simply a container format, with hooks that allow audio streams (like AAC) and video streams (like h.264) to attach content, synchronize during playback, and do other media-ish things. An .mp4 file doesn't mean very much. If the video or audio stored in the file uses an obscure or unsupported codec, you can't play it.
Leaving work this afternoon, I wondered whether the <audio> tag would support playback of just the audio stream, since it plays .m4a files, which are intended to be mp4 files with only the audio stream. Looks like it does!
I'm going to come back to this topic later, and peel back some of the .mp4 file format. Stripping the video stream isn't very difficult, conceptually, and should be able to be done with very little effort.
MP4 files... if you don't know much/anything about this file format, MP4 is simply a container format, with hooks that allow audio streams (like AAC) and video streams (like h.264) to attach content, synchronize during playback, and do other media-ish things. An .mp4 file doesn't mean very much. If the video or audio stored in the file uses an obscure or unsupported codec, you can't play it.
Leaving work this afternoon, I wondered whether the <audio> tag would support playback of just the audio stream, since it plays .m4a files, which are intended to be mp4 files with only the audio stream. Looks like it does!
I'm going to come back to this topic later, and peel back some of the .mp4 file format. Stripping the video stream isn't very difficult, conceptually, and should be able to be done with very little effort.
Tuesday, March 5, 2013
80 columns 4 j00
Annoying when coding guidelines say every line should be 80 columns, and people check in code that isn't...
Polyfills and Chrome
In an upcoming post, I'll touch on the IndexedDB, adding some persistence to the Jukebox. However, for now, a small post about bugs.
After doing some testing and getting my feet wet with the IndexedDB, I tried to do my first insertion - a hash containing a string and a Blob, and immediately saw that everything blew up. Why? After a little searching, I found that blobs are not a supported insertion type for the IndexedDB in Chrome. (This is where you try unsuccessfully to convert me to Firefox)
Software bugs exist. (So do hardware bugs... that's a completely different level of complexity) It's hard when you write applications on software stack, and run into bugs, feature limitations, etc., because it forces you to think in somewhat unnatural ways to accomplish what you're intending to accomplish - you're expecting the native API to support something; it doesn't; you have to write some code to work around that. And that piece of code... that's a polyfill.
I've been hit by a few issues in Chrome/Webkit. It's annoying. It's to the point at times that I've written my own polyfill to either manage the problem in a satisfactory way, or work around the problem completely. But, that's life as a software developer.
After doing some testing and getting my feet wet with the IndexedDB, I tried to do my first insertion - a hash containing a string and a Blob, and immediately saw that everything blew up. Why? After a little searching, I found that blobs are not a supported insertion type for the IndexedDB in Chrome. (This is where you try unsuccessfully to convert me to Firefox)
Software bugs exist. (So do hardware bugs... that's a completely different level of complexity) It's hard when you write applications on software stack, and run into bugs, feature limitations, etc., because it forces you to think in somewhat unnatural ways to accomplish what you're intending to accomplish - you're expecting the native API to support something; it doesn't; you have to write some code to work around that. And that piece of code... that's a polyfill.
I've been hit by a few issues in Chrome/Webkit. It's annoying. It's to the point at times that I've written my own polyfill to either manage the problem in a satisfactory way, or work around the problem completely. But, that's life as a software developer.
Friday, March 1, 2013
A minor detour
While working on audio visualization and processing, I thought it might be fun to try to learn HTML5's support for drag-and-drop, and see how well I could tie it into <audio> to make a jukebox of sorts. Call this version 0.1 - demo.
Just drag in any music file (.wav, .mp3, .m4a). Double-click a track to start playing. The player should rotate through the playlist and keep playing songs.
Disclaimer: I write my Javascript for Google Chrome. If this works on a Mozilla or IE browser, great. (If it works on an IE browser, I'll be really surprised... IE is pretty horrible) I'm not investing the time to make everything cross-compatible.
Disclaimer: You're on your own for sound files. I cannot and will not provide any music to use.
Disclaimer: This is a demo. If you're an enterprising person, or you're really lazy, you'll choose to copy and paste my code into your own application. Use of this code comes with no guarantee of support. If you choose to use it, learn from it, do something with it, at least do me the favor of dropping a comment in the box below and let me know.
UPDATE: The Jukebox demo has been relocated here
The Interesting Stuff
Drag-and-drop support in HTML5 is really nice, and rather painless. Quick notes on how to use it:
1. Add the 'drop' and 'dragover' event handlers to the element that is going to be dropped into. In my demo, I use a <div>. I'm not sure if all elements can be dropped into - my brief research didn't suggest that there were any restrictions on particular tags.
2. The drop handlers need to stop the browser from handling any dragged content natively. This is done through function calls to e.preventDefault() and/or e.stopPropogation() in the dragover and drop handlers.
3. When dropping files, you'll get a file list returned from e.dataTransfer.files. You'll need to handle it like you'd handle any file list in Javascript. Iterate through the array of File objects and handle each file iteratively.
I really like the File->Blob URL conversion that can be done rather painlessly through the native File API. Want a way to handle your File blobs in a way that native HTML tags and the browser will understand? Convert it to a Blob, then generate a blob:// URL for the Blob. You can treat that file as a URL for the lifetime of your program. Here's a snippet showing how I generate and store my blob URLs in an indexed hash:
fileBlobs[f.name] = window.URL.createObjectURL(f.slice(0, f.size, "image"));
Despite storing local files in a Blob, there is still some latency between when the source URL is assigned to the blob in the audio tag, and when the meta data from the file is parsed so things like the track duration can be retrieved. If you try to read the duration attribute right after assigning the src attribute, you'll likely get 'NaN' returned. The workaround - add an event listener for the 'loadedmetadata' event to the audio tag. Once the meta data has loaded, you can read the duration attribute and get a correct value (in seconds). Here's an example of how I handled support for calculating the duration:
audio.addEventListener('loadedmetadata', function(e) {
var durationDiv = document.getElementById("duration");
var minutes = Math.floor(audio.duration / 60);
var seconds = Math.floor(audio.duration) % 60;
if (seconds < 10)
durationDiv.innerHTML = "<b>Length:</b> " + minutes + ":0" + seconds;
else
durationDiv.innerHTML = "<b>Length:</b> " + minutes + ":" + seconds;
}, false);
I'll revisit this demo in the future, adding some visualizations rendered from AudioContext post-processing, a track randomizer, custom play controls, support for track seeking, maybe even real-time, controllable filtering.
Just drag in any music file (.wav, .mp3, .m4a). Double-click a track to start playing. The player should rotate through the playlist and keep playing songs.
Disclaimer: I write my Javascript for Google Chrome. If this works on a Mozilla or IE browser, great. (If it works on an IE browser, I'll be really surprised... IE is pretty horrible) I'm not investing the time to make everything cross-compatible.
Disclaimer: You're on your own for sound files. I cannot and will not provide any music to use.
Disclaimer: This is a demo. If you're an enterprising person, or you're really lazy, you'll choose to copy and paste my code into your own application. Use of this code comes with no guarantee of support. If you choose to use it, learn from it, do something with it, at least do me the favor of dropping a comment in the box below and let me know.
UPDATE: The Jukebox demo has been relocated here
The Interesting Stuff
Drag-and-drop support in HTML5 is really nice, and rather painless. Quick notes on how to use it:
1. Add the 'drop' and 'dragover' event handlers to the element that is going to be dropped into. In my demo, I use a <div>. I'm not sure if all elements can be dropped into - my brief research didn't suggest that there were any restrictions on particular tags.
2. The drop handlers need to stop the browser from handling any dragged content natively. This is done through function calls to e.preventDefault() and/or e.stopPropogation() in the dragover and drop handlers.
3. When dropping files, you'll get a file list returned from e.dataTransfer.files. You'll need to handle it like you'd handle any file list in Javascript. Iterate through the array of File objects and handle each file iteratively.
I really like the File->Blob URL conversion that can be done rather painlessly through the native File API. Want a way to handle your File blobs in a way that native HTML tags and the browser will understand? Convert it to a Blob, then generate a blob:// URL for the Blob. You can treat that file as a URL for the lifetime of your program. Here's a snippet showing how I generate and store my blob URLs in an indexed hash:
fileBlobs[f.name] = window.URL.createObjectURL(f.slice(0, f.size, "image"));
Despite storing local files in a Blob, there is still some latency between when the source URL is assigned to the blob in the audio tag, and when the meta data from the file is parsed so things like the track duration can be retrieved. If you try to read the duration attribute right after assigning the src attribute, you'll likely get 'NaN' returned. The workaround - add an event listener for the 'loadedmetadata' event to the audio tag. Once the meta data has loaded, you can read the duration attribute and get a correct value (in seconds). Here's an example of how I handled support for calculating the duration:
audio.addEventListener('loadedmetadata', function(e) {
var durationDiv = document.getElementById("duration");
var minutes = Math.floor(audio.duration / 60);
var seconds = Math.floor(audio.duration) % 60;
if (seconds < 10)
durationDiv.innerHTML = "<b>Length:</b> " + minutes + ":0" + seconds;
else
durationDiv.innerHTML = "<b>Length:</b> " + minutes + ":" + seconds;
}, false);
I'll revisit this demo in the future, adding some visualizations rendered from AudioContext post-processing, a track randomizer, custom play controls, support for track seeking, maybe even real-time, controllable filtering.
Subscribe to:
Posts (Atom)