Since I've been working on two projects just recently that deal with large amounts of binary data, here is my current solution to get the data into an array via an AJAX call. The code is based on a solution that I've found here and here, however I've modified it to be more compact and I've also removed the base 64 handling.
The code is broken out into two separate functions, one of which deals with the XHR request and the second function then loops through the incoming raw data and copies the masked low-bytes into the bytebuffer array. This allows for more flexibility during implementation, since it is quite likely that only the extract_binary function will need your attention to adjust the code to your needs.
As-is the code will only work with binaries on the local server - there are hacks for crossdomain access or it would be possible to use a server proxy script to access other domains.
Here is a working example of the code - it has been tested in Firefox 3.6.x and Safari 4.0.5. Two jpegs in the same directory are being loaded and all low-ascii characters in the first 1024 bytes are being displayed.