Android + Rhino = ?

So, what do you get when you cross an android with a rhinoceros? I have no idea, but that's exactly what I've been doing over the past few weeks.

I recently downloaded the Android Scripting Environment (ASE) that I discovered on Google Labs. ASE is a nifty little project that allows you to write scripts for your Android device. At the time, the project supported the following languages:

• Lua
• BeanShell
• Python
• Ruby (via JRuby)
• Perl

The problem is that I don't really know any of those languages. So, I started learning a little bit about each of them. I finally decided to go with Ruby since it was the most object-oriented of the bunch, but about half way through the tutorials, I had an epiphany, "What about JavaScript? Everybody knows JavaScript." So I decided to submit a feature request.

I navigated my world-wide web browser over to the project's issue tracker, and, being the good user that I am, did a quick search for the word JavaScript. The query brought back a single, five-month-old issue titled Add JavaScript support. I read through it and noticed that they suggested doing it using an open source Java implementation of JavaScript by Mozilla called Rhino. I didn't think much about it at the time, I just starred the issue (i.e. added my vote to implement it) and went on with my day.

Later that day, I got to thinking: You know, that issue is five months old. They've already implemented five different interpreters. I may not know anything about Rhino or ASE, but I'm a pretty capable developer and could probably get it working in a matter of days if I really tried. So, I asked them if they'd accept a user-contributed patch. They said they'd be glad to, and I got to work on making it happen.

Implementing it was pretty straight forward. On the ASE side, I just needed to add a couple classes: one telling it about the new interpreter, and the other telling it how to start the interpreter. For the Rhino side, I just compiled the jar with debugging off (to save storage space) and ran it through the dx tool (so Android could understand the compiled code). The dx tool took me a little while to figure out because I kept getting an out of memory error when running it, but once I learned that the default stack size was too small, things fell into place fairly quickly. Now, the only other thing I had to do was create a JavaScript proxy class that scripts could use to make JSON-RPC calls to the Android API (via the AndroidFacade). I followed suite with the BeanShell and Ruby implementations and was able to come up with a fairly elegant solution. Now everything was in place, and it was time test.

I immediately started getting errors the first time I went to test it. Using the interactive shell I was able to call methods through the proxy class but wouldn't get any return values, and the executing saved scripts just spewed an ugly stack trace all over the screen. The first problem (no return values) was caused by an ambiguity in the JSON.parse() method to Rhino so, I figured the string was coming form a fairly trusted source and just went with JavaScript's built-in eval() method instead. The next problem was not so straight forward. Investigating the ugly stack trace showed that it was yet another out-of-memory error. I fixed that problem (again by increasing the stack size), but yet another error appeared: Class format not accepted. This threw me off. At first, I thought Dalvik (Android's JVM) was trying to load the JavaScript file directly, but that didn't make sense since other interpreters were working fine. After a few days of meditating and some ritual sacrifices to the programming gods, it hit me: Rhino was compiling the JavaScript file to a Java class file and then handing that class file to Dalvik which had no idea what to do with it (since it only understands Dex files). So I turned off Rhino's optimization (thus causing it to interpret and not compile the scripts), and all was well. The Rhinodroid was complete!

I contributed my code, and waited. It took a few days, but Damon Kohle reviewed it and merged it into the main codebase. About the only change I've noticed was that he set the default buffer sizes (making a couple of debug messages go away). He was really quick to get a new release out so now, thanks to my obsession with coding, users can now choose from JavaScript along with the other languages I mentioned previously.

All in all, this was a very fun experience. I got to learn more about Android and Rhino development, and I even got to use a distributed version control system for the first time. JavaScript is probably the widest known scripting language, and I hope that my contribution can help add to the success of ASE and can be yet another of my little claim-to-fames. But most of all, I'm proud that I can finally say, "I've contributed to an open source Google project!"

Teaching My Android to Read

About a month ago, I finally gave in and decided to join the smart phone revolution. After careful consideration, I decided to go with the T-Mobile MyTouch 3G (powered by Google Android). I was attracted to the liberating UI, the elegant system architecture, and, of course, the great SDK.

I had been racking my brain for a couple weeks trying to think of a great application to make when, all of a sudden, it hit me. I was marveling over the Barcode Scanner application when I suddenly the idea: What if I combine the device's camera with OCR?

OCR is the process of turning pictures into text, and once you have raw text, the possibilities are endless. As a simple example, imagine being able take a picture of an address on a brochure and having it immediately displayed on a map. The same idea could also be applied to calling phone numbers, or even translating!

Another technology that could come in to play with this concept is the newly added TTS feature. Now you could have things read out loud to you. Whether you can't afford to take your eyes off the road, can't see because you forgot your glasses, or want to make every book a self-reading child book? No problem, just take a quick picture, and the text will be read out loud.

The next step is to do layout analysis on the picture. This tells you things like at (x, y) on the picture, there is z letter/word/block of text. So now you could just hold your phone over a page of text, and have all the instances of the word "the" highlighted in yellow -- finally, that real-life Ctrl+F (find) you've always wanted!

Well friends, after about two weeks of development, I finally have a working prototype! I stole about 98% of the code from the ZXing, Ocrad, and STLPort (Gears's fork) projects, but hey, this is open source, and that's how we roll. Here's how it works:

1. Hold your device over a document. The camera will auto-focus and send the image to Ocrad for processing.
2. Once Ocrad has identified some text, it will be returned and displayed on the screen.

Here is an example of it working on the Domino's ad that was on my door this afternoon:



Surprisingly, it only took Ocrad took about 200ms to process the entire image on the device. It takes Barcode Sanner about twice as long to process a 2D barcode. Although, I did cheat by using a native processor (Barcode Scanner has the overhead of Java), and mine only has to scan for one type of image (Barcode Scanner scans for many different formats for each picture).

Next, I'm going to get it processing the layout and visually overlaying it somehow on the screen. I have a feeling though that this is going to add a pretty big hit to the performance, but as long as I can keep it at least as good as Barcode Scanner, I think it will be acceptable.

When I have time, I'll also try to get the code posted on CodePlex for those interested in seeing the exact details of how it was all done.