Android Things and Machine Learning

Android Things allows you to make amazing IoT devices with simple code, but one of the things that can make a device extraordinary is machine learning. While there are a few services available online that will allow you to upload data and will return results, being able to use machine learning locally and offline can be incredibly useful. 

In this article, I'll share some of my experiences using the TensorFlow image classifier, starting with Google's Android Things TensorFlow example

Why Use Machine Learning?

Machine learning can help solve problems that conventional apps cannot. To provide context, let's go through a simple example where machine learning can be used with an IoT device to improve daily life.

Here in Colorado, it’s not uncommon to see news articles about wildlife coming out from the mountains and walking around a downtown:

News story of a moose in downtown Boulder Colorado

Or bears climbing trees at the local university:

News article of a bear at CU Boulder in Colorado

I've even had a friend post video of a bear outside of their home!

Snapshot from a video of a bear outside of a home

While these situations are usually taken care of without issue, there is public data available from the State of Colorado’s Division of Parks and Wildlife that details black bear-human conflicts, and various wildlife species activity/ranges. By looking at the black bear-human conflict data in Google Earth, we can find areas where bear encounters can be a risk for public safety.

Colorado bear-human conflict map

In addition, while I was reading up on wildlife data in the state, a friend posted an image of her car after having an accident involving an elk during her drive down to Durango, CO. According to the Colorado Department of Transportation (CDOT) Crash Book, just over 4,000 accidents involving wildlife occurred over one year in Colorado, with about 150 injuries and one fatality. In the entire United States, this number jumps to 725,000 to 1,500,000 accidents with about 200 human fatalities a year, according to the Defenders of Wildlife facts sheet.

Car after hitting an elk near Durango Colorado

So how can we use machine learning to help solve this problem? With image recognition, we can create a motion-triggered device that can take a picture with a Raspberry Pi and then analyze it to detect a potentially dangerous wild animal.

Using the Google sample, we can create an Android Things device that takes pictures with a Raspberry Pi and classifies the content of those pictures against over a thousand possible labels. But these labels don't exactly match what we'd need to use. 

By training TensorFlow to use our own images and labels, we can create a device that can identify an elk, moose or black bear, and then perform an action if one is detected. This allows us to make a device that can potentially save lives, while still being simple to create.

Creating a Custom TensorFlow Image Classifier

After you have installed and run Google's TensorFlow for Android Things sample, it's time to start modifying it. The first thing you will want to do is ensure that TensorFlow is on your computer and working. This can be fairly complex, and the easiest way I have found to make it work properly through the entire process of generating trained files is to install and use Docker. This program will allow you to run virtual machines on your computer that are preconfigured for TensorFlow. 

Once you have installed Docker and have it running on your computer, you should open its preferences and set memory usage for your virtual machines. I set mine to use 7 GB of memory, which may be more than you need, but I spent days attempting to get TensorFlow to properly create the required trained graphs without crashing before I realized that the virtual machine was running out of memory.

What we will be doing in this example is retraining the existing machine learning example to use our own data, as this is much faster than training a new dataset from scratch. For a more detailed explanation of what we're doing, you can take a look at the official TensorFlow documentation.

Docker advanced settings screen

Once you've installed Docker and started it on your machine, you'll need to run it from a terminal and pull down an image. For this example I am running under macOS, so the commands may be a bit different for your platform.

When everything has finished setting up, you should have a prompt in your terminal similar to this:

At this point you'll need a set of images to train TensorFlow with. I used the Fatkun Batch Download Image Chrome plugin to download returned images from a Google search en masse. When you have the plugin installed, you can search for whatever you want to classify and start to select images that you would want to save.

Chrome plugin screen for downloading bulk images

To make naming a little easier, you may also want to go into the More Options section and let the plugin rename the images while downloading them.

Bulk image download rename setting

Next you will need to move the images that you are using into a folder in tf_files under your home directory, which is a folder we created while initializing our docker machine. For this example, my images directory is called TensorFlowTrainingImages. Each classifiable item should have its own folder within that directory, as shown below.

TensorFlow training images folder structure

Once your directories are set up, you can start retraining with the following command from your Docker terminal:

The above command will generate bottlenecks, which are essentially data used by the final classification data pass, and a graph and labels file that are used for classification. 

From this point forward, the operations we run with TensorFlow can take anywhere from a few minutes to over an hour, depending on the speed of your computer. This might be a good time to make coffee or go for a quick walk. 

As the retraining command runs, you should see a lot of output in your terminal similar to this:

Once your bottlenecks have been generated, you will have a graph.pb file and a labels.txt file that represents your data. While these formats work great when running classification on your computer, they tend to not work when placed into an Android app. You will need to optimize them.

Start by running the following command. Accept all the default values.

Once configuration has finished, run the following command to set up the optimization tool. This step took about an hour on my machine, so patience is key!

Once your optimization tool is built, you can use it to optimize your graph file.

Now that your optimized graph is generated, you can find it with your labels in the tf_files folder in your home directory.

Contents of TensorFlow directory after generating machine learning data

Adding Your Custom Classifier to the Android Things Sample

Congratulations! Generating the TensorFlow graph and labels is the hard part, and may take some finagling to make generate properly. Now that you have them, it's time to edit the Android Things sample project to use your own data. First, go into the app's build.gradle file and remove the following section, as well as its plugin declaration:

The above code downloads Google's sample graph and labels and adds them to the assets directory. Since we've generated our own, we don't need to worry about those and can delete those lines. 

Next, open the file. You should see a few declared variables towards the top of the file, with a long comment about what will need to be changed if you're using your own graphs that were generated from a code lab (which is close enough to what we did above).

Change the last part of MODEL_FILE and LABEL_FILE to match the name of your optimized graph and label.

Back to the long comment in the original source, let's go ahead and edit the file to match Google's recommendations.

You'll notice that we're setting NUM_CLASSES to 4. This is the number of items available for our machine learning algorithm to classify. You can change this to match the number of classification categories you've trained with.

To wrap things up, move your optimized graph file and labels into the app/src/main/assets directory. Once that's done, you can install the sample app onto a Raspberry Pi with a camera module and give it a shot. Below is a picture I took with the device:

Picture of a bear taken with Android Things device

And the results that were sent to Firebase (with some slight modifications to the sample app).

Output data from machine learning classification of image
As you can see, the device successfully identified the black bear in the picture!


In this tutorial you got a brief introduction to using TensorFlow for machine learning and integrating it into an Android Things app. As you've seen, machine learning is a powerful tool that can be used to solve various problems that a conventional app would struggle with. When combining machine learning with the Internet of Things, you can create some amazing and useful devices that interact with the world around them. 

Using what you've learned from this article, and others in the Android Things series, you can build some really amazing new apps and devices!



Related Articles