Welcome to Part 2 of our series on building augmented reality applications on the Android platform. In Part 1, you learned how to use the camera, draw text over the camera view, and retrieve different values from the various sensors on the device. We'll now add in GPS data, mix it all up with some math, and pinpoint the location of a fixed object in space over the camera view as well.
Also available in this series:
- Android SDK Augmented Reality: Camera & Sensor Setup
- Android SDK Augmented Reality: Location & Distance
Step 0: Getting Started and Prerequisites
This tutorial moves at a pretty fast pace. We expect readers to be familiar with creating and running Android projects. We also expect readers have an Android device that is sufficiently powerful for running AR apps, such as the Nexus S. Most testing of this application will need to be done on the device, as the app relies heavily upon the camera, sensors, and location data that is not readily available in the Android emulator.
We have provided a sample application in conjunction with this tutorial for download. This project will be enhanced over the course of the AR tutorial series. So go download the Android project and follow along!
Part 4: Using the On-Device GPS
For the location-based AR application that we're building, it's critical that we know where the device is located. And, for accuracy, we need the best location provider available on the device. Let's get started.
Step 1: Location Permissions
Applications which use any location data from the device require certain location permissions. There are two options for this. One option is to use a course location, which provides a location based on cellular towers or known Wi-Fi access points. This level of granularity works perfectly well for determining what city you're in and maybe the neighborhood. However, it probably won't be able to determine which street or block you're on correctly. The other option is to leverage the fine location, which uses the GPS on the device to get an accurate position fix. This takes a little longer, but uses GPS satellites and cellular towers to increase the accuracy of the location results. Fine location data can usually help you determine what side of the street you're on. It's the level of accuracy and precision that is generally used for AR apps.
Therefore, here’s the permission you must add to the manifest file:
<uses-permission
android:name="android.permission.ACCESS_FINE_LOCATION" />
Step 2: Listening to Location
Now that we have permission to access the GPS services of the device, let's start listening to what it has to say, shall we? Let's add this information directly to the OverlayView class (for simplicity more so than design).
The OverlayView class must now implement LocationListener.
public class OverlayView extends View implements SensorEventListener, LocationListener {
// …
}
Next, implement the mandatory override methods of the LocationListener interface:
private Location lastLocation = null;
public void onLocationChanged(Location location) {
lastLocation = location;
}
public void onProviderDisabled(String provider) {
// ...
}
public void onProviderEnabled(String provider) {
// ...
}
public void onStatusChanged(String provider, int status, Bundle extras) {
// ...
}
Next, register to receive location changes, within the OverlayView(context) constructor method:
locationManager = (LocationManager) context.getSystemService(Context.LOCATION_SERVICE); Criteria criteria = new Criteria(); criteria.setAccuracy(Criteria.ACCURACY_FINE); criteria.setPowerRequirement(Criteria.NO_REQUIREMENT); String best = locationManager.getBestProvider(criteria, true); Log.v(DEBUG_TAG,"Best provider: " + best); locationManager.requestLocationUpdates(best, 50, 0, this);
Note that we request the best provider of location data at the finest accuracy we can get. At this point, you can add the GPS location to the onDraw() method for debugging purposes, if you'd like. This is in the source code provided, but we'll leave it as an exercise to the reader.
Part 5: Calculating Direction and Distance Between Two Points
This part of the tutorial turns to some mathematics – or we would if it wasn't for some nice helpers. In this two-dimensional implementation of the AR, we don't particularly care about this distance, it can be useful to apply some light scaling, or even stepped scaling (for example, three sizes), to display points of interest at varying distances. Another use could be to only show locations within – or outside of – a certain range from the device.
What we are very interested in, however, is the direction to the object relative to the device. In navigation, this is called the bearing. A path between two points on a spherical object, such as the planet Earth, is shortest by following what's called the great-circle distance. If you're using AR techniques to find a location or display where some fixed object might be in relation to your location, this would be reasonable distance mechanism to use. As you get closer, the bearing might change, but you'd still be following the shortest route to that location.
An alternate solution might be to calculate a straight line, in 3D space, between two points on the sphere. This method assumes that a path through the sphere, as opposed to around its surface, is feasible or interesting. In this way, if you were to try to locate, say, Sydney, Australia from San Francisco, California, you might need point nearly straight down towards the ground. While amusing, this mechanism is not terribly useful, unless you're locating stars.
We could get in to some math about how this works. Indeed, we'd use the haversine formula manually to determine the bear from one location to another. However, the Location object, conveniently passed to us in the onLocationChanged() method, contains many helpful navigational methods to aid us. Two of these methods are distanceTo() and bearingTo().
But what are we going to get distance and bearing to? Ultimately, you'd want the locations to be determined from a database or feed. You can then filter relevant objects to display on the view by distance, if you so choose.
For now, however, we'll set up a single location to look for: Mount Washington, the highest peak in the Northeastern USA and home of some of the world’s worst weather (we also happen to be able to see it readily from our house, which makes testing a bit easier).
Set the static coordinates for Mount Washington like this:
private final static Location mountWashington = new Location("manual");
static {
mountWashington.setLatitude(44.27179d);
mountWashington.setLongitude(-71.3039d);
mountWashington.setAltitude(1916.5d);
}
Tip #1 for developing and testing location-based AR apps: Use the most accurate location data you can get your hands on. If your data isn’t accurate, it’s not going to show up as accurate on the screen. When dealing with locations, even a few points off can make a huge difference. For example, the difference between 44 North and 44.1 North is over 11 kilometers!
Next, we can determine the bearing to this location using the following code:
float curBearingToMW = lastLocation.bearingTo(mountWashington);
At this point, you may want to add this information to the view debug output so that you can see how the bearing updates as you move about with the device. Just sitting at your desk? The bearing to your target won't change much.
Part 6: Determining the Handset Bearing
Now that you can determine the bearing to a target, you might be wondering how the phone knows which direction it's pointed. Luckily, all the sensor data we need is already in place. All that's left to do is to use this data to determine which way the phone is pointing.
How do we do that? Well, the phone has a compass. Can we just use that? Not directly. The compass reports the magnetic field around the x-, y-, and z-axis in micro-Teslas. If your response to this statement is something along the lines of, "Huh?" then you've also realized this isn't directly usable – we need to convert the data into a format that tells us which way the phone is pointing. And by pointing, we really mean which way the camera is pointing.
As it turns out, this requires that you take into account the orientation of the device, which uses a combination of the accelerometer and compass. The end result will be a vector with rotation angles around each of the 3 axes. This precisely orients the device with respect to the planet – exactly what we want. Luckily, again there are helpers to do the matrix math we need.
The first helper method we can use is conveniently located in the SensorManager class: getOrientation(). This method takes a rotation matrix and returns a vector with azimuth, pitch, and roll values. The azimuth value is rotation around the Z-axis and the Z-axis is that which points straight down to the center of the planet. The roll value is the rotation around the Y-axis, where the Y-axis is tangential to the planetary sphere and points towards geomagnetic North. The pitch value is the rotation around the X-axis, which is the vector product of the Z-axis and Y-axis – it points to what could be called magnetic West.
But where do we get the rotation matrix? As it happens, the Android SDK once again has a helper for this in the SensorManager class called getRotationMatrix(). Call it, passing in the accelerometer vector and compass vector, like so:
// compute rotation matrix
float rotation[] = new float[9];
float identity[] = new float[9];
boolean gotRotation = SensorManager.getRotationMatrix(rotation,
identity, lastAccelerometer, lastCompass);
Now we can compute the orientation:
if (gotRotation) {
// orientation vector
float orientation[] = new float[3];
SensorManager.getOrientation(rotation, orientation);
}
Ok, so we now know how the device is oriented relative to the planet. Quick, where's the camera pointed? Right, let's remap the rotation matrix so that the camera is pointed along the positive direction of the Y-axis, before we calculate the orientation vector:
if (gotRotation) {
float cameraRotation[] = new float[9];
// remap such that the camera is pointing straight down the Y axis
SensorManager.remapCoordinateSystem(rotation, SensorManager.AXIS_X,
SensorManager.AXIS_Z, cameraRotation);
// orientation vector
float orientation[] = new float[3];
SensorManager.getOrientation(cameraRotation, orientation);
if (gotRotation) {
float cameraRotation[] = new float[9];
// remap such that the camera is pointing along the positive direction of the Y axis
SensorManager.remapCoordinateSystem(rotation, SensorManager.AXIS_X,
SensorManager.AXIS_Z, cameraRotation);
// orientation vector
float orientation[] = new float[3];
SensorManager.getOrientation(cameraRotation, orientation);
}
Now might be a good time for you to add some further debug text output to the screen, or logger (We find we often have to get up from our desks to test, so the screen continues to be a convenient place to output such things).
Part 7: Marking the Target Location in the Overlay View
Now we have all of the information needed to place the target on the screen: the direction the camera is facing and the location of the target relative to our location. We currently have two vectors: the camera orientation and the bearing to the target. What we need to do is map the screen (or View, actually) to a range of rotation values, and then plot the target point on the View when it's location is within the field of view of the camera image as displayed on the screen.
A couple of things to be aware of: As of now, our orientation vector uses radians for units, and the bearing uses degrees. We'll need to be sure we're working with the same units and those of the appropriate type. The bearing calculation returned degrees, and we can use rotations in degrees on the Canvas object, so we'll convert values to degrees when they are in radians. The package java.lang.Math provides methods for doing conversions between degrees and radians.
Recall from earlier that the orientation vector starts out with the azimuth, or the rotation around the Z-axis, which points straight down. With the Y-axis pointing to Geomagnetic North, the azimuth will be compared to our bearing and this will determine how far left or right the target is on the screen – if it's on the screen at all (when it's outside the field of view of the camera).
Similarly, the pitch is used to determine how far up or down the target should be drawn on the screen. This may or may not be something your application requires. Finally, the roll would change the virtual horizon that might show on the screen – that is, this would be the total screen rotation. Again, this may not be interesting to your application. It depends upon what you’re after.
We'll apply all three to our sample application, just for kicks. Exactly to what extent the screen encompasses a real world field of view is entirely up to you. However, the Camera.Parameters class can provide the field of view of the device camera, as configured by the manufacturer, using the getVerticalViewAngle() and getHorizontalViewAngle() methods. Convenient, huh?
Camera.Parameters params = Camera.open().getParameters(); float verticalFOV = params.getVerticalViewAngle(); float horizontalFOV = params.getHorizontalViewAngle();
Now we can apply the appropriate rotation and translation to place the horizon line and the target point correctly, relative to the screen and field of view of the camera:
// use roll for screen rotation canvas.rotate((float)(0.0f- Math.toDegrees(orientation[2]))); // Translate, but normalize for the FOV of the camera -- basically, pixels per degree, times degrees == pixels float dx = (float) ( (canvas.getWidth()/ horizontalFOV) * (Math.toDegrees(orientation[0])-curBearingToMW)); float dy = (float) ( (canvas.getHeight()/ verticalFOV) * Math.toDegrees(orientation[1])) ; // wait to translate the dx so the horizon doesn't get pushed off canvas.translate(0.0f, 0.0f-dy); // make our line big enough to draw regardless of rotation and translation canvas.drawLine(0f - canvas.getHeight(), canvas.getHeight()/2, canvas.getWidth()+canvas.getHeight(), canvas.getHeight()/2, targetPaint); // now translate the dx canvas.translate(0.0f-dx, 0.0f); // draw our point -- we've rotated and translated this to the right spot already canvas.drawCircle(canvas.getWidth()/2, canvas.getHeight()/2, 8.0f, targetPaint);
Basically, the above code rotates by the roll value, translates up and down by the pitch value, and then draws a horizon line. Then the azimuth is used for left or right translation of the field of view and the location point (in this case, where the mountain is) is drawn (as a circle) where the expected.
The end result will look something like this:
(Excuse the contents. Taking screen shots while outside is difficult.)
You'll notice that the horizon on the screen shot is rotated counter-clockwise. That's because the phone, at the time we took the screenshot, was rotated clockwise. Thus, the horizon on the screen lines up with the real one.
You've now augmented your view of the real world by placing information about what the camera is seeing right on the screen! That wasn't so hard, was it? You’ve come a long way, but there’s a lot more to AR, once you get going.
Other Issues: Wigglies, Jigglies, Performance, and Mobile Best Practices
If you've run the sample application, you might have noticed that the line and target are bouncing all over the place. That's because the sensors are very sensitive. What we need to do is smooth the data. This is best done with a low-pass filter; that is, a filter that blocks the high frequency data but allows for low frequency data. This will smooth out the data and reduce the jiggles on the screen. We'll discuss this in a future AR tutorial.
Depending on your location and the location of the target, you may find that the bearing seems odd. The closer you are to the geomagnetic North, the farther from True North the compass will show. In fact, where we're located, True North is a full 16 degrees off. This is quite noticeable when locating a big object – such as Mount Washington. We'll discuss this in a future AR tutorial.
You've also probably noticed that if you run the app for a while, it tends to bog down the entire device. This is because we're now doing math wizardry that takes too long for the onDraw() method. As you know, operations that take time should always be performed off the UI thread. The OverlayView object we've created is self contained, making it's own use of sensors and the GPS services of the device. This means it will have to do threading in a self contained, safe way.
Again, these are issues that will likely be addressed in future tutorials of this series.
Conclusion
This concludes the second part of our AR tutorial. Augmented reality is an exciting genre of applications on the Android platform. The Android SDK provides everything – including many mathematical helpers -- necessary for developing sleek, interesting AR apps, but not all devices meet the hardware requirements that those apps require. Luckily, the newest generation of Android devices is the most powerful yet, and many meet or exceed the hardware specifications required for true AR app development.
About the Authors
Mobile developers Lauren Darcey and Shane Conder have coauthored several books on Android development: an in-depth programming book entitled Android Wireless Application Development and Sams Teach Yourself Android Application Development in 24 Hours. When not writing, they spend their time developing mobile software at their company and providing consulting services. They can be reached at via email to [email protected], via their blog at androidbook.blogspot.com, and on Twitter @androidwireless.
Need More Help Writing Android Apps? Check out our Latest Books and Resources!
Comments