Augmented reality (AR) is a hot topic in mobile apps today. Smartphones and tablets have the power and the hardware capable of enabling developers to write interesting new applications that incorporate live camera and video feeds, hyper-accurate sensor data, and other real-time user data in interesting ways. Today we’ll begin exploring the world of augmented reality, and what the Android platform has to offer for developers looking to build AR applications and provide deep, rich experiences to their users.
Also available in this series:
- Android SDK Augmented Reality: Camera & Sensor Setup
- Android SDK Augmented Reality: Location & Distance
Step 0: Prerequisites and Getting Started
This tutorial moves at a pretty fast pace. As such, we expect readers to be familiar with all the basics regarding creating and running Android projects. We also expect that readers will have an Android device that is sufficiently powerful for running AR apps (such as the Nexus S). Most testing of this application will need to be done on an actual Android device, as the app relies heavily upon the camera, sensors, and location data that is not readily available in the emulator.
We have provided a sample application in conjunction with this tutorial for download. This project will be enhanced over the course of this premium AR tutorial series, so go download the Android project and follow along!
Part 1: What Kind of AR App are We Developing?
First, we need to define (in broad terms) what this application will do. We want to stick with the typical features of an AR application, and improve it over time.
So what kind of application will we build? We’re going to build a location-based AR app. These types of AR apps generally show where things are in the real-world by indicating where the app thinks they are over the camera view when the user holds the phone up and moves it about.
A different type of AR app is the type that recognizes the object using computer vision algorithms and can either draw information over the camera to show what the object is, how it’s oriented, or perhaps add or show other objects in the view. While we won’t be building this sort of AR app, many of the techniques we teach here can be applied to either type of AR application.
The basic architecture of a location-based application involves a camera view, the user’s location, the phone’s orientation compared to the real world, and a list of locations for items that we want to show on the display – the data with which we augment the camera’s vision of the world.
This first part of our series will talk about the camera and the sensors. In the next part, we'll talk about location data and the math involved in building the app.
App Requirements: What Device Features Does our App Need?
Your typical AR application requires access to the following device features:
- Access to the camera view
- Access to the user’s location
- Access to other device sensors, especially a compass (accelerometer and gyroscope can also be useful)
The application may also require other services and permissions, such as network access, but these are not central to the AR aspects of the application.
A Note on Target Devices: The Nexus S
Not all Android devices (or mobile devices in general) have the hardware specs to be able to handle AR application requirements. For the purposes of this tutorial, we are targeting the Nexus S running Android 2.3.4, which meets our specific hardware requirements nicely. Therefore, our application will target a very specific version of the Android SDK. This will keep the app simple. However, the SDK requirements can be more flexible once the Android manifest correctly identifies any required hardware features, or you implement alternative functionality for less powerful devices.
While the Nexus S is more than capable of performing reasonable AR tasks, many other devices are also capable, including some older devices. If you’re building a production app that uses AR technology, you’ll want to test carefully on your target devices for response, usability, and correct functionality.
Part 2: Project Setup
Now we’re ready to start developing! Let’s get an AR project set up properly first. This application is configured much like any other application project in Eclipse.
Step 1: Create a New Eclipse Project
Begin by creating a new Android project in Eclipse. Here are the project details we used for the sample project (available for download):
- Project Name: ARTUT
- API Level Min/Target: 10
- Launch Activity Class: ArtutActivity
Step 2: Configure the Manifest File
Next, you’ll want to configure the Android Manifest File. You’ll want to specify the following settings:
On the Manifest Tab, add
- android.hardware.camera (Required=true)
- android.hardware.camera.autofocus (Required=false)
- android.hardware.location.gps (Required=true)
- android.hardware.sensor.compass (Required=true)
- android.hardware.sensor.gyroscope (Required=true)
- android.hardware.sensor.accelerometer (Required=true)
On the Application Tab:
- Debuggable should be set to true
On the Permissions Tab, add
- android.permission.CAMERA
- android.permission.FINE_LOCATION
Step 3: Connect the Target Device
Finally, if you haven’t already done so, you’ll need to setup an Android device for testing and plug it into your development machine via USB. If you’ve not done this before, you may need to go download and install drivers prior to use.
There’s little point in creating an Android Virtual Device (AVD) for use with this project, as the emulator does not have sufficient features for testing live-camera or sensor based applications with any degree of effectiveness. We will do all of our development and testing against real hardware. Besides providing a better testing environment when using sensors, the camera, and other hardware, we also find that testing on real hardware is much faster due to the performance and feature limitations of the emulator.
Part 2: Working with the Camera
Our first real step in developing an AR app involves getting camera content displayed on the screen. We do this by implementing a custom SurfaceView that is displayed within the activity’s layout.
Step 1: Defining the App Screen Layout
First, we need to define the layout resource that will be used for the main AR screen. A FrameLayout is most appropriate for our screen, as it allows the layering of views within the display area. Therefore, our layout resource, /res/layout/main.xml, can be quite simple:
<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:id="@+id/ar_view_pane"
android:layout_width="fill_parent"
android:layout_height="fill_parent">
</FrameLayout>
Step 2: Defining a Custom SurfaceView to Hold Camera Content
Next, you need to define a custom view control for use within your FrameLayout. The SurfaceView class can be used for this purpose. Therefore, you can use this class to encapsulate your camera drawing.
public class ArDisplayView extends SurfaceView
{
public static final String DEBUG_TAG = "ArDisplayView Log";
Camera mCamera;
SurfaceHolder mHolder;
Activity mActivity;
public ArDisplayView(Context context, Activity activity) {
super(context);
mActivity = activity;
mHolder = getHolder();
mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
mHolder.addCallback(this);
}
The ArDisplayView class begins with a constructor. We’ve added the Activity as a configuration parameter (more on this in an upcoming step). The constructor must initialize the underlying SurfaceHolder, which controls the drawing on the screen.
(One note here is that although the setType() method is deprecated and the Android SDK documentation says it is no longer necessary, we have found that the Camera code will not function without it, it simply fails to start up, even on the newest devices.)
We’ve added the appropriate member variables for working with a Camera, and we also keep the Activity around so that we can react to orientation changes and any other convenient access to the Context or Activity that we may need.
Step 3: Implementing the SurfaceHolder Callbacks
Next, you’ll need to implement the SurfaceHolder callbacks within the class. Update your ArDisplayView class so that it implements the SurfaceHolder.Callback interface, like this:
public class ArDisplayView extends SurfaceView implements SurfaceHolder.Callback
{
//…
}
This will necessitate overriding and implementing three new methods: surfaceCreated(), surfaceChanged(), and surfaceDestroyed().
When working with the Camera, there’s a set of steps needed to startup and shutdown. These steps are clearly documented in the Android SDK docs for the Camera class. We basically drop these steps into the appropriate SurfaceHolder callbacks.
Step 4: Implementing the surfaceCreated() Callback
We start with the surfaceCreated() callback method. Here we request the Camera, set the camera’s display orientation based on the current orientation of the device (so that the camera preview always displays right-side up), and call the setPreviewDisplay() method to tie the Camera preview to our SurfaceHolder.
public void surfaceCreated(SurfaceHolder holder) {
mCamera = Camera.open();
CameraInfo info = new CameraInfo();
Camera.getCameraInfo(CameraInfo.CAMERA_FACING_BACK, info);
int rotation = mActivity.getWindowManager().getDefaultDisplay().getRotation();
int degrees = 0;
switch (rotation) {
case Surface.ROTATION_0: degrees = 0; break;
case Surface.ROTATION_90: degrees = 90; break;
case Surface.ROTATION_180: degrees = 180; break;
case Surface.ROTATION_270: degrees = 270; break;
}
cam.setDisplayOrientation((info.orientation - degrees + 360) % 360);
try {
mCamera.setPreviewDisplay(mHolder);
} catch (IOException e) {
Log.e(DEBUG_TAG, "surfaceCreated exception: ", e);
}
}
Step 5: Implementing the surfaceChanged() Callback
Much of the interesting code for the Camera preview happens in the surfaceChanged() callback. Here we request the camera parameters and check what the supported preview sizes are. We need to find a preview size that can be accommodated by the surface, so we look for the closest match and use it as our preview size. Next, we set the camera preview format, commit the camera parameter changes, and finally call the startPreview() method to begin displaying the camera preview within our surface.
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height)
{
Camera.Parameters params = mCamera.getParameters();
List<Size> prevSizes = params.getSupportedPreviewSizes();
for (Size s : prevSizes)
{
if((s.height <= height) && (s.width <= width))
{
params.setPreviewSize(s.width, s.height);
break;
}
}
mCamera.setParameters(params);
mCamera.startPreview();
}
Step 6: Implementing the surfaceDestroyed() Callback
We end with the surfaceDestroyed() callback method. Here we shut down the camera preview and release the Camera resources.
public void surfaceDestroyed(SurfaceHolder holder) {
cam.stopPreview();
cam.release();
}
Step 7: The Custom Camera View and the Main Activity
Now you need to add your custom views to the FrameLayout control. To do this, update the onCreate() method of the ArtutActivity class, like this:
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.main);
FrameLayout arViewPane = (FrameLayout) findViewById(R.id.ar_view_pane);
ArDisplayView arDisplay = new ArDisplayView(this);
arViewPane.addView(arDisplay);
}
This adds the camera preview view class, called ArDisplayView to the FrameLayout. If you run the application now, the activity starts up and you should see a camera preview on the screen.
A note on figures: The default DDMS screen capture does not capture the camera preview. Run the app yourself to see the true functionality.
Part 3: Working with the Device Sensors
Our next step in developing an AR app involves accessing the device sensor data. We also need a way to overlay content on top of the camera preview. We do this by implementing a custom view that can be layered on top of the ArDisplayView currently displayed in the FrameLayout control of the activity.
Step 1: Displaying Overlay Content
First, you need to define a custom view control to draw overlay content in. Create a new class called OverlayView that is derived from the View class. This class starts out simple for this part of the tutorial. We simply want to overlay some sensor data on top of the camera view (no fancy math yet). To do this, we create member variables to store the last known sensor data for each sensor we are interested in. We then implement the onDraw() method of our view and draw some text on the canvas.
public class OverlayView extends View {
public static final String DEBUG_TAG = "OverlayView Log";
String accelData = "Accelerometer Data";
String compassData = "Compass Data";
String gyroData = "Gyro Data";
public OverlayView(Context context) {
super(context);
}
@Override
protected void onDraw(Canvas canvas) {
super.onDraw(canvas);
Paint contentPaint = new Paint(Paint.ANTI_ALIAS_FLAG);
contentPaint.setTextAlign(Align.CENTER);
contentPaint.setTextSize(20);
contentPaint.setColor(Color.RED);
canvas.drawText(accelData, canvas.getWidth()/2, canvas.getHeight()/4, contentPaint);
canvas.drawText(compassData, canvas.getWidth()/2, canvas.getHeight()/2, contentPaint);
canvas.drawText(gyroData, canvas.getWidth()/2, (canvas.getHeight()*3)/4, contentPaint);
}
}
The onDraw() method does all the work here. We configure a Paint with the appropriate text attributes and then draw some text on the canvas using the drawText() method.
Why use a custom view instead of regular controls? We'll want precision later on for placing AR objects within the camera view that are representative of locations of real world places. An alternate to this method is to use 3D rendering to better get a handle of depth and distance. In a location AR app, that's usually less important as the distances are often pretty vast (kilometers, not meters).
Step 2: Implementing the SurfaceHolder Callbacks
Next, you’ll need to implement the SensorEventListener interface within the class. Update your OverlayView class so that it implements the SensorEventListener interface, like this:
public class OverlayView extends View implements SensorEventListener {
This will necessitate overriding and implementing two new methods: onAccuracyChanged() and onSensorChanged(). For the purposes of this tutorial, we are really only interested in the onSensorChanged() method.
Step 3: Registering for Sensor Data Updates
Before we implement the SensorEventListener callbacks, we want to register for the appropriate sensor data updates. To do this, update the onCreate() method of the OverlayView class.
Begin by requesting the SensorManager and retrieving the sensors you want the view to watch. In this case, we’ll watch the accelerometer, the compass, and the gyroscope. Next, register to listen for events on each of these sensors.
SensorManager sensors = (SensorManager) context.getSystemService(Context.SENSOR_SERVICE);
Sensor accelSensor = sensors.getDefaultSensor(Sensor.TYPE_ACCELEROMETER);
Sensor compassSensor = sensors.getDefaultSensor(Sensor.TYPE_MAGNETIC_FIELD);
Sensor gyroSensor = sensors.getDefaultSensor(Sensor.TYPE_GYROSCOPE);
boolean isAccelAvailable = sensors.registerListener(this, accelSensor, SensorManager.SENSOR_DELAY_NORMAL);
boolean isCompassAvailable = sensors.registerListener(this, compassSensor, SensorManager.SENSOR_DELAY_NORMAL);
boolean isGyroAvailable = sensors.registerListener(this, gyroSensor, SensorManager.SENSOR_DELAY_NORMAL);
Now you’re ready to implement the sensor callback, onSensorChanged(), in order to react to these sensor updates.
Step 4: Implementing the onSensorChanged() Callback
Much of the interesting code for the sensor updates happens in the onSensorChanged() callback. Here we need to have the view react to sensor changes. We’ve registered for several different types of changes, so this method may be called for any sensor change. For now, we simply want to save off the sensor data that changed and then force the view to update its content. We can do this by simply invalidating the view, and forcing it to be redrawn.
public void onSensorChanged(SensorEvent event) {
StringBuilder msg = new StringBuilder(event.sensor.getName()).append(" ");
for(float value: event.values)
{
msg.append("[").append(value).append("]");
}
switch(event.sensor.getType())
{
case Sensor.TYPE_ACCELEROMETER:
accelData = msg.toString();
break;
case Sensor.TYPE_GYROSCOPE:
gyroData = msg.toString();
break;
case Sensor.TYPE_MAGNETIC_FIELD:
compassData = msg.toString();
break;
}
this.invalidate();
}
Step 5: Your Custom Overlay and the Main Activity
Now you need to add your custom view to the FrameLayout control. To do this, update the onCreate() method of the ArtutActivity class. Add the following code just below the addView() call that adds the ArDisplayView:
OverlayView arContent = new OverlayView(getApplicationContext()); arViewPane.addView(arContent);
This stacks the two views: first the camera preview view class, called ArDisplayView, and then overlays the OverlayView which displays the sensor data on top of it. If you run the application now, the activity starts up and you should see a camera preview on the screen with sensor data displayed on top of it (recall you cannot take screenshots of the camera view, so the following figure only shows the overlay content).
If you are running the app at this point, you've probably noticed that the sensor data updates very quickly, despite asking for a rate slow enough to work with orientation changes. These device sensors have a very high degree of sensitivity. Ultimately, this will lead to shakiness in the UI. We'll eventually need to implement some smoothing algorithms to keep the jitters away when we begin to use the sensor data within our application.
You might also be wondering if we'll be using all of these sensors and what their units are. We won't necessarily be using all of the sensors, but we thought we'd show you what they look like, and how to access them, in case your particular AR interests lay with different sensors.
- The accelerometer numbers are in SI units (m/s^2, i.e. meters per second squared, where Earth’s gravity is 9.81 m/s^2).
- The magnetic sensor units are in micro-Teslas. As they come in x, y, and z forms we can measure the vector and compare to gravity to determine where (magnetic) north is with respect to the device.
- The gyroscope measures rotation around each axis in radians per second. This can be used to calculate the relative device orientation, too.
Conclusion
This concludes the first part of our AR tutorial. Augmented reality is an exciting genre for applications on the Android platform. The Android SDK provides everything necessary for developing sleek and interesting AR apps, but not all devices can meet the hardware requirements that those apps demand. Luckily, the newest generation of Android devices is the most powerful yet, and many of the Android devices out there are perfectly capable of running advanced AR applications.
In the next part of this tutorial series, we'll move on to collecting location data and placing AR objects on the screen in a specific location. This will involve various mathematical algorithms, but the end result will be…augmentation!
Comments