Introduction
Testing an app on Android or iOS isn't all that different. The purpose is the same, the desired outcome is the same, and the process is similar. The major difference comes when we begin to look at the details. That's what I plan to do in this article.
1. Fundamentals
Before we dive in, let’s talk about some testing fundamentals. It’s impossible to review our options unless we know and understand the complete picture.
Challenges on Android
What makes Android stand out is the myriad of possibilities. On iOS, there's iPhone, iPad, and iPod Touch. They're different, but the common factor between iOS devices is pixel density, screen resolution, processor speeds, memory size, etc.
In the case of Android, there are thousands of combinations when we look at those same factors, screen resolution and size, processor speeds, memory size, and, the cherry on the cake, the fragmentation of the operating system.
Speaking of operating system versions, it's not uncommon for carriers and phone manufacturers to stop pushing out updates not too long after the product's release. Is this a problem? Yes. Take a look at Google’s official Android market share statistics to get an idea of the problem's scale.
In descending order of market share, we have Jelly Bean (4.1-4.3), Gingerbread (2.3), and Ice Cream Sandwich (4.0).
Compare that with the adoption rate of Apple's iOS 7. At the end of January of this year, 80% of iOS devices were running iOS 7. Mind you that iOS 7 was released in September of 2013. That's a major difference.
Study, Contrast & Compare
Have you ever used a really bad Android application? Worse than an outright bad application is a really great one that has a few persistent bugs.
I feel a large factor in good testing is paying attention to what you use, like, and hate. Hate is a strong word, but I’m sure there's something that always stands out.
Ask yourself the following questions:
- What are some of my favorite apps? Why is that?
- What are some bad apps that I use?
- What makes an app great? Is it attention to detail?
- Does that bad app freeze from time to time? Does it constantly crash? Or is its design no good?
Know What You're Working With
Let's reference that Android OS market share chart that we saw earlier. It's unrealistic to approach testing thinking that you'll support every device and every flavor of Android.
My point is that we need to think about the distribution. What does our app do and what does the target market look like? Is it a game or a utility application?
If it’s a game, the focus may likely be only newer and higher-end devices. A utility application, however, could be used by a broader audience and needs to function on a wider array of devices.
2. Approach
A problem I feel most of us run into is that we're too close to our projects. We know how we can make our app fail and how to get it to work. For this reason, I intentionally try to put my mind into that of a user. I put users in two broad categories, the Button Masher and the User.
Button Masher
The Button Masher is the type of user that will just start tapping the screen, a button here, a button there. "That last button didn't do anything. I'll hit another one."
What we can learn from this user type is where we have intensive processes within our app. If something is happening and another request or action occurs, do we spike the processor or fill up the device's memory? Does it cause the application to crash?
The other question that surfaces is "How well do we inform the user of what's going on." Why did they hit another button instead of waiting? Can we remedy this by showing a loading screen?
User
The User has intention. A better way to explain this type of user would be to look at use cases. There's a specific task that they're wanting to accomplish and a specific flow they'll try to follow.
We can learn how clear the app is at guiding a person through a process or action. It will show us where a user gets lost and what areas require more attention or refining.
We've talked through our purpose and different user types, but what are our options and how should we test them? Thankfully, there are a lot. And I recommend you do as many as you possibly can.
3. Options
Phone a Friend
If you don't have the luxury of being able to walk down to the QA department or a testing lab, you can call a friend. You need eyes and you need devices.
When it comes to testing mobile apps, volume can make a difference, especially if you can get a variety of devices.
Tools & Unit Testing
Automated testing is your friend. While nothing will beat hands-on time with a complete application, it's also important to see what's going on under the hood and how your app will react programmatically to certain situations or when put under stress.
More importantly, unit testing allows you to test as you go, which can save a lot of time during testing and QA, prior to release.
Android SDK
The Android SDK comes with the Android testing framework, which consists of a testing API based on JUnit and monkeyrunner.
The Android JUnit extension allows developers to write unit tests against Android components and the Android API with pre-built component-specific test case classes.
Monkeyrunner is a Python-based API that allows you to write programs that control a device from a user's perspective. This means that you can create tests to run on numerous devices or emulators that will interact with your app, sending it keystrokes and capturing screenshots.
Other Testing Frameworks
There are a lot of testing frameworks available. A few of the more popular ones are Robolectric and Robotium.
Robolectric is a unit test framework that runs in your IDE. This is great for auditing your code pre-build. Robotium runs tests against the Android API in emulators. While it takes more time to complete the tests, your application code will be put through much more of a real-world test against devices and the API.
One other interesting option is Espresso. It serves a very specific purpose when compared to the previous two options. It's an API to run tests against the Android UI.
All of the above options are great, but if you're building a hybrid app, you may not be able to use them. Appium is a cross-platform automation framework that allows you to build tests with whatever language you like for both major mobile platforms.
Reporting and Analytics
It's also really helpful to be able to see some statistics and, more importantly, collect error and crash logs. This is particularly useful if you have many people testing your application, because it can become cumbersome to collect logs from every single user.
Aside from tracking app usage, Google Analytics can also send you exceptions. Flurry is another tested option. They've been around for a long time and their reporting and crash reports are a bit more detailed.
While it doesn't help during the development phase of your application, Google also collects crash reports for apps in the Play Store.
4. Third Party Options
We'd all like to have 400 devices to test on, like those massive testing labs we've seen online. However, that isn't realistic. To answer this problem, there are many services available if you’re willing to invest in testing.
These services range from one-on-one real-human tests to fully-automated testings on hundreds of devices. If you’re willing to pay for it, it’s available.
I don't have experience with most of these, but the one I have used is User Testing. It's very helpful to see a person follow your test script as they tap through your application and give you their thoughts.
These are a few services to consider:
Conclusion
Too many times I've come across the situation where QA and testing seemed like an afterthought. In reality, it's probably the most important part of the development process.
Android, with its many flavors, may seem intimidating, but when approached almost programmatically, it really just becomes part of the process. It's worth the extra time and effort. Quality apps don’t just happen.
Comments