Testing is often neglected in programming, and web development is no different. Many developers have not yet realized that automated tests can make you more productive, less stressed, and more confident about coding the next feature. In this article, we'll focus on using Selenium to automate browser testing.
As web developers, we need tests of some sort, because we certainly don't want bug reports from the users of our applications to be our means of testing. We want tests to be automated because manual testing, although sometimes a necessary evil, is slow, error-prone and boring. Repeated manual testing of a Web application in multiple browsers can be, frankly, soul destroying! A tool like Selenium can get you addicted to automated testing.
Get Test-Infected
Perhaps you can relate to this experience: you open up your project with the intent of coding a new feature or fixing a bug, and you wonder, "Could the changes I'm about to make have unintended side effects? Will I break my code?"
This fear of making changes only get worse as the project progresses, and it often ruins the fun of coding.
But, if you have a good set of automated tests and run them frequently, you stand a good chance of knowing very quickly if you have broken your code. This gives you a sense of confidence rather than one of fear, which allows you to simply get on with what you need to do, whether that be implementing new features, bug fixing, or refactoring. It's very refreshing.
This is easier to grasp when you've been through the pain of programming without good tests. It's tempting to think, "I just want to get on with coding the next part of my application." This is often more so the case when you are working on something relatively simple. But as any developer can tell you, things can quickly become more complex. Suddenly it's scary to modify the code, and that's when you really appreciate a comprehensive set of tests to back you up.
But reducing fear is only one benefit. Well written tests act to document the system under development, and this promotes better a understanding among developers and customers. By looking at a test you should be able to tell exactly how a particular aspect of the system should behave. This is a concept emphasized by Behaviour-Driven Development (discussed later).
An important idea is that considering how to test your application is as important as how you go about building it. That's worth restating: thinking about how to test your system is as important as how you actually write the application code.
It's a major shift in thinking, but once you are in the mindset of seeing automated tests as a fundamental part of programming, and have reaped the benefits, you will never look back. I became hooked on testing while being introduced to TDD, but to my mind being test-infected doesn't necessarily come through TDD or unit testing. You just have to have experienced the enormous value of automated tests and feel weird about programming if not in the routine of writing them.
Once you are in the mindset and have reaped the benefits, you will never look back
A response to these arguments might be, "This all sounds like something that would take a lot of time; time that could be coding the next feature." After all, we normally have limited time to spend on a project. And it's true, setting up and composing automated tests does take time and effort. But the amount of time it saves in the long run, and the improved quality it tends to bring to code, makes a rigorous routine of automated testing well worth the investment.
We'll be using a free tool called Selenium. Selenium automates browsers; it simulates a user interacting with your Web application, performing mouse clicks, text entry, and even drag-and-drop (among other things). It can also be used to check what is displayed on-screen.
Knowing how to write good tests is a skill which you develop over time, but in this tutorial we will be discussing getting started with browser testing using Selenium.
A View of Testing From 10,000 Feet
If you're new to testing, it's useful to get a general idea of the kinds of tests commonly in use. Different types of tests are used for different purposes. Bear in mind that the terminology surrounding testing is somewhat inconsistent—different people use the same term to mean slightly different things.
Unit tests are used to check the correctness of individual classes, methods and functions. The code being exercised should be kept isolated from other parts of the system and this is achieved using substitutes for things that the code under the test depends on. That way, it's easy to see where the problem occurs when a test fails. Unit tests tend to be the fastest tests to run, and no code involved should do things like hit a database or the access the network.
Unit tests shouldn't be concerned with verifying that individual components of the system work together properly; that's where integration tests come in.
Low level integration tests might deal with the interaction between two or three classes, while others might check that code works properly with external resources, for example a database or HTTP server.
System tests, which is where this tutorial fits in, are run against the whole, integrated system to check if the requirements of the entire system are met. System tests can concern things like performance and scalability, but the kind of tests we will focus on relate to whether or not the system behaves as the customer expects and implements the features they have specified. In Agile development circles these tests fall into the category of acceptance tests.
The example code presented below does this kind of testing. These tests tell us whether or not our application behaves the way we want it to, from the user's point of view. We can use Selenium to automate tests of this kind because it can simulate a user interacting with the system (and it can do so using real Web browsers, as well as headless systems like HtmlUnit).
Because we will be only interested in what the system does, and not how it does it, we will be engaged in black box testing. It's also worth noting that, in contrast with most other kinds of test, acceptance tests should be written in collaboration with customers.
No Need to Choose
Which kind of tests should you use?
We can use Selenium to automate tests because it can simulate a user interacting with the system
Cake is a kind of food but most people (not me) would advise against eating it exclusively; it complements rather than replaces other food. It's important to note that the various types of testing complement each other rather than being in competition. As mentioned above they serve different purposes. Each of them has advantages and disadvantages and they are certainly not mutually exclusive.
System-level, GUI-driven tests such as the examples below tend to be relatively slow to run and so don't provide quick feedback. Tests of this kind also have a tendency to be brittle, and because they touch so much of the application code, tracking down the source of a failure can be difficult without an accompanying comprehensive set of unit and integration tests. In fact it's a good idea to have many more unit-level tests than the kind of GUI-based, system-level, tests that Selenium is used for. That's not to say Selenium tests aren't useful! The point is, no one type of testing is enough on its own.
Two is Better Than One
We'll be using Selenium 2. More specifically we'll use WebDriver, a component of Selenium 2. WebDriver replaces Selenium 1's Remote Control (RC) API, and it offers a number of advantages over RC. For example, it's better for testing AJAX and it has a cleaner, more object-oriented, API. It also works in a completely different way to RC. Rather than using JavaScript to interact with a page, WebDriver uses the browser automation interface that is particular to each browser. The result is that it better simulates an actual user interacting with the Website under test.
Another component of Selenium is IDE, a record-and-playback tool and Firefox plugin. It doesn't require knowledge of programming and is useful for exploratory testing.
Its tests tend to be more brittle than RC and WebDriver scripts and an obvious big drawback is that it can only be used in Firefox. IDE is intended as a prototyping tool and is not recommended for serious testing.
WebDriver supports a wide variety of browsers including Chrome, IE, iOS and Android. Later we'll look at the use of cloud testing services in order that tests may be run against browser-operating system combinations that you may not otherwise have access to.
Here, WebDriver will be used with Python, but a number of language bindings are available including those for Java, C# and PHP. If you're unfamiliar with Python, fear not, you should still be able to follow along with the examples as it reads pretty much like pseudo-code.
Python...reads pretty much like pseudo-code
A number of other interfaces are available but the two key parts of the WebDriver API we'll need are WebDriver and WebElement. Each example below will work with a WebDriver object, which corresponds to the browser, and one or more objects of type WebElement, which represent elements on a page.
The methods for locating elements on a page (discussed later) are common between these two interfaces. On the other hand, methods such as tag_name are only available on WebElement. Similarly it makes sense for methods like get_cookies and refresh to be available on WebDriver but not on WebElement, and this is indeed the case.
It's interesting to note that there is an effort to make WebDriver a W3C standard.
Grab What You Need
Currently Selenium 2 supports Python 2.6 and Python 2.7, so install one of these if you need to. To find out which version you have, at the command line type python -V. Linux and Mac users normally have Python already, but should take care when upgrading their version of Python as the operating system may depend on the version the OS came with.
Once you have Python 2.6 or 2.7, the best way to install packages for it is with pip. Once you have pip, to install Selenium 2 type: pip install -U selenium. (-U will upgrade any previous version you might have. Linux and Mac users might need sudo).
To get pip on Windows, see this Stack Overflow question.
We'll also use Firefox, as that's the browser that works with WebDriver out of the box.
You'll Never Guess
We need a Web application to test, and we'll use a simple number guessing game. It is a deliberately simple program. A Web application is often tested on a developer's machine using a locally run development Web server, as this is convenient for testing before deployment. However in this instance we'll be running tests against a deployed Web app: http://whats-my-number.appspot.com. This will be the application under test (AUT). (In the event that this site is down, try http://whats-my-number-backup.appspot.com/).
The answer (sorry to spoil the fun) is 42.
Whatever the user's input, a hint should be displayed. The program expects whole numbers from 0 to 100 (inclusive) and should the user enter a value that does not fit this rule the hint should advise of this requirement. When the user tries a whole number guess from 0 to 100 that is incorrect, the hint shown should be either "too low" or "too high". When 42 is entered, "Congratulations" should be the hint displayed.
Something we touched on earlier is the idea that a great way to be explicit about how a system should behave is to write tests, and later examples will involve a fairly comprehensive set of tests which will act to communicate the system's intended behavior. We will have a form of executable documentation.
We will have a form of executable documentation
One of the great things about a language like Python is that you can use an interactive interpreter. To run the interactive Python interpreter simply type python at the command line, and you should see its prompt (>>>). Alternatively, to run a script file, use python script_name.py
It's not the way test code is typically run of course, but when you're just getting started with browser automation it can be helpful to use the interactive interpreter and type in a line of Python at a time. This way it's easier to get a feel for how WebDriver controls the browser and simulates a real user. Although you can instead run a script file and sit back and watch while Selenium does its thing, things work at a much faster rate than with a human user, so running commands one line at a time makes it easier to get a good appreciation for what the commands you are issuing are actually doing. It's a great way to learn and experiment.
Can We Actually Do Some Testing Now?
Type the following lines of code at the interpreter's prompt, pressing Enter after each. The first step is to perform an import:
from selenium import webdriver
Next, we open a browser window and visit the AUT:
browser = webdriver.Firefox()
browser.get('http://whats-my-number.appspot.com/')
Now we'll do something that makes this a test. Python's built in assert statement can be used to check that something is true, and in this case we use it to verify that the page title is "What's My Number". This may be referred to as a content test:
assert 'What\'s My Number?' == browser.title
Since the page title is correct, Python simply gives us another prompt. The title being incorrect would have meant assert throwing an AssertionError. An AssertionError while running a script file causes the program to crash (which is useful).
The next part of our test is what the Selenium documentation calls a function test. We want to verify that when 1 is entered as a guess, the program responds with content that includes a hint stating that the guess is too low. Later examples will deal with multiple user entries.
To do this we must fill out the form. If you look at the HTML of the guessing game page you'll see that the text input field has a name attribute with the value of 'guess'. This can be used to obtain a WebElement object to represent the input field:
guess_field = browser.find_element_by_name('guess')
We can now type in the guess. WebElement has a send_keys method:
guess_field.send_keys('1')
We could find the submit button and click it, or use the provided submit method, but instead let's press the Return key:
from selenium.webdriver.common.keys import Keys guess_field.send_keys(Keys.RETURN)
The form is submitted and the page is reloaded (AJAX is not in use), and since the guess is too low, "Your guess is too low" should be displayed somewhere in the body of the document. To verify this first we need a WebElement object that represents the HTML body:
body = browser.find_element_by_tag_name('body')
The text property of WebElement will in this case reveal the text of the page. Let's make use of that in an assert statement:
assert 'Your guess is too low' in body.text
Again, success, so Python simply gives us another prompt. Since this is an incorrect guess, "Congratulations" is nowhere to be seen:
assert 'Congratulations' not in body.text
Finally we close the instance of Firefox we've been using:
browser.quit()
Target Acquisition
If you're familiar with programming using JavaScript and the DOM you'll know about the need to obtain references to DOM elements on a Web page, and as we've seen we need to do something similar here. The two situations aren't exactly the same however, because rather than getting a reference to a DOM element, we get a WebElement object that corresponds to a DOM element.
Above we used
find_element_by_name, which is useful for form elements, as well asfind_element_by_tag_name. Other locator methods includefind_element_by_idandfind_element_by_css_selector. See the Selenium documentation for the complete list.
Performance-wise, using an element ID or name locator (as we did above) is the best way to select an element. Of course, once we have a WebElement object that corresponds to the desired DOM element it's common to want to interact with it in some way, which is where methods like send_keys and click are useful.
Brittle Can Be Dangerous
Brittle tests are dangerous because, if tests sometimes fail when in fact they should pass, you come to ignore the tests results, and the whole process of testing becomes devalued.
In the downloadable zip file accompanying this tutorial, ftests1.py lists the example test code above in the form of a script file. However there is an omission: you might notice that the call to implicitly_wait, included in ftests1.py, was not listed or discussed.
If you run a test against a system ten times, it should give you the same result each of those ten times. However, brittle and unreliable tests of the kind we are doing are quite common, and you might well come across this problem as you experiment with Selenium testing. Brittle tests are dangerous because, if tests sometimes fail when in fact they should pass, you come to ignore the tests results, and the whole process of testing becomes devalued. implicitly_wait is a very useful tool in combating brittle tests, and from this point a call to implicitly_wait will be used in all of the example code.
I Thought You Said We're Not Unit Testing
Being a test-infected developer, you'll want to know about the xUnit tools. They are available for many programming languages. unittest is an xUnit tool that comes as standard with Python. It may seem confusing but, although we're not writing unit tests, unittest is useful. For example, unittest helps with structuring and running tests, and test failures result in more helpful messages.
The version of unittest in Python 2.7 has additional features as compared to older versions (some of which we'll use), so if you're running Python 2.6 then you'll need to install the backport: pip install unittest2
The code below is a unittest version of the test code presented earlier.
As before, the title of the browser window is checked, 1 is tried as a guess, and the program's response is checked:
try:
import unittest2 as unittest #for Python 2.6
except ImportError:
import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
class GuessTest(unittest.TestCase):
def setUp(self):
self.browser = webdriver.Firefox()
self.browser.implicitly_wait(3)
def tearDown(self):
self.browser.quit()
def test_should_see_page_title(self):
# Brian visits the guessing game website
self.browser.get('http://whats-my-number.appspot.com/')
# He sees that "What's My Number?" is the title of the page
self.assertEqual('What\'s My Number?', self.browser.title)
def test_should_get_correct_hint_from_guess_too_low(self):
# Brian visits the guessing game website
self.browser.get('http://whats-my-number.appspot.com/')
# He types his guess into the form field and hits the return key
guess_field = self.browser.find_element_by_name('guess')
guess_field.send_keys('1')
guess_field.send_keys(Keys.RETURN)
# The page is reloaded and since the guess is too low,
# 'Your guess is too low' is displayed
body = self.browser.find_element_by_tag_name('body')
self.assertIn('Your guess is too low', body.text)
# Since this is an incorrect guess, 'Congratulations' is nowhere to be seen
self.assertNotIn('Congratulations', body.text)
if __name__ == '__main__':
unittest.main()
Brain is the name of our "robot user". See also ftests2.py in the zip file accompanying this tutorial.
The individual tests are methods of the class GuessTest, which inherits from unittest.TestCase. For more of an idea about the self keyword and other object oriented aspects of Python see the Nettuts Session on Python. Test method names must begin with the letters test. It's essential to make method names descriptive.
Of course an assert is essential to any test, but in fact rather than using the assert statement as before we have access to unittest's assert methods. In this case assertEqual, assertIn and assertNotIn are used.
The setUp and tearDown methods are executed before and after each of the test methods, and here we use them to start up and shut down a WebDriver browser instance.
The final block, if __name__ == '__main__': unittest.main(), allows this unittest script to be run from the command line. To run the script go to the directory containing ftests2.py and type: python ftests2.py. Doing should result in output like this:
Ideally, tests should fail "noisily" but pass "quietly", and as you can see that's exactly what unittest is doing: only a period is printed for each passing test method. Lastly we see a welcome "OK" (shouldn't it be "Well done"?).
As you can see, the "Don't Repeat Yourself" principle is being violated, in that the URL of the AUT is in the code twice. Thorough testing allows the refactoring of application code, but don't forget to also refactor test code.
Guess Again
So far our tests have only involved a single guess: 1, and clearly this is not very comprehensive. The next script will do something about this, see ftests3.py in the zip file.
The
importstatements, class declaration,setUpandtearDownmethods, and theif __name__ == '__main__':block, are all exactly the same as the last example. So let's concentrate on the things that are different.
As this is something we'll do repeatedly, filling out the form has been put into its own helper method, named _enter_guess_hit_return:
def _enter_guess_hit_return(self, guess):
guess_field = self.browser.find_element_by_name('guess')
guess_field.send_keys(guess)
guess_field.send_keys(Keys.RETURN)
Another helper method, _unsuccessful_guess, deals with visiting the AUT, calling _enter_guess_hit_return, and calling the assert methods. Again, our robot user could do with a name, this time let's refer to it as Bertie.
def _unsuccessful_guess(self, berties_guesses, expected_msg):
self.browser.get('http://whats-my-number.appspot.com/')
for berties_guess in berties_guesses:
self._enter_guess_hit_return(berties_guess)
body = self.browser.find_element_by_tag_name('body')
self.assertIn(expected_msg, body.text)
self.assertNotIn('Congratulations', body.text)
You might notice that calling _enter_guess_hit_return and performing the asserts happens in a loop. This is because we are looping over berties_guesses, which is a list. berties_guesses will be passed to this method by the calling test methods, which will also pass in an expected message, expected_msg.
Now to make use of our helpers in the test methods:
def test_should_get_correct_hint_from_guess_too_low(self): berties_guesses = ['0', '01', '17', '23', '041'] expected_msg = 'Your guess is too low' self._unsuccessful_guess(berties_guesses, expected_msg) def test_should_get_correct_hint_from_guess_too_high(self): berties_guesses = ['43', '80', '100'] expected_msg = 'Your guess is too high' self._unsuccessful_guess(berties_guesses, expected_msg) def test_should_get_correct_hint_from_invalid_input(self): berties_guesses = ['a', '5a', 'c7', '1.2', '9.9778', '-1', '-10', '101', 'hkfjdhkacoe'] expected_msg = 'Please provide a whole number from 0 to 100' self._unsuccessful_guess(berties_guesses, expected_msg)
For brevity, checking the page title has been dispensed with. Of course, there should be a method to verify that when the correct guess is provided, "Congratulations" is indeed displayed, and you are invited to write this method (it'll be fun, I promise!).
Whoops, I Pressed The Red Button
The last example script gives us a good degree of confidence that the AUT functions as it should. But suppose that the application's code must now change. For example, the customer wants a new feature, or we want to refactor, or perhaps unit or integration tests have uncovered a fault that system level tests did not reveal (and we now wish to fix that fault). During the process of modifying the code, existing tests should be run frequently so problems show up sooner rather than later.
Let's simulate a change to the application code. A modified version of the game is at http://whats-my-number-broken.appspot.com and if you run ftests3.py against this version you'll see a test failure:
test_should_get_correct_hint_from_guess_too_high is failing. The test shows that in modifying the application's code a regression was introduced. We run the tests regularly, and we only have to consider the changes that were made since the tests were last passing to narrow down the problem. In this way writing tests has rewarded us with a sense confidence, as opposed to a sense of fear.
"It Works on My Machine"
Web applications are generally expected to function properly on a wide variety of browsers, so it's normally best to test with as many browsers on as many platforms as you can get your hands on. When a problem with a system is discovered it's not uncommon to hear a developer say: "Well, it works on my machine". This often amounts to: "We've not tested it properly". In the case of the number guessing game you may wonder whether cross-browser testing is required, but of course it is a deliberately simple system.
Cloud-based services such as Sauce Labs can help. Sauce Labs offers a variety of browser-OS combinations. Another service is Testingbot, which offers testing on mobile platforms.
As you've seen, we're running tests against a publicly accessible site, but for sites still in development and intranet sites, Sauce Labs offers Sauce Connect and Testingbot offers Tunnel.
The code samples so far have been hard-coded to use Firefox. ftests3_remote.py, available in the zip file, is an enhanced version of ftests3.py that may be easily configured to run using a given browser-OS combination (within the limits of what is offered by whichever cloud testing service we use). The platform, browser and browser version are specified on the command line when the script is run.
The code samples so far have been hard-coded to use Firefox
You'll need to sign up for a service like Sauce Labs or TestingBot to obtain an API key, and edit the setUp method (as shown in the file) to include this key. Both services can be tried without charge.
ftests3_remote.py expects the platform as the first command line argument, the name of the required browser second, and the browser version is expected last. With reference to Sauce Lab's available browser-OS combinations we could, for example, run the script as follows:
python ftests3_remote.py LINUX chrome
In the particular case of Chrome no version number should be specified. To use Internet Explorer, because the browser name consists of two words, quotes must be used. As an example, let's run the tests using our old friend, IE6:
python ftests3_remote.py XP 'internet explorer' 6
Test results are output to the terminal just as if you were running the tests on your own machine. However, you can expect this script to run more slowly than previous example test scripts. Interestingly, Sauce Labs lets you watch a video of each test running once it's complete.
A shell script could easily be created to run ftests3_remote.py a number of times, each time with a different platform-browser-version combination.
Earlier we looked at the need for implicitly_wait. It's interesting to note that the value passed to implicitly_wait as suggested by Sauce Lab's own example code is 30 seconds.
Try And Behave
Behaviour-Driven Development (BDD) extends TDD and emphasizes a clear understanding of the desired behavior of a system through good communication between developers and users. A tool like behave can support this collaboration and help avoid misinterpretation by developers. behave will provide us with an abstraction layer on top of Selenium to make tests more customer-readable.
A tool like behave can support collaboration and help avoid misinterpretation
behave uses the Gherkin language, familiar to users of Cucumber in Ruby and Behat in PHP. The words Given, When and Then are used to aid the communication process and create plain-text descriptions of what the system should do. We can back these descriptions with Python, and the result is that the plain-text descriptions can be executed as automated tests.
A full explanation of behave is outside the scope of this tutorial, and you're encouraged to look at the documentation. To run the example test you'll need to install behave: pip install behave
The zip file accompanying this tutorial includes the directory behave/features, which contains valid_inputs.feature, steps/steps.py, and environment.py. There isn't space here to examine each of them in detail.
valid_inputs.feature contains some plain-text descriptions of how the guessing game should function. The phrases beginning "Given", "When", "Then" and "But" map to step implementations, which are included in steps/steps.py.
environment.py can define code to run before and after certain events and in this case it's used to start and stop a browser session.
To run the tests switch to the behave directory and simply type behave at the command line.
The TestingBot site has a page about how to use behave with their cloud testing service.
Conclusion
It's important to think of testing as an integral part of coding. The various types of testing complement each other, and having a comprehensive set of tests gives developers the confidence to fix bugs and develop new features in the knowledge that the quality of code can be kept high.
Webdriver offers an API usable with a number of languages, and a tool like behave can be useful for writing tests in collaboration with customers. We've also seen how it's possible to test a Web application in a variety of browser-OS combinations using one of the cloud testing services on offer.
This has been only an introduction; hopefully, you'll be inspired to dig deeper!
Comments