Unit Testing Succinctly: NUnit

This is an extract from the Unit Testing Succinctly eBook, by Marc Clifton, kindly provided by Syncfusion.

The following table maps the attributes used for writing tests with NUnit with Visual Studio:

NUnit Attribute

Visual Studio Attribute

Description

TestFixture

TestClass

Defines a test fixture

Test

TestMethod

Defines a test method within the test fixture class

TestFixtureSetUp

ClassInitialize

Specifies the code that runs before all test methods in the test fixture run.

TestFixtureTearDown

ClassCleanup

Specifies the code that runs after all tests in the fixture are complete.

SetUp

TestInitialize

Specifies the code to run prior to running each test.

TearDown

TestCleanup

Specifies the code to run at the completion of each test.

SetUpFixture (see the following)

AssemblyInitialize

Specifies the code to run when the assembly containing all the test fixtures is loaded.

SetUpFixture (see the following)

AssemblyCleanup

Specifies the code to run when the assembly containing all the test fixtures is unloaded.

Ignore

Ignore

Ignores the specific test.

Description (also applies to test fixtures).

Description

A description of the test method. In NUnit, this attribute can also decorate a test fixture.

The following Visual Studio attributes do not correspond to any NUnit attributes:

  • Owner
  • DeploymentItem
  • HostType
  • Priority
  • WorkItem
  • CssIteration
  • CssProjectStructure
  • TestProperty

The SetUpFixture Attribute

The SetUpFixture attribute, which applies to classes, is different from Visual Studio’s AssemblyInitialize and AssemblyCleanup because it applies to fixtures in a given namespace. If all your test fixture classes are in the same namespace, then this attribute does behave similarly to AssemblyInitialize and AssemblyCleanup. Given the code:

The resulting output is:

However, adding another namespace:

The ability to execute code specific to the namespace context has the advantage of being able to organize a suite of tests in different fixtures, but all in the same namespace that require a specific setup and teardown process.

Also see “Assembly Actions” under “User Defined Action Attributes,” in the following section.


Additional NUnit Attributes

NUnit has an extensive set of attributes that provide considerable additional functionality to unit testing.

Test Grouping and Control

  • Category
  • Suite
  • Explicit
  • Timeout

Category

The Category attribute allows you to group tests and run tests in selected categories. This attribute can be applied to tests in a fixture as well as individual tests within a fixture. Specific categories can be selected from the console runner using the /include and /exclude arguments or from the GUI runner:

NUnit Categories
NUnit Categories

The tab for selecting the categories to include or exclude is on the left edge of the GUI runner.

Suite

The Suite attribute provides a means of programmatically specifying the test fixtures that the console runner should test. This option is available only in the console runner. The concept of running only specific tests or test suites (fixtures) is supported better by the Category attribute described earlier.

Explicit

This attribute designates that a test is run only if the GUI or console runner is explicitly told to run the test by selecting the test. The test will also run if the test is part of a category of tests to run.

Uses for this attribute are to only run long-running tests when explicitly required, when a service is up and running, etc.

Timeout

This attribute can be used to ensure that a test runs only within the specified time in milliseconds. If the test exceeds the specified timeframe, the runner cancels the execution and reports the test as failed.

Compare this attribute with the MaxTime attribute described below.

Also note that, when running unit tests in the debugger, the Timeout attribute is ignored—otherwise, the test that you are debugging could terminate as you are manually stepping through code, inspecting values, etc.

Culture Attributes

  • Culture
  • SetCulture
  • SetUICulture

Culture

The Culture attribute can be used for a fixture to define the cultures for which tests should be run. Tests that are culture specific should then be decorated with this attribute as well, describing the cultures that they test. The test or fixture is skipped if the current culture does not match the culture for which the test is written. For example:

In the preceding code, the fixture runs if the current culture is either “fr” or “en”; however, TestA specifies that the test should only be run if the culture is “fr.” Therefore, this fixture will result in:

Culture Unit Testing
Culture Unit Testing

as “TestA” does not run because the current culture is not “fr.”

Tests such as “TestB” that omit any culture specification always run unless the entire fixture is excluded because the current culture does not match.

SetCulture

This attribute, applied to either the entire fixture or specific tests, sets the current Culture for the duration of the test (or tests in the fixture) and then restores the original value. Unlike the Culture attribute before this, only one culture can be specified, though according to the NUnit documentation, running tests for multiple cultures is planned as a future enhancement. Note that setting the culture in the fixture does not change whether the test using the Culture attribute runs. For example:

results in TestA not running. Also observe the default behavior of TestB when the culture is set in the fixture and how it is overridden in TestC:

Overriding in TestC
Overriding in TestC

SetUICulture

This attribute is supposed to set the culture for the user interface; however, it apparently does nothing. For example (don’t write a unit test like this, this is for illustration purposes only):

results in the value “1.2” being displayed in my current culture, being en-US:

SetUICulture Attribute Does Nothing
SetUICulture Attribute Does Nothing

If I change the attribute to “SetCulture,” then the user interface displays the value in the correct culture format (you may have to squint to see the difference between “1 point 2” and “1 comma 2”:

SetCulture Changes the UI Representation
SetCulture Changes the UI Representation

Parameterized Tests

Parameterized tests are ones in which the unit test describes the parameters and their values that will be used to run a test iteratively until all parameter combinations are run.

For example, this test:

results in the following output (and notice the test graph also displays the test parameters):

Parameterized Tests
Parameterized Tests

The following attributes are used for parameterized testing:

  • Values
  • ValuesSource
  • Combinatorial
  • Pairwise (not implemented)
  • Sequential
  • Random
  • Range
  • TestCase
  • TestCaseSource

Values

As illustrated in the previous example, the Values attribute is used to specify the values passed into the test method for each parameter. Because these are values assigned in an attribute, they must be of value type: constant expressions, typeof expressions, or an array of attribute types.

Also note that NUnit will by default test every combination of values for each parameter (see the attributes Combinatorial, Pairwise, and Sequential).

ValuesSource

With the ValueSource attribute, you can specify an IEnumerable source for the values of a specific parameter. The source can be a field, property, or method of a separate class or the current class. For example, these are all valid ways of expressing value sources:

Note the use of the Sequential attribute on the test, which ensures that the source values are used sequentially rather than combinatorially. Therefore the result is:

ValueSource Tests
ValueSource Tests

Combinatorial

This attribute is optional, as NUnit will automatically apply test parameters in all possible combinations. For example:

results in all combinations of “x” and “s,” whether or not “Combinatorial” is specified:

Combinatorial Test
Combinatorial Test

Pairwise

This attribute exists but is not implemented—it is intended to reduce the number of combinations when combinatorial parameters are specified; however, this attribute is ignored.

Sequential

This attribute specifies that tests will be run by sequentially stepping through parameter values. If the number of values is not the same for each parameter, the default value will be used. Note that the default value for a string is null. For example, this code, using the Sequential attribute:

results in this output:

The Sequential Attribute
The Sequential Attribute

Note how the string (having only two values) defaults to null for the last two parameterized tests, and the int parameter, having three cases, defaults to 0 for the last case.

Random

The Random attribute can be applied with combinatorial parameterized tests to provide random values rather than specific values for parameters. Random value parameters are always double types. For example:

results in five tests with random values:

Random Test
Random Test

A minimum and maximum can also be specified for the random value range.

It is easy to create hundreds, if not thousands, of test cases with combinatorial parameterized tests. Consider using the Sequential attribute to prevent large numbers of combinations from being tested. The Sequential attribute applied to the test can be used in conjunction with the Random attribute on the test’s parameters.

Range

The Range attribute specifies a range of values to test. Except for an integer range, other ranges (long, float, double) all require a step to be specified. The step for integer values is optional.

Ranges are combined with other parameterized values, so again, consider using the Sequential attribute to limit the number of test combinations that are generated.

TestCase

Rather than a combinatorial or sequential parameterized test, you can explicitly define the parameter values passed to each test case to run for a particular test. This has the advantage of reducing the number of combinatorial tests as well as targeting specific test scenarios. Furthermore, the TestCase attribute includes the additional characteristic of being able to test the return value (as long as it is a “value” type) to a value specified in the metadata of the attribute. Lastly, additional properties, such as mirroring attributes, are exposed in the TestCase attribute.

A basic test case looks like this (note the lack of the TestFixture attribute):

However, the real power of the TestCase attribute is in the ability to apply the test cases to the method directly and inspect the result:

Remember that the values specified in attributes must be value types: a constant expression, a typeof expression, or an array creation expression of an attribute parameter type.

Also beware that the TestCase attribute might encourage test cases to be placed in the same assembly as the production code rather than in a separate test assembly—attributes are not removed in the release build of an assembly. It is recommended that when using the TestCase attribute, you do not apply the attribute directly to the code for which you want the test cases but rather the test fixture itself. So, ideally, the correct code should look something like this:

in which the class MyMath is in the application’s assembly and the test fixture is in the unit test assembly. The DivideTest method then becomes a thin wrapper to the application’s method.

Other properties that can be assigned to the TestFixture are illustrated in this example:

There are also alternatives specifying the expected exception:

  • ExpectedExceptionName
  • ExpectedExceptionMessage
  • MatchType

which allows you to specify the full name (matching the value of Type.FullName property) or the exception message, as well as how the exception message should be matched—Contains, Exact, Regex, or StartsWith.

TestCaseSource

The TestCaseSource attribute behaves similarly to the ValueSource attribute described earlier; however, the preferred implementation is to specify a class type that implements IEnumerable. This preferred implementation constrains you to something like this:

Note how the test case values are an int[] (integer array), in which each element of the array maps to a parameter in the test function. Also note again that the Sequential attribute is specified for the test, ensuring that the test data is used sequentially rather than combinatorially.

Other NUnit Attributes

There are several other NUnit attributes, described next.

Platform

The Platform attribute can be used to specify the platform for which the test should run. If the platform does not match the current platform, the test is not run and is not even included in the ignored tests or the total number of tests. Review the documentation at http://www.nunit.org/index.php?p=platform&r=2.6.2 to see the list of supported platform specifiers.

Property

You can use this attribute to set a name or value pair on a test or fixture, which is then displayed in the XML output file. Via reflection, you can access these attributes. Custom property attributes can also be created by deriving a custom attribute class from the PropertyAttribute class.

MaxTime

The MaxTime attribute (if compared to the Timeout attribute described earlier) will cause the test to fail if it exceeds the specified time in milliseconds. However, unlike the Timeout attribute, the test will be allowed to finish.

Repeat

The Repeat attribute can only be applied to tests and is ignored for parameterized tests. It instructs the test engine to repeat the test the specified number of times. If any repetition of the test fails, the remaining tests are not run and the engine reports a failure.

RequiredAddin

With this attribute, you can specify additional assemblies (in the AssemblyInfo file) that the unit tests require.

RequiresMTA

This attribute, applied to a test or a fixture, indicates that the test should run in a multithreaded apartment. NUnit will create a new thread for the test if:

  • The assembly is not already running on an MTA thread.
  • The containing fixture is not already running on an MTA thread.

If a thread is created for the entire fixture, all tests in that fixture run on the new thread.

RequiresSTA

This attribute, applied to a test or a fixture, indicates that the test should run in a single-threaded apartment. NUnit will create a new thread for the test if:

  • The assembly is not already running on an STA thread.
  • The containing fixture is not already running on an STA thread.

If a thread is created for the entire fixture, all tests in that fixture run on the new thread.

RequiresThread

This attribute, applied to a test or a fixture, indicates that the test should run in a separate thread. NUnit will create a new thread if this attribute is specified in the AssemblyInfo file:

If the fixture specifies this attribute, all tests in the fixture run on a newly created thread.

If the test specifies this attribute, the test will run on a newly created thread.

Optionally, the apartment state can be specified as well, which causes either an MTA or STA thread to be created.

For example:

Notice the thread IDs when the test runs:

Thread Examples
Thread Examples

Test A and Test C run on a thread created by the fixture, whereas Test B, having specified the RequiresThread attribute, runs on a separate thread from that of the other tests in the fixture.

Theory and Datapoint(s)

A theory is a parameterized test that verifies assumptions about the values being passed in. If the assumption fails, the test does not fail. It works with the Datapoint(s) attribute.

These two classes are equivalent. The first illustrates the use of the Datapoint attribute, the second, the Datapoints attribute:

Note in the first class, TheoryExample1, discrete data point values are used, whereas in the second class, TheoryExample2, an array is used.

In both cases, NUnit will create all possible combinations of values from the data points. Note how the test method makes certain assumptions:

  • The denominator cannot be 0.
  • The expected result must equal n/d.

The reason for the second assumption is that the test will fail for a majority of the combinations—for example, 5/10 does not equal 2. As can be seen from the test run, the assumptions do not cause the test to fail:

Theories
Theories

The combinations of parameter values that do not pass the assumptions are omitted from the test. The typical pattern for a theory test is:

  • State the assumptions about the parameters.
  • Perform the test.
  • Make assertions about the results.

Care should be taken with theory tests because of the exploding number of combinations of arguments.

Also note that unless explicitly specified, NUnit will automatically populate the combinations of parameter values for bool and enum types.


User Defined Action Attributes

Action attributes are user-defined attributes that allow for the composition of unrelated test setup code for each test. Often, putting all the setup code in the fixture or namespace setup violates the single responsibility principle—some tests require certain setups while other tests require other setups. Action attributes allow you to specify exactly the setup that a test requires.

One of the advantages of user-defined action attributes is that it moves the setup and teardown code out of the fixture itself and into a more reusable and maintainable code base.

Defining an Action

A user-defined action is either derived from TestActionAttribute:

or can be derived from System.Attribute and implement the ITestAction interface:

In the latter case, the property “Targets” must also be defined.

The Action Targets

The action target (either a fixture or suite) or a test method can be defined through the ActionTargets enumeration (defined in NUnit.Framework):

ActionTargets.Default

Specifying a value of ActionTargets.Default allows the user-defined attribute to be applied to both test fixtures (classes) and tests (methods). However, when applied to a fixture (class), the attribute behaves as if ActionTargets.Suite had been specified—the user-defined action runs once for the entire suite of tests.

ActionTargets.Test

If ActionTargets.Test is returned by the user-defined action, this changes the behavior of when the user-defined action is called. If applied to the fixture (class), then the user-defined action is executed for each test. If the user attribute is applied to a specific test, then the user-defined action is executed for that specific test.

ActionTargets.Suite

If an attribute action target is a suite but is used in the context of a test method, no error is reported—the action simply does not run.

Test and Suite Action Targets

ActionTargets are flags, so they can be combined:

to specify that the user-defined action should execute for both the fixture (the suite of tests) and the individual tests.

The TestDetails Class

This class is instantiated with information about the fixture or test that is about to run (from NUnit.Framework.TestDetails):

These properties can be inspected and specific actions can be taken, such as providing additional console output diagnostic messages.

Assembly Actions

Adhering more to Visual Studio’s concept of AssemblyInitialize and AssemblyCleanup, you can use an assembly-based user action. For example, with the previous code, if the following line is added to the AssemblyInfo.cs file:

then the user-defined action will run when the assembly is loaded:

If the target action for the user-defined action is Suite, the user-defined action executes before and after all fixture tests, behaving as “before or after all suites.” However, note that if the TargetActions.Test is specified, then the action runs before or after all tests in all suites. Therefore, care should be taken with regard to the return value of the Target property and the desired behavior of the user-defined action.

Passing Information to/from Tests from User-Defined Actions

Two questions might be:

  • How do I pass information from the fixture to the user-defined action?
  • How do I use an object that is instantiated by a user-defined action in the test itself?

Passing Information to a User-Defined Action

This is handled straightforwardly as parameters to the user action. Let’s say you want to pass in a connection string to test the behavior for a particular user role. For example, using this stub:

and applying it to the fixture:

results in the following test flow:

Note that the user-defined action for a suite is executed after the “namespace” setup and after the fixture setup. If the user-defined action needs some information that cannot be passed in using the constructor (attribute constructors are restricted to constants) then an approach similar to that outlined next might be useful. However, keep in mind that the user-defined actions should be as autonomous as possible—entanglement with test states should be avoided.

Passing Information from a User-Defined Action to a Test/Test Suite

This is not possible without some form of boundary-crossing container. An extremely simple implementation is to create an object bag from a key-value dictionary:

which allows disconnected objects to easily share data, such as in the user-defined action attribute:

The data can then be accessed in the test methods:

More sophisticated approaches exist and the preceding examples are intended to illustrate how to coordinate a user-defined action with the sharing of information. The prior example has numerous problems—type safety, no checks for whether an object already exists in the container, and so forth—issues that a robust container class should handle.


NUnit Core Assertions

NUnit provides the same or similar assertions in the NUnit.Framework.Assert class as those found in Visual Studio’s Assert class, as described earlier. NUnit also provides the additional assertions:

IsEmpty/IsNotEmpty

These assertions verify that a string is empty or not, or that a collection is empty or not:

Greater/Less

These two assertions work with numeric values and objects implementing IComparable, asserting that the first value is either greater or less than the second value. The purpose for this assertion is that it improves readability of the test.

GreaterOrEqual/LessOrEqual

These two assertions work with numeric values and objects implementing IComparable, asserting that the first value is either “greater than or equal” or “less than or equal” to the second value. The purpose for this assertion is that it improves readability of the test.

IsAssignableFrom/IsNotAssignableFrom

This pair of methods asserts that an expected type is assignable from an object instance of compatible type. This is similar to the IsInstanceOfType assertion in Visual Studio’s test engine. For example, given:

This test passes because the object of type A can be assigned an instance of type B.

Throws/Throws<T>/DoesNotThrow

The non-generic form of this assertion validates that an exception is thrown (or not) when invoking the specified method. For example:

This test passes.

The generic version places the exception type as the generic parameter:

Catch/Catch<T>

These two methods assert that a method throws a type of exception or one derived from it. In contrast, the Throws method before this asserts that exactly the specified exception type is thrown. For example, this code passes:


Collection Assertions

In the NUnit.Framework.CollectionAssert class, NUnit implements the same assertions as found in Visual Studio’s CollectionAssert class, with these additional assertions:

  • IsEmpty/IsNotEmpty
  • IsOrdered

Note that the collection parameter in these methods expects the collection to implement IEnumerable (contrast this with Visual Studio’s test framework, which expects ICollection).

IsEmpty/IsNotEmpty

The CollectionAssert.IsEmpty method is the same as the Assert.IsEmpty method.

IsOrdered

This method verifies that the values in a collection are ordered. For example, because AnObject implements IComparable:

this test passes:


String Assertions

These assertion methods are members of the NUnit.Framework.StringAssert class and are the same as found in Visual Studio’s StringAssert class, with a couple of differences:

  • AreEqualIgnoringCase
  • IsMatch

which are discussed next.

AreEqualIgnoringCase

The assertion “AreEqualIgnoringCase” performs a comparison of two strings ignoring case.

IsMatch

This method is similar to Visual Studio’s StringAssert.Matches method—it asserts that a string matches a regex pattern.


File Assertions

These assertion methods are members of the NUnit.Framework.FileAssert class.

AreEqual/AreNotEqual

These methods assert that two files are identical or not. A byte-by-byte comparison is made of the two files. If the files are empty, they are considered equal. If either or both files do not exist, a FileNotFoundException is thrown.


Directory Assertions

These assertion methods are members of the NUnit.Framework.DirectoryAssert class.

AreEqual/AreNotEqual

These methods assert that the files in the two directories are equal nor not.

IsEmpty/IsNotEmpty

These methods assert that a directory is empty or not.

IsWithin/IsNotWithin

These methods assert that the path contains the specified subdirectory or not.


Other Assertions

That

While one form of this method does take a Boolean parameter as an assertion of truth (behaving exactly like Assert.IsTrue), the predominant form of this method is the ability to compare an actual value using a specific constraint. The NUnit documentation on constraints is quite extensive regarding constraints, so only a short example will be provided here. For example, given this test:

Using the constraint system, the assertion could be rewritten as:

The constraint syntax facilitates the readability of the unit test as well as promotes an extensible architecture for new methods.

IsNan

This method tests whether a double value is “not a number.” A typical case where a value is not a number is division by zero or the square root of a negative value.


Utility Methods

The assert class also provides several utility methods:

Pass

Assert.Pass ends the test immediately by throwing a SuccessException, which results in the test being marked as passing.

Fail

Assert.Fail ends the test immediately by throwing an AssertionException, which results in the test being marked as failing.

Ignore

Assert.Ignore ends the test immediately by throwing an IgnoreException, reporting the test as being ignored.

Inconclusive

Assert.Inconclusive ends the test immediately by throwing an InconclusiveException, reporting the test as being inconclusive. This might be useful when a unit test or fixture determines that a required service is down and therefore the tests against that service cannot proceed.

Tags:

Comments

Related Articles