Python unittest. A bit more practice after theory. Basics and a bit more than basics.

The main purpose of this article is just to share my experience on using unittest module in Python. 

I chose version 2.7.10 for this case, but I think we may expect the same behavior with Python 3.X

Documentation about unittest module available via following link: https://docs.python.org/2/library/unittest.html, the site interface with documentation is easy and you can simply switch to the same topic, but another version of Python.

In this article I will demonstrate my source code snippets with links for GitHub (where you can find full version)

Initially I've prepared a special class with functionality that we will be using to demonstrate unittest usage.

The class can be used for the following:

1. To generate a dictionary with random keys and values
2. To store generated dictionary like a JSON object into a file (by using standard library in Python)
3. To upload from the file the JSON object back to the dictionary

Latest version of the class could be found here (GitHub)

Now let's start with preparation of simple unit tests lists. You can find basic information about the unit test on Wikipedia. It provides enough knowledge to start working with that.

Let's switch to practice. 

First step

The First step is just to create a simple test that will check that the created dictionary is dumped and uploaded correctly.

    def testGenerateAndLoadJSON(self):

        file_name = "generate_and_load_unittest.json"

        expected_data = json_processing.generate_data_for_json_obj()

        json_processing.generate_json_file_with_data(file_name, expected_data)

        actual_data = json_processing.load_data_from_json_file(file_name)

        for exp_key, exp_value in expected_data.items():

            self.assertTrue(actual_data.has_key(exp_key), "Expected key '{}' has not been found in loaded json".format(exp_key))

            self.assertEquals(exp_value, actual_data.get(exp_key), "Dictionaries have different values '{}' for first and '{}' for second for the same key".format(exp_value, actual_data.get(exp_key)))

        for act_key, act_value in actual_data.items():

            self.assertTrue(expected_data.has_key(act_key), "Loaded key '{}' has not been found in dumped json".format(act_key))

            self.assertEquals(act_value, expected_data.get(act_key), "Dictionaries have different values '{}' for first and '{}' for second for the same key".format(act_value, expected_data.get(act_key)))

        # Second execution of the test can be failed because if we do not delete created file

        os.remove(file_name)

What is wrong with that test and what can we do better for future tests?

1. Test includes data generation. What will happen in case we have ~ 100 similar tests and complicated data generation with approximate time ~ 1 sec? Our unit test will violate the rule that execution should go very quickly without any delays
2. Test is responsible for both creating and removing the file. Stop. Is it right? Should we check functionality that creates and removes files during testing for correct data upload? - No.

Let's try to fix some inconsistencies for the next step

Second step

The goal of the second step is to make the size of the test smaller and move data preparation to special functions setUp and clean-up into tearDown. They are pre-defined functions not only for unittest in Python, but for other modules that can be used for unit testing in different programming languages. For example:

1. xUnit for .NET
2. NUnit and testNG for Java

As well as the points above, lets separate the key correctness check from the value correctness check

    def setUp(self):       
        print "{} for {} has been called".format(self.setUp.__name__, self._testMethodName)

        self.file_name = "generate_and_load_unittest.json"

        self.expected_data = json_processing.generate_data_for_json_obj()

    def tearDown(self):

        print "{} for {} has been called".format(self.tearDown.__name__, self._testMethodName)

        # Second execution of the test can be failed because if we do not delete created file

        os.remove(self.file_name)

    def testGenerateAndLoadJSONValidKeys(self):

        original_name = json_processing.generate_json_file_with_data(self.file_name, self.expected_data)

        print "Processing file {}".format(original_name)

        actual_data = json_processing.load_data_from_json_file(original_name)

        for exp_key in self.expected_data.keys():

            self.assertTrue(actual_data.has_key(exp_key), "Expected key '{}' has not been found in loaded json".format(exp_key))

        for act_key in actual_data.keys():

            self.assertTrue(self.expected_data.has_key(act_key), "Loaded key '{}' has not been found in dumped json".format(act_key))

    def testGenerateAndLoadJSONValidValues(self):

        original_name = json_processing.generate_json_file_with_data(self.file_name, self.expected_data)

        print "Processing file {}".format(original_name)

        actual_data = json_processing.load_data_from_json_file(original_name)

        for exp_key, exp_value in self.expected_data.items():

            self.assertEquals(exp_value, actual_data.get(exp_key), "Dictionaries have different values '{}' for first and '{}' for second for the same key".format(exp_value, actual_data.get(exp_key)))

        for act_key, act_value in actual_data.items():

            self.assertEquals(act_value, self.expected_data.get(act_key), "Dictionaries have different values '{}' for first and '{}' for second for the same key".format(act_value, self.expected_data.get(act_key)))

What kind of problems do we have here?

1. We create a file inside the test and we should move this functionality into setUp()
2. Since setUp() function is executed before each test method call, we haven't solved the date generation issue.

But we have achieved one positive change: tests are easy to understand and most of functionality that does not relate to test itself has been moved to special functions

Third step

For the third step lets try to solve problems that we still have in step 2

In details:

1. File preparation should be moved from tests into the setUp() method - it is an easy step
2. Second case is a bit more complicated:
1. Data generation should be moved to special method that is executed only once, before all tests' execution. unittest framework has such capability and it is encapsulated in functions setUpClass() and tearDownClass().
2. We can move functionality that removes the file that has been created during execution of the test into tearDownClass() function, but in this case we should remove all the files that were created. We may notice that initial class already has such functionality, so we may implement our idea without any problems

Final version with required changes below:

    expected_data = {}

    @classmethod

    def setUpClass(cls):

        print "{} for {} has been called".format(cls.setUpClass.__name__, cls.__name__)

        cls.expected_data = json_processing.generate_data_for_json_obj()

    def setUp(self):

        print "{} for {} has been called".format(self.setUp.__name__, self._testMethodName)

        self.file_name = "generate_and_load_unittest.json"

        self.original_name = json_processing.generate_json_file_with_data(self.file_name, self.expected_data)

    def tearDown(self):

        print "{} for {} has been called".format(self.tearDown.__name__, self._testMethodName)

    @classmethod

    def tearDownClass(cls):

        print "{} for {} has been called".format(cls.tearDownClass.__name__, cls.__name__)

        json_processing.clean_up()

    def testGenerateAndLoadJSONValidKeys(self):

        print "Processing file {}".format(self.original_name)

        actual_data = json_processing.load_data_from_json_file(self.original_name)

        for exp_key in self.expected_data.keys():

            self.assertTrue(actual_data.has_key(exp_key), "Expected key '{}' has not been found in loaded json".format(exp_key))

        for act_key in actual_data.keys():

            self.assertTrue(self.expected_data.has_key(act_key), "Loaded key '{}' has not been found in dumped json".format(act_key))

    def testGenerateAndLoadJSONValidValues(self):

        print "Processing file {}".format(self.original_name)

        actual_data = json_processing.load_data_from_json_file(self.original_name)

        for exp_key, exp_value in self.expected_data.items():

            self.assertEquals(exp_value, actual_data.get(exp_key), "Dictionaries have different values '{}' for first and '{}' for second for the same key".format(exp_value, actual_data.get(exp_key)))

        for act_key, act_value in actual_data.items():

            self.assertEquals(act_value, self.expected_data.get(act_key), "Dictionaries have different values '{}' for first and '{}' for second for the same key".format(act_value, self.expected_data.get(act_key)))

Fourth step

Imagine the following situation. We receive a new version of initial class that now allows us to use different combinations of letters and digits as a key (previously - only a combination of letters can be used for a key)

For the first version the following test has been developed:

    def testGenerateAndLoadJSONValidKeysHasOnlyLetters1(self):

        print "Processing file {}".format(self.original_name)

        actual_data = json_processing.load_data_from_json_file(self.original_name)

        for act_key in actual_data.keys():

            self.assertTrue(re.match("[^a-zA-Z]", act_key) is None, "Key should contains only alpha symbols: {}".format(act_key))

We would not like to remove it since, as an example, we have a list of our customers that would not like to switch for a new version of our class. What should we do in this case? We may skip the execution of this test as an outdated functionality. For that purpose we should use decorator skip from unittest module of Python. Lets think a bit about what this case will bring us as a result. The test will be excluded from execution, but it is better to exclude it only if the version of our module is less then 2.0. For that purpose we should use decorator skipIf. In that case for initial (1.0) version we will not execute this test. Both versions of the test with different decorators are given below:

    # general version of skip

    @skip("old functionality")

    def testGenerateAndLoadJSONValidKeysHasOnlyLetters1(self):

        print "Processing file {}".format(self.original_name)

        actual_data = json_processing.load_data_from_json_file(self.original_name)

        for act_key in actual_data.keys():

            self.assertTrue(re.match("[^a-zA-Z]", act_key) is None, "Key should contains only alpha symbols: {}".format(act_key))

    # version of skip that check version of our json_file_generator

    @skipIf(json_file_generator_version > 1, "This functionality is not supported in this version on the json file generator")

    def testGenerateAndLoadJSONValidKeysHasOnlyLetters2(self):

        print "Processing file {}".format(self.original_name)

        actual_data = json_processing.load_data_from_json_file(self.original_name)

        for act_key in actual_data.keys():

            self.assertIsNone(re.match("[^a-zA-Z]", act_key), "Key should contains only alpha symbols: {}".format(act_key))

Version of the full file can be found here. Following the same logic we may prepare special tests for initial version of our class and these tests could be executed only for that version.

Fifth step

Unit tests should also check functionality for exceptional cases. For example, we have the requirement that if a user tries to load json object from file that doesn't exist, exception should be raised. How can we check that situation with our tests? It is possible with a special type of asserts like assertRaise(). We can use it in our tests as part of with .. as statement. If exception occurs, we may check its error number and text to make sure that all is fine and it is the exception that we are looking for.

    def testGenerateAndLoadJSONForInvalidFile(self):

        """

        This test checks that valid exception will be raised if required file will not be found

        """

        invalid_name = "invalid_" + self.original_name

        print "Processing file {}".format(invalid_name)

        with self.assertRaises(IOError) as io_exception:

            # attempt to read file that doesn't exist

            json_processing.load_data_from_json_file(invalid_name)

       

        self.assertEqual(io_exception.exception.errno, 2)

        self.assertEqual(io_exception.exception.strerror, 'No such file or directory')

 You may find the full version of the file here.

Sixth step

So, it is time to think about following: how can we control loading of test cases and load test cases by ourselves?
For the test purpose we will use the same list of tests like we did in the previous step, but we will need to copy-paste our test class three times. So our module will contain three classes like:

1. class GenerateAndLoadJSONTestUpdateFivePart1(unittest.TestCase):
2. class GenerateAndLoadJSONTestUpdateFivePart2(unittest.TestCase):
3. class GenerateAndLoadJSONTestUpdateFivePart3(unittest.TestCase):

In order to load test suites and test we should use TestLoader class from unittest module.

As the first substep we may try to load all test cases from selected files. Instance of that class has the function discover that allows us to load the objects lists of the TestSuite type from files that satisfy the pattern. Minor notice here is that the pattern should be written using the bash shell rules. To run the suite we should simply call function run with instance of TestResult class as a parameter. Example of such implementation is given below:

def loadTestSuitesWithAllTestByDiscover():

    # pattern should be like a bash shell pattern matching

    example_of_loader = unittest.TestLoader()

    list_of_modules = example_of_loader.discover(".", "generate_and_load_unittest_update_[a-z][a-z][a-z].py")

    # Each module has type unittest.TestSuite

    # Lets run all these modules one by one

    for module in list_of_modules:

        actual_result = unittest.TestResult()

        print "<<< Launching of {} test cases >>>".format(module.countTestCases())

        module.run(actual_result)

        print "Launch statistic"

        print "No errors occur " if len(actual_result.errors) == 0 else "{} error(s) occurs".format(len(actual_result.errors))

        print "No skipped tests " if len(actual_result.skipped) == 0 else "{} test(s) has been skipped".format(len(actual_result.skipped))

        print "Run was successful " if actual_result.wasSuccessful() else "Run contains test that did not complete successfully"

Now imagine that for some reason, we would like to allow a user to load only a pre-defined list of test suites. How can we do that? We may notice the following information in description of the discover function:

If load_tests exists then discovery does not recurse into the package, load_tests is responsible for loading all tests in the package.

Based on above, let's add a function below to our test module:

test_cases = (GenerateAndLoadJSONTestUpdateFivePart1, GenerateAndLoadJSONTestUpdateFivePart2)

def load_tests(loader, tests, pattern):

    suite = unittest.TestSuite()

    for test_class in test_cases:

        tests = loader.loadTestsFromTestCase(test_class)

        suite.addTests(tests)

   

    return suite

Unittest framework will load only classes from 'test_cases' in this case, and we may check to make sure that the tests from 'GenerateAndLoadJSONTestUpdateFivePart3' have not been loaded and executed. The function responsible for loads and launchs required test suites that, as seen below, will not be changed and will have the same implementation as shown in the above example.

def loadTestSuitesWithSelectedTestsByDiscover():

        # pattern should be like a bash shell pattern matching

    example_of_loader = unittest.TestLoader()

    list_of_modules = example_of_loader.discover(".", "generate_and_load_unittest_update_[a-z][a-z][a-z][a-z].py")

    # Each module has type unittest.TestSuite

    # Lets run all these modules one by one

    for module in list_of_modules:

        actual_result = unittest.TestResult()

        print "<<< Launching of {} test cases >>>".format(module.countTestCases())

        module.run(actual_result)

        print "Launch statistic"

        print "No errors occur " if len(actual_result.errors) == 0 else "{} error(s) occurs".format(len(actual_result.errors))

        print "No skipped tests " if len(actual_result.skipped) == 0 else "{} test(s) has been skipped".format(len(actual_result.skipped))

        print "Run was successful " if actual_result.wasSuccessful() else "Run contains test that did not complete successfully"

You may find the full version of the module that can load and prepare test suites here.

This is all what I would like to share as an initial step...

I hope you will find it helpful!