In a previous post on unit testing I described the motivation behind our recent move to pytest as a unit testing framework and the experiences we made in the early phase of the migration. Here, I want to highlight a more advanced issue we encountered during the arduous work of porting our domain model unit tests to the new pytest
based framework: The issue of testing complex, hierarchical data structures.

The @pytest.parametrized decorator is very useful for setting up simple test objects on the fly, but since the decorator is called at module load time, you may not be able to create complex test objects with it. Also, the syntax is fairly cumbersome to read with more complex object initializations.

To illustrate the problem, consider the following simple test class diagram ^[1]:

Suppose we want to perform tests on MyEntity instances. Suppose further that we need also properly initialized MyEntityParent, MyEntityChild and MyEntityGrandchild instances attached to our test object. The standard way to achieve this would be a pytest fixture looking like this:

However, if we now needed a differently configured MyEntity instance – say, with two MyEntityChild instances or a different id, we would end up with a lot of code duplication.

After some time playing around with various ideas on how to make the generation of complex test object trees more flexible, I had the idea to simply add one layer of indirection by defining test object factory fixtures for the individual test object classes like this:

When used as a fixture parameter in a test function (or another fixture), calling these factories with no arguments always returns an instance fully initialized with the default arguments defined by the factory fixture:

Very convenient. However, it is also straightforward to create customized test object fixtures using these factories:

There is one thing to keep in mind here: The default behavior of the test object factories is to create only one instance for each unique combination of positional and keyword arguments. So, in the example above, since the children attribute was not customized in the calls to the my_entity_child_fac factory, both MyEntityChild instances created end up with the exact same MyEntityGrandchild instance in their children list. If you absolutely need new instances, you can either pass unique arguments to the factory or call its new method.

On another, perhaps slightly esoteric, side note I want to point out that the code above has the unnerving effect of immediately generating a pylint warning in line 10, code number W0621, informing you that a name from the outer scope in line 4 was just redefined. Getting rid of this warning without disabling it manually or moving the test fixture into a separate conftest module turned out to be not all that easy; I eventually resorted to the following way of declaring my fixtures at the top of my test modules:

and use one of the ingenious pytest hooks to pre-process the module at test collection time (this needs to go somewhere pytest can see it, e.g. in a conftest module in the test package):

Again, I am happy to report that pytest gives me all the tools needed to cope with our sometimes complex testing scenarios and that I do not regret our decision to migrate to pytest at all.

With increasing complexity of the data structures and business logic of our main development project at work we are slowly arriving at a point where the current way of testing is simply no longer tenable: The whole test suite with close to 2000 tests takes about 3 hours to run now. Bad habits are starting to creep in – such as committing your changes without running a full test suite on your local machine and relying on your build system to tell you what you have broken with your last commit. Yes, really bad habits.

There are a number of approaches to improve this desolate situation (cf. also this blog post):

• Speed up data access by staging all testing data in an in-memory database prior to the test run;
• Introduce test tiers, with the simple (and, presumably, faster) model level tests always enabled while the more complex functional tests only run over night;
• Avoid repeated test object construction through careful analysis of which test objects can be reused for which tests.

Ultimately, the solution will probably require a combination of all approaches listed above, but I decided to start with the latter one and to review our current testing framework along the way.

Our group is currently using a “classical” unittest.TestCase testing infrastructure with nose as the test driver. Carefully crafted set_up and tear_down methods in our test base classes ensure that the framework for our application (a REST application based on everest) is initialized and shut down properly for each test. Data shared between tests are kept in the instance namespace or in the class namespace of the test class itself or any of its super classes.

After an in depth review of our complex hierarchy of test classes I realized that it would be difficult to implement the desired flexibility in reusing test objects across tests because the unittest framework offers very limited facilities to separate the creation of test objects from the tests themselves. Looking for alternative frameworks, I quickly came across pytest, which promises to be more modular through rigorous use of dependency injection, i.e., passing each test function exactly the test objects (or “fixtures”) it needs to perform the test. I decided to give pytest a shot and the remainder of this post is about reporting on the experiences I made over the course of this experiment.

For easier porting of the existing tests, I started out replicating the functionality of the current testing base classes derived from unittest.TestCase with pytest fixtures. As it turned out, this made it also easy to understand the different philosophies behind these two testing environments. For example, the base class for tests requiring a Pyramid Configurator looked like this in the unittest framework:

BaseTestCaseWithConfiguration inherits from TestCaseWithIni which provides ini file parsing functionality. The Pyramid Configurator instance is created in the set_up method using parameters that are defined in the class namespace and stored in the test case instance namespace. To avoid cross talk between tests, the tear_down method deconstructs the configurator’s registry.

This test base class is used along the lines of the following contrived example:

Now, the equivalent pytest fixture and test module look like this:

While the mechanics of the pytest fixture has not changed very much (and is perhaps a tad harder to read because of the inline tear_down function), the test module has gotten a lot simpler: There is no need to derive from a base class and the properly initialized Configurator test object is passed automatically to the test function by the pytest framework. Moreover, while the pytest fixtures can depend on each other (in the example, the configurator fixture depends on the ini fixture which again is passed in automatically by pytest), they are much more modular than the TestCase classes and you can pull in whichever fixtures you need in a given test function.

Since pytest comes with unit test integration, most of our old tests ran right out of the box. However, there was no support for the __test__ attribute that nose offers to manually exclude base classes from testing; also, the unittest plugin of pytest does not automatically exclude classes with names starting with an underscore from testing like nose does. Fortunately, fixing these problems was trivial using one of the many, many pytest hooks:

To make the basic test fixtures for everest applications usable from other projects, I bundled them with the test collection hook above as a pytest plugin which I published using the setuptools entry point mechanics alongside the old nose plugin entry point in the everest.setup module like this:

Next, I needed to add support for the custom app-ini-file option that is used to pass configuration options, particularly the configuration of the logging system, to everest applications. This was also straightforward using the pytest_addoption and pytest_configure hooks:

If you now configure a console handler for your logger that uses a logging.StreamHandler to direct output to sys.stderr in an ini file and then pass this to the pytest driver with the app-ini-file option, the logging output from failed tests will be reported by pytest in the “Captured stderr” output section.

Finally, I needed to get code coverage working with pytest. The simplest way to achieve this seemed the pytest-cov plugin for pytest. However, I could not get correct coverage results for the everest sources when using this plugin, presumably because quite a few of the everest modules are loaded when the everest plugin for pytest is initialized, so I decided to run coverage separately from the command line which is not much of an inconvenience and perhaps the right thing to do anyways.

With this setup, we can now use the pytest test driver to run the existing test suite for our REST application while we gradually replace our complex test case class hierarchy with more modular pytest test fixtures that will ultimately help us cut down our test run time.

Of course, every piece of magic has its downsides. If your IDE pampers you with a feature to transport you to the definition of an identifier at the push of a button (F3 in pydev, for instance), then you might find it disconcerting that this will not work with the test fixtures that are magically passed by pytest. However, in most cases it should be very straightforward to find the definition of a particular fixture, since there are only a couple of places where you would sensibly put it (inside the test module if it is not shared across modules or in a confest.py module if it is).

In summary, I think that the benefits of using pytest far outweigh its downsides and I encourage everyone to give it a try.

Inspired by my foray into OData land, I decided to extend the everest query language to support expressions with arbitrary logical operators. During the implementation of this feature, I learned a few things about parsing query expressions in Python which I thought I could share here.

Query strings in everest consist of three main parts: The filtering part which determines the subset to select from the queried resource; the ordering part specifying the sorting order of the elements in the selected subset; and the batching part used to further partition the selected subset into manageable chunks.

So far, the filtering expressions in everest had consisted of a flat sequence of attribute:operator:value triples, separated by tilde (“~“) characters. The query engine interprets tilde characters as logical AND operations and multiple values passed in a criterion as logical OR operations.

The filtering expression grammar was written using the excellent pyparsing package from Paul McGuire (API documentation can be found here). With pyparsing, you can not only define individual expressions in a very readable manner, but also define arbitrary parsing actions to modify the parsing tree on the fly. Let’s illustrate this with a simplified version of the filter expression grammar that permits only integer numbers as criterion values:

Unlike a regular expression, this is largely self-explanatory. Note how we can give an expression a name using the setName method and reuse that name in subsequent expressions and how the suppress method can be used to omit parts of a match from the enclosing group.

The following transcript from an interactive Python session shows this grammar in action:

Not bad, but wouldn’t it be nice if the number literals were converted to integers? This is where parsing actions come in handy; to try this out, update the definition of the number expression like this:

Running our parsing example with the updated grammar will now produce this output:

As desired, the criteria values are returned as Python integers now.

As nifty as these parsing actions are, there is one thing you should know about them: Your callback code should never raise a TypeError. Let me show you why; first, we change the convert_number callback to raise a TypeError like this:

Now, we parse our query string again:

What happened?! The error message seems to suggest that something in your parsing expression went wrong, causing the parser not to pass anything into the parse action callback. Never would you suspect the callback itself since it looks like that has not even been called yet. What is really going on, though, is a piece of dangerous magic in the pyparsing which allows you to define your callbacks with different numbers of arguments:

The wrapper is calling our callback repeatedly with fewer and fewer arguments, expecting a TypeError if the number of arguments passed does not match funcs signature. If your callback itself raises a TypeError, the wrapper gives up and re-raises the last exception – which then produces the misleading traceback shown above. I think this is an example where magic introduced for convenience is actually causing more harm than good ^[1].

Let us now turn to the main topic of this post, adding support for arbitrary query expressions with explicit AND and OR operators. We start by adding a few more elements to our grammar:

The new AND and OR logical operators replace the tilde operator from the original grammar. Giving the AND operator precedence over the OR operator and calling the composition of two criteria a “junction”, we can now extend our query expression as follows:

Testing this with a slightly fancier query string:

Note that the result indeed reflects the operator precedence rules. Let us briefly check if we can still match our simple tilde-separated criteria query strings:

Oops – that did not go so well. The problem is that both the junctions and the delimitedList expression match a single criterion, so the parser stops after extracting the first of the two criteria. We can solve this issue easily by putting the simple criteria expression first and changing it such that it requires at least two criteria:

Checking:

Lovely. By now, we only have one problem left: How can we override the precedence rules and form the OR junction before the AND junction in the above example? To achieve this, we introduce open (“(“) and close (“)“) parentheses as grouping operators and make our junctions expression recursive as follows:

The key element here is the forward declaration of the junctions expression which allows us to use it in the definition of a junction_element before it is fully defined.

Let’s test this by adding parentheses around the OR clause in the query string above:

Great, it works! Beware, however, the common pitfall of introducing “left recursion” into recursive grammars: If you use a forward-declared expression at the beginning of another expression declaration, you instruct the parser to replace the former with itself until the maximum recursion depth is reached. Luckily, pyparsing provides a handy validate method which detects left-recursion in grammar expressions. With the following code fragment used in place of the junction_element expression above, the grammar will no longer load but instead fail with a RecursiveGrammarException:

I hope this post has given you a taste of the wealth of possibilities that pyparsing offers; if you are curious, the full everest filter parsing grammar can be found here. I would recommend pyparsing to anyone trying to solve parsing problems of moderate to high complexity in Python, with the few caveats I have already mentioned.